Paper deep dive

Provable Weak-to-Strong Generalization via Benign Overfitting

David X. Wu, Anant Sahai

Year: 2024Venue: arXiv preprintArea: Scalable OversightType: TheoreticalEmbeddings: 389

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 6:55:02 PM

Summary

The paper provides a theoretical framework for 'weak-to-strong generalization' in machine learning, where a weak teacher model supervises a strong student model using imperfect pseudolabels. Using a stylized overparameterized spiked covariance model with Gaussian covariates, the authors identify two asymptotic phases for the strong student: random guessing and successful generalization. They prove that even when weak teacher labels are essentially random, a strong student can extract sufficient signal to generalize, provided the model architecture allows for the representation of the weak teacher's logic.

Entities (5)

Anant Sahai · researcher · 100%David X. Wu · researcher · 100%Weak-to-Strong Generalization · paradigm · 98%Spiked covariance model · theoretical-model · 95%Minimum l2 interpolation · algorithm · 92%

Relation Signals (3)

David X. Wu → authored → Provable Weak-to-Strong Generalization via Benign Overfitting

confidence 100% · Paper title and author list

Anant Sahai → authored → Provable Weak-to-Strong Generalization via Benign Overfitting

confidence 100% · Paper title and author list

Weak-to-Strong Generalization → studiedin → Spiked covariance model

confidence 95% · We theoretically investigate weak-to-strong generalization... in a stylized overparameterized spiked covariance model

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The classic teacher-student model in machine learning posits that a strong teacher supervises a weak student to improve the student's capabilities. We instead consider the inverted situation, where a weak teacher supervises a strong student with imperfect pseudolabels. This paradigm was recently brought forth by Burns et al.'23 and termed \emph{weak-to-strong generalization}. We theoretically investigate weak-to-strong generalization for binary and multilabel classification in a stylized overparameterized spiked covariance model with Gaussian covariates where the weak teacher's pseudolabels are asymptotically like random guessing. Under these assumptions, we provably identify two asymptotic phases of the strong student's generalization after weak supervision: (1) successful generalization and (2) random guessing. Our techniques should eventually extend to weak-to-strong multiclass classification. Towards doing so, we prove a tight lower tail inequality for the maximum of correlated Gaussians, which may be of independent interest. Understanding the multilabel setting reinforces the value of using logits for weak supervision when they are available.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

388,805 characters extracted from source content.

Expand or collapse full text

Provable Weak-to-Strong Generalization via Benign Overfitting David X. Wu Department of EECS UC Berkeley Berkeley, CA 94720 david_wu@berkeley.edu Anant Sahai Department of EECS UC Berkeley Berkeley, CA 94720 sahai@eecs.berkeley.edu Abstract The classic teacher-student model in machine learning posits that a strong teacher supervises a weak student to improve the student’s capabilities. We instead consider the inverted situation, where a weak teacher supervises a strong student with imperfect pseudolabels. This paradigm was recently brought forth by Burns et al. (2023) and termed weak-to-strong generalization. We theoretically investigate weak-to-strong generalization for binary and multilabel classification in a stylized overparameterized spiked covariance model with Gaussian covariates where the weak teacher’s pseudolabels are asymptotically like random guessing. Under these assumptions, we provably identify two asymptotic phases of the strong student’s generalization after weak supervision: (1) successful generalization and (2) random guessing. Our techniques should eventually extend to weak-to-strong multiclass classification. Towards doing so, we prove a tight lower tail inequality for the maximum of correlated Gaussians, which may be of independent interest. Understanding the multilabel setting reinforces the value of using logits for weak supervision when they are available. 1 Introduction Motivated by the problem of aligning increasingly capable models, Burns et al. (2023) introduced the framework of weak-to-strong generalization111This situation is related, but slightly different, from the earlier identified problem of easy-to-hard generalization (see e.g. Schwarzschild et al. (2021); Hase et al. (2024)) wherein a model is trained on “easy” cases but has to generalize to “hard” cases. The most important distinction is that in weak-to-strong generalization, the weak teacher actually gets things wrong., which draws an analogy between humans supervising superhuman AIs and weaker teacher models supervising stronger student models. This inverts the classic teacher-student model framework, which typically assumes that the teacher is stronger than the student. Burns et al. (2023) found that using GPT-2 to finetune GPT-4 can recover most of the performance of standard supervised finetuning with human-annotated data across standard NLP benchmarks, but struggles on more difficult tasks such as chess puzzles or reward modeling. One failure mode observed by Burns et al. (2023) is that the strong student sometimes learns to mimic the weak teacher, i.e. the strong model overfits to the limitations of the weaker one. Since Burns et al. (2023), this phenomenon has been studied in a variety of other empirical settings (see e.g. Ji et al. (2024); Guo et al. (2024); Liu and Alahi (2024); Yang et al. (2024); Tao and Li (2024)). These empirical observations naturally lead to the question that we aim to answer in this paper: Can we identify a simple, concrete theoretical setting where we can provably exhibit different phases of weak-to-strong generalization? Below, we survey closely related areas of research. Pseudolabeling and synthetic data. In semi-supervised learning, pseudolabeling refers to the method of using one model’s outputs to generate pseudolabels for unlabeled data (Lee et al., 2013; Arazo et al., 2020; Rizve et al., 2021; Zhang et al., 2021; Cascante-Bonilla et al., 2021; He et al., 2024). These pseudolabels are then used to supervise the target model. We consider using the weak model to generate pseudolabels for the strong model in our concrete setting. Synthetic data generation is another widely popular paradigm for scaling up models in the absence of human-annotated data; see, e.g. (Nikolenko, 2021; Liu et al., 2024; Chen et al., 2021; Figueira and Vaz, 2022) and references therein. In this approach, one uses generative models to generate synthetic data, which can then be labeled and used to supervise other models. The use of synthetic data to train models resembles the weak-to-strong setup that we study, although we assume the unlabeled datapoints are generated from the ground-truth distribution. Nevertheless, we believe that extending our techniques may yield interesting insights on the success and failure modes of synthetic data. Supervised finetuning and scaling laws. In contemporary deep learning, the dominant paradigm is to pretrain large (e.g. billions of parameters) models on copious (e.g. trillions of tokens) unlabeled data in a self-supervised fashion and then finetune the model on relatively tiny (e.g. less than 100K) finetuning datasets. The prevailing wisdom suggests that (1) pretraining helps the model learn useful features that can be repurposed for specific tasks and (2) the optimal size of models should scale with the amount of data (see, e.g. (Kaplan et al., 2020; Wei et al., 2021; Wang et al., 2022b; Liu et al., 2022; Hoffmann et al., 2022)). Much effort has gone into improving the effectiveness and efficiency of finetuning (Houlsby et al., 2019; Hu et al., 2021; Lester et al., 2021; Dettmers et al., 2024). According to NTK theory, if the model weights do not move far from initialization during finetuning, we can approximate finetuning as training a generalized linear model using (tangent) features. These features arise from the gradients of the network’s outputs with respect to the parameters at finetuning initialization. This motivates our studying the behavior of interpolating linear models in the context of supervised finetuning. Our concrete theoretical setting sits within the Gaussian-features linear-model style of Wei et al. (2022); Belkin et al. (2020); Mei and Montanari (2022); Bartlett et al. (2020); Muthukumar et al. (2020; 2021); Chatterji and Long (2021); Wang and Thrampoulidis (2021); Subramanian et al. (2022); Wang et al. (2022a; 2021); Cornacchia et al. (2023). Knowledge distillation. The topic of weak-to-strong generalization is related to the extensive literature on knowledge distillation, which originated in the desire to compress large powerful models (teachers) into smaller ones (students) (Hinton et al., 2015; Buciluǎ et al., 2006). See Gou et al. (2021) for a general survey. Theoretical perspectives for the essentially underparameterized case were developed in Phuong and Lampert (2019); Ji and Zhu (2020), and the theoretically engaged literature in this area has continued to expand; see Yuan et al. (2024); Ojha et al. (2023); Safaryan et al. (2023); Alballa and Canini (2024); Hong et al. (2024); Sarnthein et al. (2023); Zhao and Zhu (2023); Das and Sanghavi (2023); Xu et al. (2024); Nagarajan et al. (2023); Borup and Andersen (2023); Harutyunyan (2023); Stanton et al. (2021); Mobahi et al. (2020) for a few representative more recent examples. The question of how knowledge distillation can improve the student’s generalization to surpass the teacher (especially when they have the same architecture) has led to the identification of several different underlying mechanisms (Yuan et al., 2024; Safaryan et al., 2023; Sarnthein et al., 2023; Das and Sanghavi, 2023; Nagarajan et al., 2023; Mobahi et al., 2020). Of these, the closest in spirit to our approach is the regularization viewpoint of Mobahi et al. (2020), which studies a kernel-regression model. They call out the crucial role of the spectrum of the Gram matrix — more specifically, self-distillation accentuates the importance of the larger eigenvalues, which has a regularizing effect by making the corresponding basis functions more prominent in the learned pattern. Concurrent theoretical work. Zhang et al. (2024) engages with generative models and observes that temperature can play an important role in allowing a trained model to surpass its training sources. Somerstep et al. (2024) takes a transfer-learning perspective and asserts that naive fine-tuning on weak pseudolabels tends not to work; our results show conditions where this does in fact succeed. Charikar et al. (2024) takes a representation-centric perspective in a regression setting and zooms in on the question of how much better the representation is for the stronger model. Lang et al. (2024) studies the classification setting and takes a neighborhood perspective that posits that the stronger model’s neighborhood structure allows it to average over the weak labels to get generalization. At a high level, our work along with Mobahi et al. (2020); Charikar et al. (2024); Lang et al. (2024) all circle around the idea that weak-to-strong generalization works when the cascading learning process purifies representations in the true direction and contracts in false directions. 1.1 Contributions In this paper, we explore weak-to-strong generalization in a stylized theoretical model that captures the dynamics of finetuning with weak supervision studied in Burns et al. (2023). Under a simple overparameterized spiked covariance model for the pretraining features, we prove that finetuning an overparameterized linear classifier using minimum ℓ2subscriptℓ2 _2ℓ2 norm interpolation on top of these features provably exhibits two distinct phases of weak-to-strong generalization. In particular, under certain scalings, as we increase the number of weakly labeled finetuning examples, the strong learner’s asymptotic accuracy transitions from (1) random guessing to (2) perfect generalization; see Section 3 for a precise statement. To be specific, we study the generalization of interpolating linear models with and without weak supervision. Although our results can be generalized to any weak teacher which produces logits using a linear head, we assume for the sake of concreteness that the weak teacher is also an interpolating linear model which can be fully expressed by the strong student. We discuss how our theoretical results connect to realistic supervised finetuning scenarios in Section 4. Our results strongly hinge on the tight analysis for benign overfitting in multiclass classification in Wu and Sahai (2024), although the study of benign overfitting for regression and binary classification was already carried out by Bartlett et al. (2020); Muthukumar et al. (2021). 2 Preliminaries and setup Below, we set up the weak-to-strong learning task in Section 2.1, along with the data assumptions in Section 2.2, and finally specify the concrete end-to-end learning algorithm we study in Section 2.3. 2.1 Weak-to-strong setup To study weak-to-strong generalization, we will consider a simple setup which encapsulates the dynamics of standard supervised finetuning as well as training linear probes on intermediate activations. We will now give a high level description of the weak-to-strong setting, and in subsequent sections formally define the specific assumptions we make to theoretically study weak supervision. Suppose we have n labeled datapoints and m=nusuperscriptm=n^um = nitalic_u unlabeled datapoints, where u>11u>1u > 1. This matches the modern ML paradigm where labeled data is scarce and unlabeled data is abundant. We assume that we have access to two sets of features extracted from the datapoints: the weak and strong features. Using these features, we will create a weak-to-strong setup with a weak model and a strong model, where the weak model is used to generate hard pseudolabels to train the strong model (see 1 for a formal definition). The learning task at hand is binary classification, where each model uses its respective features obtained from the datapoints — see Section 4 for how our techniques apply to the multiclass setting. Clearly, to get nontrivial learning guarantees, there needs to be some relationship between the weak and strong features. To specify this, we will assume that the weak and strong features come from an appropriate weak-to-strong ensemble of features, which we formally define in Section 2.2. One way to interpret the weak-to-strong ensemble is that the true hidden direction is highlighted more in the strong features than the weak features; see Figure 1. In supervised finetuning and linear probing, the strong and weak features come from pretraining via the neural tangent features and intermediate activations, respectively. For example, in the GPT2 to GPT4 weak-to-strong setup of Burns et al. (2023), the weak features come from GPT2 pretraining, whereas the strong features come from GPT4 pretraining. Broadly speaking, we study the case where the true labels are generated by a distinguished (but unknown) low-rank subspace hidden in very high dimensional space. To be concrete, we will study two different classifiers in this setup: (1) subscript f_ weakitalic_fsansserif_weak: train/finetune on n datapoints using weak features in ℝdsuperscriptℝsubscriptR^d_ weakblackboard_Rdsansserif_weak and ground-truth labels. (2) subscript f_ w2sitalic_fsansserif_w2s: train/finetune on m≫nmuch-greater-thanm nm ≫ n datapoints using strong features in ℝdsuperscriptℝR^dblackboard_Rd and hard pseudolabels generated from subscript f_ weakitalic_fsansserif_weak. We study a scheme where subscript f_ weakitalic_fsansserif_weak and subscript f_ w2sitalic_fsansserif_w2s are linear models trained by performing minimum ℓ2subscriptℓ2 _2ℓ2 interpolation (MNI) on their respective training sets; see Section 2.3 for a formal definition of MNI. Remark 2.1. Instead of training with hard (categorical) pseudolabels, one could use the real-valued scores from subscript f_ weakitalic_fsansserif_weak. This would only affect constants which are not crucial to any of our results. We will measure generalization via the test accuracy of these classifiers. In particular, let ℓ⁢(⋅)ℓ⋅ (·)ℓ ( ⋅ ) be the 0-1 loss function, and let ⁢[ℓ⁢(⋅)]delimited-[]ℓ⋅ E[ (·)]E [ ℓ ( ⋅ ) ] be the expected test error over a fresh test sample. We introduce the following desiderata to define weak-to-strong generalization for binary classification. Desiderata 1. The main desiderata are the following: (i) The strong model asymptotically generalizes222The natural extension to multiclass settings with k different classes would require ⁢[ℓ⁢()]=1−Θ⁢(1k)delimited-[]ℓsubscript1Θ1 E[ ( f_ weak)]=1- ( 1k)E [ ℓ ( italic_fsansserif_weak ) ] = 1 - Θ ( divide start_ARG 1 end_ARG start_ARG k end_ARG ). when trained on the m weakly labeled datapoints: ⁢[ℓ⁢()]=on⁢(1).delimited-[]ℓsubscriptsubscript1 E[ ( f_ w2s)]=o_n(1).E [ ℓ ( italic_fsansserif_w2s ) ] = oitalic_n ( 1 ) . (i) The strong model can fully represent the weak model. (i) The weak model asymptotically does not generalize: ⁢[ℓ⁢()]=12−on⁢(1)delimited-[]ℓsubscript12subscript1 E[ ( f_ weak)]= 12-o_n(1)E [ ℓ ( italic_fsansserif_weak ) ] = divide start_ARG 1 end_ARG start_ARG 2 end_ARG - oitalic_n ( 1 ). In Section 3, we show that the above desiderata are achievable in a simple toy model; see Theorem 3.3 for a formal statement. This paints a rather striking picture: there are situations where the weak labels asymptotically look like random guessing (Item 1.i), the strong model can perfectly imitate the weak labels (Item 1.i), yet the strong model still asymptotically generalizes by extracting enough signal out of the plentiful weak labels (Item 1.i).333In particular, Item 1.i allows the failure mode of the strong model imitating the weak model and also rules out more trivial sources of weak labels, such as independent label noise. If the strong model cannot represent the noise in the weak labels, then weak-to-strong generalization is intuitively much simpler. We also include some bonus desiderata, which paint a comparison to natural alternative training methods. We can also provably achieve these bonus desiderata in certain regimes; see Remark 3.4. Desiderata 2. The extra desiderata are the following: (i) PCA cannot recover the low-rank structure from n+mn+mn + m observations of the strong features. (i) Let subscript f_ strongitalic_fsansserif_strong be the strong model when trained on n datapoints using strong features in ℝdsuperscriptℝR^dblackboard_Rd and ground-truth labels. Then subscript f_ strongitalic_fsansserif_strong asymptotically fails: ⁢[ℓ⁢()]=12−on⁢(1)delimited-[]ℓsubscript12subscript1 E[ ( f_ strong)]= 12-o_n(1)E [ ℓ ( italic_fsansserif_strong ) ] = divide start_ARG 1 end_ARG start_ARG 2 end_ARG - oitalic_n ( 1 ). 2.2 Data model Throughout, we will consider data with zero mean Gaussian covariates, which can be viewed as an idealized version of the pretraining features or representations. These covariates are generated from an ambient standard Gaussian vector =(g1,…,gD)∈ℝDsubscript1…subscriptsuperscriptℝ g=(g_1,…,g_D) ^Ditalic_g = ( g1 , … , gitalic_D ) ∈ blackboard_RD. We make Gaussianity assumptions for the sake of theoretical tractability; in Section 4 we discuss potential extensions to other settings. Covariates. A learner observes iid features i∼N⁢(0,Σ)similar-tosubscript0Σ x_i N(0, )italic_xitalic_i ∼ N ( 0 , Σ ), where we emphasize that the feature covariances Σ∈ℝd×dΣsuperscriptℝ ^d× dΣ ∈ blackboard_Rd × d are unknown and different for each learner. The different Σ Σ’s capture how weak or strong the features are for each model; we make this precise below. Note that isubscript x_iitalic_xitalic_i is a linear transformation of the underlying randomness i∼N⁢(0,ID)similar-tosubscript0subscript g_i N(0,I_D)italic_gitalic_i ∼ N ( 0 , Iitalic_D ). We will often refer to the eigendecomposition Σ=U⁢Λ⁢U⊤ΣΛsuperscripttop =U U Σ = U Λ U⊤, where U∈ℝd×dsuperscriptℝU ^d× dU ∈ blackboard_Rd × d is orthogonal and Λ∈ℝd×dΛsuperscriptℝ ^d× dΛ ∈ blackboard_Rd × d is diagonal. Labels. For binary classification, we consider hard labels generated by the signs of Gaussians, so that y=sgn⁢(⟨,∗⟩)sgnexpectation-valuesubscripty=sgn( g, v_*)y = sgn ( ⟨ start_ARG italic_g , italic_v∗ end_ARG ⟩ ), where ∗subscript v_*italic_v∗ is an unknown unit-norm direction. To analyze the generalization of various learners, we will allow ourselves to study labels generated by various directions vitalic_v, not just the true ∗subscript v_*italic_v∗, all of which we assume are unknown.444To study weak supervision, we take =subscript v= w_ weakitalic_v = italic_wsansserif_weak, the direction that the weak model learns. For multiclass classification, we assume the labels are generated via y=arg⁢maxj∈[k]⁡⟨,∗(j)⟩subscriptargmaxdelimited-[]superscriptsubscripty= *arg\,max_j∈[k] g, v_*^(j) = start_OPERATOR arg max end_OPERATORj ∈ [ k ] ⟨ italic_g , italic_v∗( j ) ⟩, where ∗(1),…,∗(k)superscriptsubscript1…superscriptsubscript v_*^(1),…, v_*^(k)italic_v∗( 1 ) , … , italic_v∗( k ) are all unknown. Following Muthukumar et al. (2021); Subramanian et al. (2022); Wu and Sahai (2024), we will make the following assumption on the true label directions to simplify the analysis. However, to study weak-to-strong generalization, we will eventually have to analyze a weak label direction which is not 1-sparse. One of our main contributions is showing how to analyze this case. Assumption 1 (1-sparse assumption). We say that the labels satisfy the 1111-sparse assumption relative to covariance Σ Σ if the following holds. The label defining direction ∗subscript v_*italic_v∗ (directions ∗(1),…,∗(k)superscriptsubscript1…superscriptsubscript v_*^(1),…, v_*^(k)italic_v∗( 1 ) , … , italic_v∗( k ) for multiclass) is aligned with a top eigenvector (top-k eigenvectors for multiclass) of Σ Σ such that, in an eigenbasis where ∗=1subscriptsubscript1 v_*= e_1italic_v∗ = italic_e1 and ∗(i)=isuperscriptsubscriptsubscript v_*^(i)= e_iitalic_v∗( i ) = italic_eitalic_i, respectively, we have y=sgn⁢(x1)sgnsubscript1 y=sgn(x_1)y = sgn ( x1 ) (Binary) y=arg⁢maxj∈[k]⁡xjsubscriptargmaxdelimited-[]subscript y= *arg\,max_j∈[k]x_jy = start_OPERATOR arg max end_OPERATORj ∈ [ k ] xitalic_j (Multiclass) For example, if the 1111-sparse assumption holds for the strong covariance, then in a strong eigenbasis where the strong features are independent, the labels are generated by axis-aligned directions. Bi-level ensemble and weak-to-strong ensemble. To simplify the analysis, we follow Muthukumar et al. (2021); Subramanian et al. (2022); Wu and Sahai (2024) and assume the eigenvalues Λ Λ of the covariance are parameterized by the bi-level ensemble defined shortly. The bi-level ensemble is a simple overparameterized version of the well-known spiked covariance model for PCA. Definition 1 (Bi-level ensemble). Let ∼N⁢(0,Σ)similar-to0Σ x N(0, )italic_x ∼ N ( 0 , Σ ), where the covariance Σ=U⁢Λ⁢U⊤ΣΛsuperscripttop =U U Σ = U Λ U⊤. The bi-level ensemble parameterizes Λ=Λ⁢(p,q,r)Λ = (p,q,r)Λ = Λ ( p , q , r ), where p>11p>1p > 1, 0⩽r<1010 r<10 ⩽ r < 1, and 0<q<(p−r)00<q<(p-r)0 < q < ( p - r ). The number of features (d), number of spiked directions (s), and degree of favoring (a) all scale with the number of training points (n) as follows: d=⌊np⌋,s=⌊nr⌋,a=n−q.formulae-sequencesuperscriptformulae-sequencesuperscriptsuperscript d= n^p ,s= n^r ,a=n^-q\,.d = ⌊ nitalic_p ⌋ , s = ⌊ nitalic_r ⌋ , a = n- q . (1) Then Λ=diag(λi)i∈[d] =diag( _i)_i∈[d]Λ = diag ( λitalic_i )i ∈ [ d ], where λj=a⁢ds,1⩽j⩽s(1−a)⁢d−s,otherwise.subscriptcases11otherwise _j= cases ads,&1 j s\\ (1-a)dd-s,&otherwise cases.λitalic_j = start_ROW start_CELL divide start_ARG a d end_ARG start_ARG s end_ARG , end_CELL start_CELL 1 ⩽ j ⩽ s end_CELL end_ROW start_ROW start_CELL divide start_ARG ( 1 - a ) d end_ARG start_ARG d - s end_ARG , end_CELL start_CELL otherwise end_CELL end_ROW . (2) If the above holds, we refer to xitalic_x, Λ Λ, or Σ Σ as being drawn from the bi-level ensemble. We use the shorthands λF≜a⁢ds≜subscript _F adsλitalic_F ≜ divide start_ARG a d end_ARG start_ARG s end_ARG and λU≜(1−a)⁢d−s≜subscript1 _U (1-a)dd-sλitalic_U ≜ divide start_ARG ( 1 - a ) d end_ARG start_ARG d - s end_ARG to denote the favored and unfavored eigenvalues, respectively. In particular, the parameterization controls the total number of features, d=np≫nsuperscriptmuch-greater-thand=n^p nd = nitalic_p ≫ n, as well as the dimension of the low-rank subspace s=nr≪nsuperscriptmuch-less-thans=n^r ns = nitalic_r ≪ n. For multiclass classification, we allow the number of classes to also scale with the number of datapoints n. Definition 2 (Scaling for multiclass). For multiclass classification with k classes, we have k=ck⁢⌊nt⌋subscriptsuperscriptk=c_k n^t = citalic_k ⌊ nitalic_t ⌋, where 0⩽t<r00 t<r0 ⩽ t < r and cksubscriptc_kcitalic_k is a positive integer. We now pin down a concrete weak-to-strong ensemble, which boils down to specifying the joint distribution of the weak and strong features. We impose a subset relationship between the weak and strong features in the strong basis; see Figure 1 for a diagram. Assumption 2 (Weak-to-strong subset ensemble). Let Λ=Λ⁢(p,q,r)∈ℝd×dΛsuperscriptℝ = (p,q,r) ^d× dΛ = Λ ( p , q , r ) ∈ blackboard_Rd × d denote the strong eigenvalues and Λ=Λ⁢(p,q,r)∈ℝd×dsubscriptΛsubscriptsubscriptsubscriptsuperscriptℝsubscriptsubscript _ weak= (p_ weak,q_ weak,r_ % weak) ^d_ weak× d_ weakΛsansserif_weak = Λ ( psansserif_weak , qsansserif_weak , rsansserif_weak ) ∈ blackboard_Rdsansserif_weak × dsansserif_weak denote the weak eigenvalues, both drawn from the bi-level ensemble. Let λF,≜a⁢ds≜subscriptsubscriptsubscriptsubscript _F, weak a_ weakd_ weaks% _ weakλitalic_F , sansserif_weak ≜ divide start_ARG asansserif_weak dsansserif_weak end_ARG start_ARG ssansserif_weak end_ARG and λU,≜(1−a)⁢d−s≜subscript1subscriptsubscriptsubscriptsubscript _U, weak (1-a_ weak)d_ weak% d_ weak-s_ weakλitalic_U , sansserif_weak ≜ divide start_ARG ( 1 - asansserif_weak ) dsansserif_weak end_ARG start_ARG dsansserif_weak - ssansserif_weak end_ARG denote the weak favored and unfavored eigenvalues, respectively. Let U be any distinguished eigenbasis of Σ Σ where ∗=1subscriptsubscript1 v_*= e_1italic_v∗ = italic_e1 (Assumption 1). The weak and strong features in the basis U are related as follows. (1) ∼N⁢(0,Λ)similar-tosubscript0Λ x_ strong N(0, )italic_xsansserif_strong ∼ N ( 0 , Λ ), where Λ=λF⁢I[s]+λU⁢I[d]∖[s]Λsubscriptsubscriptdelimited-[]subscriptsubscriptdelimited-[]delimited-[] = _FI_[s]+ _UI_[d] [s]Λ = λitalic_F I[ s ] + λitalic_U I[ d ] ∖ [ s ]. (2) There exists subsets of coordinates S⊆[s],T⊆[d]∖[s]formulae-sequencedelimited-[]delimited-[]delimited-[]S [s],T [d] [s]S ⊆ [ s ] , T ⊆ [ d ] ∖ [ s ], with 1∈S11∈ S1 ∈ S and |S|=ssubscript S=s_ weak| start_ARG S end_ARG | = ssansserif_weak, such that =(λF,λF⁢ΠS+λU,λU⁢ΠT)⁢=dN⁢(0,λF,⁢IS+λU,⁢IT).subscriptsubscriptsubscriptsubscriptΠsubscriptsubscriptsubscriptΠsubscriptsuperscript0subscriptsubscriptsubscriptsubscript x_ weak=( _F, weak _F% _S+ _U, weak _U _T) x% _ strong d=N(0, _F, weak% I_S+ _U, weakI_T)\,.italic_xsansserif_weak = ( square-root start_ARG divide start_ARG λitalic_F , sansserif_weak end_ARG start_ARG λitalic_F end_ARG end_ARG Πitalic_S + square-root start_ARG divide start_ARG λitalic_U , sansserif_weak end_ARG start_ARG λitalic_U end_ARG end_ARG Πitalic_T ) italic_xsansserif_strong start_RELOP SUPERSCRIPTOP start_ARG = end_ARG start_ARG d end_ARG end_RELOP N ( 0 , λitalic_F , sansserif_weak Iitalic_S + λitalic_U , sansserif_weak Iitalic_T ) . Here, ΠSsubscriptΠ _SΠitalic_S denotes the projection onto the axis-aligned subspace indexed by S. We will often abuse notation and restrict ∈ℝdsubscriptsuperscriptℝ x_ weak ^ditalic_xsansserif_weak ∈ blackboard_Rd to the coordinates in S∪TS∪ TS ∪ T, viewing ∈ℝdsubscriptsuperscriptℝsubscript x_ weak ^d_ weakitalic_xsansserif_weak ∈ blackboard_Rdsansserif_weak. Hence, the weak favored features are a subset of the strong favored features, the weak unfavored features are a subset of the strong unfavored features, with different bi-level scalings, and furthermore the 1111-sparse assumption holds for both Σ Σ and ΣsubscriptΣ _ weakΣsansserif_weak. Clearly, under Assumption 2, Item 1.i is satisfied, and the subset ensemble is essentially the simplest relationship between the features one could impose to achieve the desideratum. In Section 4, we discuss how we expect the results to change if we relax these data modeling assumptions. Figure 1: Visualization of subset ensemble (Assumption 2) relating weak and strong features. Notice the decreased favoring for the weak features, and how the weak features are a subset of the strong features in their respective category. Hence, a linear model on the strong features can simulate one on the weak features (Item 1.i). Moreover, the label defining directions, represented by the green shaded box, are in the span of both the weak and strong features. 2.3 Learning algorithm Our models are all trained using minimum ℓ2subscriptℓ2 _2ℓ2-norm interpolation (MNI), which corresponds to the asymptotic behavior of gradient descent with zero initialization (Ji and Telgarsky, 2021). Define the data matrix ∈ℝn×dsuperscriptℝ X ^n× ditalic_X ∈ blackboard_Rn × d by ⊤≜[1⋯n]≜superscripttopmatrixsubscript1⋯subscript X [ x_1&·s& x_n]italic_X⊤ ≜ [ start_ARG start_ARG start_ROW start_CELL italic_x1 end_CELL start_CELL ⋯ end_CELL start_CELL italic_xitalic_n end_CELL end_ROW end_ARG end_ARG ]. Let (i)∈ℝnsuperscriptsuperscriptℝ y^(i) ^nitalic_y( i ) ∈ blackboard_Rn be the (centered) one-hot vector for class i∈[k]delimited-[]i∈[k]i ∈ [ k ]. In MNI, we learn a linear score function (i)superscript f^(i)italic_f( i ) via the optimization problem (i)superscript f^(i)italic_f( i ) =arg⁡min⁡‖2s.t.⁢=(i).formulae-sequenceabsentsubscriptsubscriptnorm2s.t.superscript = _ f\| f\|_2 .t.\ X% f= y^(i).= arg minbold_italic_f ∥ italic_f ∥2 s.t. italic_X italic_f = italic_y( i ) . (MNI) The coefficients of (i)superscript f^(i)italic_f( i ) can be easily computed to be (i)=⊤⁢(⁢⊤)−1⁢(i)≜⊤⁢−1⁢(i),superscriptsuperscripttopsuperscriptsuperscripttop1superscript≜superscripttopsuperscript1superscript f^(i)= X ( X X )^-1 y^(i)% X A^-1 y^(i),italic_f( i ) = italic_X⊤ ( italic_X italic_X⊤ )- 1 italic_y( i ) ≜ italic_X⊤ italic_A- 1 italic_y( i ) , where the matrix ≜⁢⊤∈ℝn×n≜superscripttopsuperscriptℝ A X X ^n× nitalic_A ≜ italic_X italic_X⊤ ∈ blackboard_Rn × n is the unnormalized Gram matrix and invertible almost surely. At test time, given a fresh sample ∼N⁢(0,Σ)similar-tosubscript0Σ x_ test N(0, )italic_xsansserif_test ∼ N ( 0 , Σ ), in the binary case we predict y^=sgn⁢(⟨,⟩)^sgnexpectation-valuesubscript y=sgn( f, x_ test)over start_ARG y end_ARG = sgn ( ⟨ start_ARG italic_f , italic_xsansserif_test end_ARG ⟩ ), and for multiclass we predict the class with the highest score: y^=arg⁢maxi∈[k]⁡⟨(i),⟩.^subscriptargmaxdelimited-[]expectation-valuesuperscriptsubscript y= *arg\,max_i∈[k] f^(i)% , x_ test.over start_ARG y end_ARG = start_OPERATOR arg max end_OPERATORi ∈ [ k ] ⟨ start_ARG italic_f( i ) , italic_xsansserif_test end_ARG ⟩ . To be very explicit, let us formally define the end-to-end traning procedure for weak-to-strong binary classification (cf. the description before Remark 2.1), with an obvious extension to multiclass settings. Procedure 1 (Weak-to-strong training). The weak learner observes an initial dataset of n datapoints (~i,,yi)i∈[n]subscriptsubscript~subscriptdelimited-[]( x_i, weak,y_i)_i∈[n]( over~ start_ARG italic_x end_ARGi , sansserif_weak , yitalic_i )i ∈ [ n ], where ~i,subscript~ x_i, weakover~ start_ARG italic_x end_ARGi , sansserif_weak are the weak features for the iiith datapoint and yi=sgn⁢(⟨i,∗⟩)subscriptsgnexpectation-valuesubscriptsubscripty_i=sgn( g_i, v_*)yitalic_i = sgn ( ⟨ start_ARG italic_gitalic_i , italic_v∗ end_ARG ⟩ ) is the corresponding clean hard label. We train ∈ℝdsubscriptsuperscriptℝsubscript f_ weak ^d_ weakitalic_fsansserif_weak ∈ blackboard_Rdsansserif_weak using MNI on these n clean datapoints. Then, both learners observe m extra unlabeled datapoints, where the weak model sees weak features (j,)j∈[m]subscriptsubscriptdelimited-[]( x_j, weak)_j∈[m]( italic_xitalic_j , sansserif_weak )j ∈ [ m ] and the strong model sees the corresponding strong features (j,)j∈[m]subscriptsubscriptdelimited-[]( x_j, strong)_j∈[m]( italic_xitalic_j , sansserif_strong )j ∈ [ m ]. Generate m hard pseudolabels via y^j,=sgn⁢(⟨,j,⟩)subscript^sgnexpectation-valuesubscriptsubscript y_j, weak=sgn( f_ % weak, x_j, weak)over start_ARG y end_ARGj , sansserif_weak = sgn ( ⟨ start_ARG italic_fsansserif_weak , italic_xitalic_j , sansserif_weak end_ARG ⟩ ), and use MNI to train ∈ℝdsubscriptsuperscriptℝ f_ w2s ^ditalic_fsansserif_w2s ∈ blackboard_Rd on (j,,y^j,)j∈[m]subscriptsubscriptsubscript^delimited-[]( x_j, strong, y_j, weak)_j∈[m]( italic_xitalic_j , sansserif_strong , over start_ARG y end_ARGj , sansserif_weak )j ∈ [ m ]. 3 Main results In this section, we state our main results for binary classification (Theorem 3.3) and multilabel classification (Theorem 3.5). We focus on the regime q+r>uq+r>uq + r > u and q+r>1subscriptsubscript1q_ weak+r_ weak>1qsansserif_weak + rsansserif_weak > 1, since Muthukumar et al. (2021, Theorem 13) implies that these are the only nontrivial regimes for binary classification under the bi-level ensemble. In this regime, we also tighten the previous error rates for binary and multiclass classification in the bi-level ensemble from (Wu and Sahai, 2024, Theorem 3.2); the proof can be found in Appendix E. To that end, we establish a new concentration inequality for the lower tail of the maximum of correlated Gaussians, which may be of independent interest; we defer its statement to the end of this section. Theorem 3.1 (Regimes with clean labels). Suppose the strong features have bi-level covariance Σ=Σ⁢(p,q,r)Σ = (p,q,r)Σ = Σ ( p , q , r ) (Definition 1), where q+r>11q+r>1q + r > 1, the true multiclass labels are 1111-sparse (Assumption 1), and the number of classes k=⌊nt⌋superscriptk= n^t = ⌊ nitalic_t ⌋ (Definition 2). Then the test error for subscript f_ strongitalic_fsansserif_strong MNI-trained with n clean multiclass labels satisfies ⁢[ℓ⁢()]delimited-[]ℓsubscript E[ ( f_ strong)]E [ ℓ ( italic_fsansserif_strong ) ] =on⁢(1),if⁢t<min⁡1−r,τ1−Θ⁢(1k),if⁢t>min⁡1−r,τ,absentcasessubscript1if1subscript1Θ1if1subscript = caseso_n(1),&\ if\ t< 1-r, _% strong\\ 1- ( 1k),&\ if\ t> 1-r, _% strong cases,= start_ROW start_CELL oitalic_n ( 1 ) , end_CELL start_CELL if t < min start_ARG 1 - r , τsansserif_strong end_ARG end_CELL end_ROW start_ROW start_CELL 1 - Θ ( start_ARG divide start_ARG 1 end_ARG start_ARG k end_ARG end_ARG ) , end_CELL start_CELL if t > min start_ARG 1 - r , τsansserif_strong end_ARG end_CELL end_ROW , (3) where τ≜p+1−2⁢(q+r)≜subscript12 _ strong p+1-2(q+r)τsansserif_strong ≜ p + 1 - 2 ( q + r ). Furthermore, for binary classification (i.e. k=22k=2k = 2), the explicit error rates are given by ⁢[ℓ⁢()]=12−1π⁢arctan⁡(Θ⁢(nτ)).delimited-[]ℓsubscript121arctangentΘsuperscriptsubscript E[ ( f_ strong)]= 12-% 1π ( (n _ strong)).E [ ℓ ( italic_fsansserif_strong ) ] = divide start_ARG 1 end_ARG start_ARG 2 end_ARG - divide start_ARG 1 end_ARG start_ARG π end_ARG arctan ( start_ARG Θ ( nitalic_τsansserif_strong ) end_ARG ) . (4) By generalizing other results from Wu and Sahai (2024) to handle imperfect labels, we formally establish weak-to-strong generalization for binary classification in the subset ensemble. We give a brief technical overview of the proof in Section 3.1 and prove it formally in Appendix C. Theorem 3.2 (Weak-to-strong generalization for subset ensemble). Consider the setup in 1 where the weak model subscript f_ weakitalic_fsansserif_weak is MNI-trained on n correctly labeled examples and the strong model subscript f_ w2sitalic_fsansserif_w2s is MNI-trained on m=nusuperscriptm=n^um = nitalic_u weakly pseudolabeled examples, where q+r>uq+r>uq + r > u and q+r>1subscriptsubscript1q_ weak+r_ weak>1qsansserif_weak + rsansserif_weak > 1. In addition, we make the following data assumptions: (1) The true binary labels are 1111-sparse (Assumption 1) for the strong covariance. (2) The weak and strong features follow the subset ensemble (Assumption 2) with bi-level eigenvalues Λ⁢(p,q,r)Λ (p,q,r)Λ ( p , q , r ) and Λ⁢(p,q,r)Λsubscriptsubscriptsubscript (p_ weak,q_ weak,r_ weak)Λ ( psansserif_weak , qsansserif_weak , rsansserif_weak ), respectively, scaled relative to n. (3) There are not too many weakly labeled examples: u<p+1+q+r−(q+r)21subscriptsubscript2u< p+1+q+r-(q_ weak+r_ weak)2u < divide start_ARG p + 1 + q + r - ( qsansserif_weak + rsansserif_weak ) end_ARG start_ARG 2 end_ARG. Recall τ≜p+1−2⁢(q+r)≜subscript12 _ strong p+1-2(q+r)τsansserif_strong ≜ p + 1 - 2 ( q + r ). Then, the resulting test error for subscript f_ w2sitalic_fsansserif_w2s satisfies ⁢[ℓ⁢()]delimited-[]ℓsubscript E[ ( f_ w2s)]E [ ℓ ( italic_fsansserif_w2s ) ] =on⁢(1),if⁢u>q+r−min⁡1−r,τ12−on⁢(1),if⁢u<q+r−min⁡1−r,τ.absentcasessubscript1ifsubscriptsubscript1subscript12subscript1ifsubscriptsubscript1subscript = caseso_n(1),&\ if\ u>q_ weak+r_% weak- 1-r, _ strong\\ 12-o_n(1),&\ if\ u<q_ weak+r_ weak-% 1-r, _ strong. cases= start_ROW start_CELL oitalic_n ( 1 ) , end_CELL start_CELL if u > qsansserif_weak + rsansserif_weak - min start_ARG 1 - r , τsansserif_strong end_ARG end_CELL end_ROW start_ROW start_CELL divide start_ARG 1 end_ARG start_ARG 2 end_ARG - oitalic_n ( 1 ) , end_CELL start_CELL if u < qsansserif_weak + rsansserif_weak - min start_ARG 1 - r , τsansserif_strong end_ARG . end_CELL end_ROW (5) In other words, under the stated parameter regimes, weak-to-strong generalization occurs when the strong model is trained on sufficiently many weak binary labels.555We expect weak imitation to occur as soon as m≫dmuch-greater-thanm dm ≫ d if the models are trained using gradient descent, i.e. the classic underparameterized regime with more weakly labeled datapoints than features. Figure 2 demonstrates that the conditions for weak-to-strong generalization are not vacuous. We make the third enumerated assumption for technical reasons, but we believe this condition is essentially tight. By combining Theorems 3.1 and 3.2, we obtain the exact conditions where the main desiderata from Section 2 are satisfied in the bi-level ensemble. The regimes where our theorem applies are depicted in Figures 2(a) and 2(b), and we validated our theory with numerical simulations of MNI with n=5050n=50n = 50 in Figures 2(c) and 2(d); see Appendix F for more details on the experiments. Theorem 3.3 (Main result). Under the same setup as Theorem 3.2, all the core weak-to-strong desiderata Items 1.i, 1.i and 1.i are satisfied under the following conditions: u u >q+r−min⁡1−r,τabsentsubscriptsubscript1subscript >q_ weak+r_ weak- 1-r, _% strong> qsansserif_weak + rsansserif_weak - min start_ARG 1 - r , τsansserif_strong end_ARG (Weak-to-strong generalization) 00 0 >τ,absentsubscript > _ weak,> τsansserif_weak , (Weak model does not generalize) where τ≜p+1−2⁢(q+r)≜subscriptsubscript12subscriptsubscript _ weak p_ weak+1-2(q_ weak+r_% weak)τsansserif_weak ≜ psansserif_weak + 1 - 2 ( qsansserif_weak + rsansserif_weak ) and τ≜p+1−2⁢(q+r)≜subscript12 _ strong p+1-2(q+r)τsansserif_strong ≜ p + 1 - 2 ( q + r ). In particular, the conditions on the number of weak examples m=nusuperscriptm=n^um = nitalic_u are non-vacuous as long as τ≜p+1−(q+r+q+r)>0.≜subscript1subscriptsubscript0 _ w2s p+1-(q+r+q_ weak+r_% weak)>0.τsansserif_w2s ≜ p + 1 - ( q + r + qsansserif_weak + rsansserif_weak ) > 0 . Note that under Assumption 2, Item 1.i is immediately satisfied since the weak features are spanned by the strong features. Also, as a sanity check, we confirm that whenever weak-to-strong generalization occurs with m weak labels, the strong model would generalize with m clean labels. By Theorem 3.1, if the strong model was instead trained on m=nusuperscriptm=n^um = nitalic_u clean examples, it would generalize whenever u>2⁢(q+r)−p2u>2(q+r)-pu > 2 ( q + r ) - p. By expanding the definition of τsubscript _ strongτsansserif_strong and using q+r>1subscriptsubscript1q_ weak+r_ weak>1qsansserif_weak + rsansserif_weak > 1, u>q+r−τ=2⁢(q+r)−p+(q+r−1)>2⁢(q+r)−p,subscriptsubscriptsubscript2subscriptsubscript12u>q_ weak+r_ weak- _ strong=2(q+r)-p+(q_% weak+r_ weak-1)>2(q+r)-p,u > qsansserif_weak + rsansserif_weak - τsansserif_strong = 2 ( q + r ) - p + ( qsansserif_weak + rsansserif_weak - 1 ) > 2 ( q + r ) - p , which recovers the condition with m clean labels stated above. Remark 3.4 (Bonus desiderata). If q+r>uq+r>uq + r > u, as in the setup of Theorem 3.3, PCA fails to recover the spiked subspace (Shen et al., 2013; Fan and Wang, 2015), so the bonus Item 2.i automatically holds. Additionally, by using the condition for where the strong model fails with n clean examples from Theorem 3.1 (τ<0subscript0 _ strong<0τsansserif_strong < 0), one can construct parameter regimes where the bonus Item 2.i also holds. The theorem also implies the strong model cannot bootstrap its own performance. Indeed, some basic algebra shows τ=τ=τsubscriptsubscriptsubscript _ w2s= _ strong= _ weakτsansserif_w2s = τsansserif_strong = τsansserif_weak if (p,q,r)=(p,q,r)subscriptsubscriptsubscript(p,q,r)=(p_ weak,q_ weak,r_ weak)( p , q , r ) = ( psansserif_weak , qsansserif_weak , rsansserif_weak ). We can extend the above result to the multilabel setting, which is a variant of multiclass classification where a datapoint can have multiple positive labels. The basic idea is that multilabel training can be approximated as several independent binary classification problems, and then we can apply the binary analysis. Since this requires additional setup, we defer the formal details to Appendix D. Theorem 3.5 (Informal, see Theorem D.3). Under the analogous assumptions as Theorem 3.3 for multilabel classification, there is weak-to-strong generalization for subscript f_ w2sitalic_fsansserif_w2s trained on m weakly hard multilabeled examples from subscript f_ weakitalic_fsansserif_weak in the exact same regimes. The above result also suggests an interesting result for true multiclass problems with one-hot labels. In particular, one can use the multilabel scheme to generate weak multilabels for a multiclass weak-to-strong classifier subscript f_ w2sitalic_fsansserif_w2s. One can argue that whenever multilabel weak-to-strong generalization occurs, so too does multiclass weak-to-strong generalization (see Section D.1 for more justification). By comparing to the regimes for clean multiclass labels due to Theorem 3.1, there exists regimes where the strong model would fail if trained using the same number of clean multiclass labels (i.e. only one positive label per example). At a high level, this can happen because there are too many classes, which sparsifies the label vectors. In contrast, the weak multilabels do not suffer from the sparsity issue, even though the underlying signal is noisier. This appears to be related to the strategy of using soft labels or logits in the pseudolabeling and knowledge distillation literature. (a) (q,r)=(0.9,0.8),(p,q,r)=(1.4,0.9,0.4)formulae-sequence0.90.8subscriptsubscriptsubscript1.40.90.4(q,r)=(0.9,0.8),(p_ weak,q_ weak,r_ weak)=(1.4,0% .9,0.4)( q , r ) = ( 0.9 , 0.8 ) , ( psansserif_weak , qsansserif_weak , rsansserif_weak ) = ( 1.4 , 0.9 , 0.4 ). (b) (q,r)=(0.9,0.5)0.90.5(q,r)=(0.9,0.5)( q , r ) = ( 0.9 , 0.5 ), (p,q,r)=(1.1,0.9,0.2)subscriptsubscriptsubscript1.10.90.2(p_ weak,q_ weak,r_ weak)=(1.1,0.9,0.2)( psansserif_weak , qsansserif_weak , rsansserif_weak ) = ( 1.1 , 0.9 , 0.2 ) (c) (p,q,r)=(2,0.6,0.6),(p,q,r)=(1.4,0.9,0.5)formulae-sequence20.60.6subscriptsubscriptsubscript1.40.90.5(p,q,r)=(2,0.6,0.6),(p_ weak,q_ weak,r_ weak)=(1% .4,0.9,0.5)( p , q , r ) = ( 2 , 0.6 , 0.6 ) , ( psansserif_weak , qsansserif_weak , rsansserif_weak ) = ( 1.4 , 0.9 , 0.5 ). (d) (p,q,r)=(1.5,0.6,0.8),(p,q,r)=(1.4,0.9,0.5)formulae-sequence1.50.60.8subscriptsubscriptsubscript1.40.90.5(p,q,r)=(1.5,0.6,0.8),(p_ weak,q_ weak,r_ weak)=% (1.4,0.9,0.5)( p , q , r ) = ( 1.5 , 0.6 , 0.8 ) , ( psansserif_weak , qsansserif_weak , rsansserif_weak ) = ( 1.4 , 0.9 , 0.5 ). Figure 2: (Top): Regime plots for weak-to-strong generalization based on Theorem 3.3. The blue region is the successful w2s regime, and the red region is where w2s training fails. The white region corresponds to regimes where either the hypotheses of the theorem fail to hold, or invalid settings of parameters for the bi-level ensemble. (Bottom): Comparison of simulations of MNI test accuracies for two different regimes where n=5050n=50n = 50. Observe how the weak accuracy is close to random guessing and how the weak-to-strong accuracy increases as m increases. As corroborated by the plots, Theorem 3.3 predicts w2s success in Figure 2(c) and failure in Figure 2(d). Finally, we state the aforementioned concentration inequality for the maximum of correlated Gaussians, which may be of independent interest. Our result complements the sharp bound of Lopes and Yao (2022) for more moderate deviations; the proof can be found in Appendix E. Theorem 3.6 (Lower tail for correlated Gaussians). Let ρ0∈(0,1)subscript001 _0∈(0,1)ρ0 ∈ ( 0 , 1 ) be a parameter bounded away from 00 and 1111, and let (gi)i∈[N]subscriptsubscriptdelimited-[](g_i)_i∈[N]( gitalic_i )i ∈ [ N ] be jointly Gaussian with zero mean and unit variance. Suppose ⁢[gi⁢gj]⩽ρ0delimited-[]subscriptsubscriptsubscript0 E[g_ig_j] _0E [ gitalic_i gitalic_j ] ⩽ ρ0 for all distinct i,j∈[N]delimited-[]i,j∈[N]i , j ∈ [ N ]. For any 0⩽tN=δ0⁢2⁢(1−ρ0)⁢log⁡N0subscriptsubscript021subscript00 t_N= _0 2(1- _0) N0 ⩽ titalic_N = δ0 square-root start_ARG 2 ( 1 - ρ0 ) log N end_ARG where δ0∈[0,1)subscript001 _0∈[0,1)δ0 ∈ [ 0 , 1 ) is bounded away from 1111, there is a constant C>00C>0C > 0 depending only on ρ0subscript0 _0ρ0 such that ⁢[maxi∈[N]⁡gi⩽tN]⩽C⋅N(1−δ0)2⁢(1−1ρ0)⁢(log⁡N)1−ρ0⁢(2−δ0)−δ02⁢ρ0.subscriptdelimited-[]subscriptsubscript⋅superscriptsuperscript1subscript0211subscript0superscript1subscript02subscript0subscript02subscript0 Pr [ _i∈[N]g_i t_N] C· N% ^(1- _0)^2(1- 1 _0)( N) 1- _0(2-δ% _0)- _02 _0.Pr [ start_ARG maxitalic_i ∈ [ N ] gitalic_i ⩽ titalic_N end_ARG ] ⩽ C ⋅ N( 1 - δ0 ) start_POSTSUPERSCRIPT 2 ( 1 - divide start_ARG 1 end_ARG start_ARG ρ0 end_ARG ) end_POSTSUPERSCRIPT ( log N )divide start_ARG 1 - ρ0 ( 2 - δ0 ) - δ0 end_ARG start_ARG 2 ρ0 end_ARG . In particular, one can take C=ρ01−ρ0subscript01subscript0C= _01- _0C = square-root start_ARG divide start_ARG ρ0 end_ARG start_ARG 1 - ρ0 end_ARG end_ARG. If we further have ⁢[gi⁢gj]=ρ0delimited-[]subscriptsubscriptsubscript0 E[g_ig_j]= _0E [ gitalic_i gitalic_j ] = ρ0 for all distinct i,ji,ji , j and tN=O⁢(log⁡log⁡Nlog⁡N)subscriptt_N=O ( N N)titalic_N = O ( start_ARG divide start_ARG log log N end_ARG start_ARG square-root start_ARG log N end_ARG end_ARG end_ARG ), then ⁢[maxi∈[N]⁡gi⩽tN]=Θ⁢(N1−1ρ0⁢(log⁡N)12⁢ρ0−1).subscriptdelimited-[]subscriptsubscriptΘsuperscript11subscript0superscript12subscript01 Pr [ _i∈[N]g_i t_N]= (% N^1- 1 _0( N) 12 _0-1).Pr [ start_ARG maxitalic_i ∈ [ N ] gitalic_i ⩽ titalic_N end_ARG ] = Θ ( start_ARG N1 - divide start_ARG 1 end_ARG start_ARG ρ0 end_ARG ( log N )divide start_ARG 1 end_ARG start_ARG 2 ρ0 end_ARG - 1 end_ARG ) . 3.1 Technical overview In this section, we illustrate the main ideas of the proof of Theorem 3.2. Recall that the true label is sgn⁢(x∗)sgnsubscriptsgn(x_*)sgn ( x∗ ), where x∗=⟨,∗⟩subscriptexpectation-valuesubscriptsubscriptx_*= g_ test, v_*x∗ = ⟨ start_ARG italic_gsansserif_test , italic_v∗ end_ARG ⟩. Thus, studying weak-to-strong generalization amounts to analyzing ⁢[sgn⁢(⟨,⟩)=sgn⁢(x∗)],delimited-[]sgnexpectation-valuesubscriptsubscriptsgnsubscript Pr[sgn( f_ % w2s, x_ test)=sgn(x_*)],Pr [ sgn ( ⟨ start_ARG italic_fsansserif_w2s , italic_xsansserif_test end_ARG ⟩ ) = sgn ( x∗ ) ] , (6) where the fresh test point ∼N⁢(0,Σ)similar-tosubscript0Σ x_ test N(0, )italic_xsansserif_test ∼ N ( 0 , Σ ). Observe that conditioned on the training data Xitalic_X, the true Gaussian x∗subscriptx_*x∗ and ⟨,⟩expectation-valuesubscriptsubscript f_ w2s, x_ test⟨ start_ARG italic_fsansserif_w2s , italic_xsansserif_test end_ARG ⟩ are jointly Gaussian random variables, so there exist λ∈ℝλ λ ∈ blackboard_R and g∼N⁢(0,1)similar-to01g N(0,1)g ∼ N ( 0 , 1 ) independent of x∗subscriptx_*x∗ such that ⟨,⟩=λ⁢x∗+g.expectation-valuesubscriptsubscriptsubscript f_ w2s, x_ test% =λ x_*+g.⟨ start_ARG italic_fsansserif_w2s , italic_xsansserif_test end_ARG ⟩ = λ x∗ + g . (7) The exact value of λ depends on exactly how much of x∗subscriptx_*x∗’s signal survives through the weak-to-strong training process. Intuitively, the larger λ is, the higher the accuracy. Indeed, it is well-known from the classical study of noise stability that the probability in Equation 6 is precisely 12+1π⁢arctan⁡(λ)121arctangent 12+ 1π (λ)divide start_ARG 1 end_ARG start_ARG 2 end_ARG + divide start_ARG 1 end_ARG start_ARG π end_ARG arctan ( start_ARG λ end_ARG ). To compute λ, we use Gram-Schmidt to decompose subscript f_ w2sitalic_fsansserif_w2s to obtain the signal and noise with respect to x∗subscriptx_*x∗. We term these quantities the survival and contamination of x∗subscriptx_*x∗, respectively. Without loss of generality, we can rotate to the basis U where the strong features are drawn iid from N⁢(0,Λ)0ΛN(0, )N ( 0 , Λ ), where Λ=diag⁡(λ1,…,λd)Λdiagsubscript1…subscript =diag( _1,…, _d)Λ = diag ( λ1 , … , λitalic_d ). After applying the basis change, one easily computes ⟨,⟩=dN⁢(0,‖Λ⁢‖22)superscriptexpectation-valuesubscriptsubscript0superscriptsubscriptnormΛsubscript22 f_ w2s, x_ test % d=N(0, f_ w2s_2^2)⟨ start_ARG italic_fsansserif_w2s , italic_xsansserif_test end_ARG ⟩ start_RELOP SUPERSCRIPTOP start_ARG = end_ARG start_ARG d end_ARG end_RELOP N ( 0 , ∥ start_ARG Λ italic_fsansserif_w2s end_ARG ∥22 ). The survival and contamination are then defined as follows. Definition 3 (Survival and contamination). Let ∈ℝDsuperscriptℝ v ^Ditalic_v ∈ blackboard_RD be a unit vector and let ∈ℝdsuperscriptℝ f ^ditalic_f ∈ blackboard_Rd be a linear classifier for features drawn from N⁢(0,∑i∈[d]λi⁢i⁢i⊤)0subscriptdelimited-[]subscriptsubscriptsuperscriptsubscripttopN(0, _i∈[d] _i v_i v_i )N ( 0 , ∑i ∈ [ d ] λitalic_i italic_vitalic_i italic_vitalic_i⊤ ), where 1,…,d∈ℝDsubscript1…subscriptsuperscriptℝ v_1,…, v_d ^Ditalic_v1 , … , italic_vitalic_d ∈ blackboard_RD are orthonormal. The survival and contamination of vitalic_v in fitalic_f is defined as SU⁢()≜∑i∈[d]λi⁢[i]⁢⟨i,⟩,⁢()≜‖Λ⁢‖22−SU⁢()2.formulae-sequence≜SUsubscriptidelimited-[]dsubscriptidelimited-[]iexpectation-valuesubscripti≜superscriptsubscriptnormΛ22SUsuperscript2 SU( v) _i∈[d] _i% f[i] v_i, v, CN( % v) f_2^2- SU( v)^2.SU ( v ) ≜ ∑i ∈ [ d ] square-root start_ARG λroman_i end_ARG f [ i ] ⟨ start_ARG vroman_i , v end_ARG ⟩ , sansserif_CN ( v ) ≜ square-root start_ARG ∥ start_ARG Λ f end_ARG ∥22 - SU ( v )2 end_ARG . If fitalic_f is trained by MNI on =sgn⁢(⟨,⟩)sgnexpectation-value y=sgn( g, w)italic_y = sgn ( ⟨ start_ARG italic_g , italic_w end_ARG ⟩ ), we write SUn⁢(|)subscriptSUnconditional SU_n( v| w)SUn ( v | w ) for the survival of vitalic_v given n labels generated from witalic_w, and similarly for n⁢(|)subscriptconditional CN_n( v| w)sansserif_CNn ( italic_v | italic_w ). If n or witalic_w is clear from context, we omit them. To interpret these notions, consider the simple case where =e1subscript1 v=e_1italic_v = e1 and i=eisubscriptsubscript v_i=e_iitalic_vitalic_i = eitalic_i, which is the 1-sparse setting studied by previous papers. There, the survival is simply just λ1⁢[1]subscript1delimited-[]1 _1 f[1]square-root start_ARG λ1 end_ARG italic_f [ 1 ] and the contamination is ∑i>1λi⁢[i]2subscript1subscriptsuperscriptdelimited-[]2 _i>1 _i f[i]^2square-root start_ARG ∑i > 1 λitalic_i italic_f [ i ]2 end_ARG; this was used to analyze binary classification in (Muthukumar et al., 2021). In the weak-to-strong setting, we have ⟨,⟩=SUm⁢(∗|)⁢x∗+m⁢(∗|)⁢gexpectation-valuesubscriptsubscriptsubscriptSUmconditionalsubscriptsubscriptsubscriptxsubscriptmconditionalsubscriptsubscriptg f_ w2s, x_ test= % SU_m( v_*| v_ weak)x_*+ CN_m( v_*|% v_ weak)g⟨ start_ARG italic_fsansserif_w2s , italic_xsansserif_test end_ARG ⟩ = SUm ( v∗ | vsansserif_weak ) x∗ + sansserif_CNm ( v∗ | vsansserif_weak ) g, where =/‖subscriptsubscriptnormsubscript v_ weak= f_ weak/ f_ weakitalic_vsansserif_weak = italic_fsansserif_weak / ∥ start_ARG italic_fsansserif_weak end_ARG ∥ is the unit-norm direction learned by the weak model, and g is a standard Gaussian independent of x∗subscriptx_*x∗. Plugging in x∗=⟨,∗⟩subscriptexpectation-valuesubscriptx_*= g, v_*x∗ = ⟨ start_ARG italic_g , italic_v∗ end_ARG ⟩ into (7) yields ⁢[sgn⁢(SUm⁢(∗|)⁢x∗+m⁢(∗|)⁢g)=sgn⁢(x∗)],delimited-[]sgnsubscriptSUmconditionalsubscriptsubscriptsubscriptxsubscriptmconditionalsubscriptsubscriptgsgnsubscriptx Pr[sgn( SU_m( v_*| v% _ weak)x_*+ CN_m( v_*| v_ weak)g)=% sgn(x_*)],Pr [ sgn ( SUm ( v∗ | vsansserif_weak ) x∗ + sansserif_CNm ( v∗ | vsansserif_weak ) g ) = sgn ( x∗ ) ] , (8) from which we can read off λ=SUm⁢(∗|)m⁢(∗|)subscriptSUmconditionalsubscriptsubscriptsubscriptconditionalsubscriptsubscriptλ= SU_m( v_*| v_ weak) CN% _m( v_*| v_ weak)λ = divide start_ARG SUm ( v∗ | vsansserif_weak ) end_ARG start_ARG sansserif_CNm ( italic_v∗ | italic_vsansserif_weak ) end_ARG. Thus, the explicit formula for the probability yields that subscript f_ w2sitalic_fsansserif_w2s approaches perfect accuracy if and only if SUm⁢(∗|)m⁢(∗|)=ω⁢(1)subscriptSUmconditionalsubscriptsubscriptsubscriptconditionalsubscriptsubscript1 SU_m( v_*| v_ weak) CN_m(% v_*| v_ weak)=ω(1)divide start_ARG SUm ( v∗ | vsansserif_weak ) end_ARG start_ARG sansserif_CNm ( italic_v∗ | italic_vsansserif_weak ) end_ARG = ω ( 1 ), whereas its accuracy approaches random guessing if and only if SUm⁢(∗|)m⁢(∗|)=o⁢(1)subscriptSUmconditionalsubscriptsubscriptsubscriptconditionalsubscriptsubscript1 SU_m( v_*| v_ weak) CN_m(% v_*| v_ weak)=o(1)divide start_ARG SUm ( v∗ | vsansserif_weak ) end_ARG start_ARG sansserif_CNm ( italic_v∗ | italic_vsansserif_weak ) end_ARG = o ( 1 ). We now turn to the strategy for controlling the survival and contamination. Using the explicit MNI formula for subscript f_ weakitalic_fsansserif_weak and subscript f_ w2sitalic_fsansserif_w2s, this analysis reduces to controlling certain random bilinear forms. Applying standard eigenvalue concentration results and a version of the Hanson-Wright concentration inequality for bilinear forms developed by Wu and Sahai (2024), we can precisely control the coordinates of subscript f_ weakitalic_fsansserif_weak, and hence subscript v_ weakitalic_vsansserif_weak. Once we have control over subscript v_ weakitalic_vsansserif_weak, we can then study subscript f_ w2sitalic_fsansserif_w2s. One technical difficulty is that the weak labels are no longer 1111-sparse, even if the true labels are. This marks a departure from previous analyses of benign overfitting in classification, which all assumed 1-sparse labels (Muthukumar et al., 2021; Wang et al., 2021; Wu and Sahai, 2024). To overcome this obstacle, we develop tools to analyze MNI beyond the 1-sparse regime, namely by using the Woodbury inversion formula and carefully bounding error terms using Hanson-Wright. Under the subset ensemble, the explicit parameterizations for the survival and contamination then yield the stated conditions for weak-to-strong generalization. 4 Discussion We now discuss further extensions which suggest interesting new weak-to-strong phenomena. Modeling assumptions. In this paper, we assumed the features were Gaussian for the sake of theoretical tractability. One could relax this to vector-subgaussian, as this notion is preserved under rotation. On the other hand, the bi-level and subset ensembles were chosen as a minimal set of tractable theoretical conditions to study weak-to-strong generalization. In principle, our techniques could be extended to more complicated feature ensembles, but it would likely be quite involved. As corroborated by our experiments in Figs. 2(c) and 2(d), in practice one does not observe sharp transitions in test error from 1212 12divide start_ARG 1 end_ARG start_ARG 2 end_ARG to 00 using weak supervision — such a sharp transition corresponds to a Performance Gap Recovered (PGR) equal to 1 in Burns et al. (2023) This phenomenon can nevertheless be justified by our theory. First, the transition we prove is asymptotic; the rate of convergence matters to predict the PGR for finite n. Another factor is the 1111-sparse assumption for the true labels. We argue in Appendix G that this is essentially necessary to get this sharp transition in asymptotic test error. In practice, the true directions are rarely aligned with the eigenvectors of the covariance, so the transition would instead between different constant levels of test error. Weak imitation. Our main result Theorem 3.2 uses the subset ensemble Assumption 2 to establish two distinct phases of weak-to-strong generalization. By adding a third intermediate eigenvalue level — thus relaxing the bi-level ensemble to a tri-level ensemble — and changing the relationship between the strong and weak basis, we expect a third distinct regime of weak imitation to occur in an overparameterized setting. At a high level, this can happen if the weak classifier puts most of its mass in its own unfavored direction vitalic_v, but in the strong feature space vitalic_v has an intermediate level of favoring. Given enough weakly labeled datapoints, the strong model will eventually learn to imitate the weak model. Nevertheless, it is still possible for weak-to-strong generalization to occur, with the regime for weak-to-strong generalization widening as the intermediate favoring level decreases. Multiclass classification. By leveraging Theorem 3.1 and the techniques developed for binary and multilabel classification in the appendix, we expect our result to extend to the multiclass setting. The main technical challenge is controlling the expected weak-to-strong training behavior, which is significantly more complicated. However, multilabel weak supervision overcomes this challenge and is more tractable to analyze, as discussed after Theorem 3.5. Relevance to realistic settings. One connection to more realistic settings comes through the NTK perspective, as touched upon in the introduction. To reiterate, if finetuning remains in the lazy training phase, then the supervised finetuning dynamics are captured by an appropriate generalized linear model defined by the NTK approximation. The strong and weak features are therefore learned from the pretraining phase. Another relevant setting is training linear probes on intermediate activations to interpret large models (Alain and Bengio, 2016; Marks and Tegmark, 2023; Nanda et al., 2023; Zou et al., 2023). Hence, under appropriate overparameterization of the intermediate activations, our results also apply there. To go beyond classification problems, one could hope to use some recent implicit bias results for next token prediction (Thrampoulidis, 2024) to study language models. Acknowledgments We would like to acknowledge support from an OpenAI Superalignment grant and NSF AST-2132700. References Alain and Bengio [2016] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016. Alballa and Canini [2024] Norah Alballa and Marco Canini. Practical insights into knowledge distillation for pre-trained models, 2024. Arazo et al. [2020] Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International joint conference on neural networks (IJCNN), pages 1–8. IEEE, 2020. Bartlett et al. [2020] Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30063–30070, 2020. Belkin et al. [2020] Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science, 2(4):1167–1180, 2020. Borup and Andersen [2023] Kenneth Borup and Lars Nørvang Andersen. Self-distillation for gaussian process regression and classification, 2023. Buciluǎ et al. [2006] Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535–541, 2006. Burns et al. [2023] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023. Cascante-Bonilla et al. [2021] Paola Cascante-Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 6912–6920, 2021. Charikar et al. [2024] Moses Charikar, Chirag Pabbaraju, and Kirankumar Shiragur. Quantifying the gain in weak-to-strong generalization. arXiv preprint arXiv:2405.15116, 2024. Chatterji and Long [2021] Niladri S Chatterji and Philip M Long. Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. Journal of Machine Learning Research, 22(129):1–30, 2021. Chen et al. [2021] Richard J Chen, Ming Y Lu, Tiffany Y Chen, Drew FK Williamson, and Faisal Mahmood. Synthetic data in machine learning for medicine and healthcare. Nature Biomedical Engineering, 5(6):493–497, 2021. Cornacchia et al. [2023] Elisabetta Cornacchia, Francesca Mignacco, Rodrigo Veiga, Cédric Gerbelot, Bruno Loureiro, and Lenka Zdeborová. Learning curves for the multi-class teacher–student perceptron. Machine Learning: Science and Technology, 4(1):015019, 2023. Das and Sanghavi [2023] Rudrajit Das and Sujay Sanghavi. Understanding self-distillation in the presence of label noise. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 7102–7140. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/das23d.html. Dettmers et al. [2024] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024. Fan and Wang [2015] Jianqing Fan and Weichen Wang. Asymptotics of empirical eigen-structure for ultra-high dimensional spiked covariance model. arXiv preprint arXiv:1502.04733, 2015. Figueira and Vaz [2022] Alvaro Figueira and Bruno Vaz. Survey on synthetic data generation, evaluation methods and gans. Mathematics, 10(15):2733, 2022. Gou et al. [2021] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789–1819, 2021. Guo et al. [2024] Jianyuan Guo, Hanting Chen, Chengcheng Wang, Kai Han, Chang Xu, and Yunhe Wang. Vision superalignment: Weak-to-strong generalization for vision foundation models. arXiv preprint arXiv:2402.03749, 2024. Harutyunyan [2023] Hrayr Harutyunyan. On information captured by neural networks: connections with memorization and generalization, 2023. Hase et al. [2024] Peter Hase, Mohit Bansal, Peter Clark, and Sarah Wiegreffe. The unreasonable effectiveness of easy training data for hard tasks. arXiv preprint arXiv:2401.06751, 2024. He et al. [2024] Chunming He, Kai Li, Yachao Zhang, Guoxia Xu, Longxiang Tang, Yulun Zhang, Zhenhua Guo, and Xiu Li. Weakly-supervised concealed object segmentation with sam-based pseudo labeling and multi-scale feature grouping. Advances in Neural Information Processing Systems, 36, 2024. Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. Hong et al. [2024] Chi Hong, Robert Birke, Pin-Yu Chen, and Lydia Y. Chen. On dark knowledge for distilling generators. In De-Nian Yang, Xing Xie, Vincent S. Tseng, Jian Pei, Jen-Wei Huang, and Jerry Chun-Wei Lin, editors, Advances in Knowledge Discovery and Data Mining, pages 235–247, Singapore, 2024. Springer Nature Singapore. ISBN 978-981-97-2253-2. Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR, 2019. Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. Ji and Zhu [2020] Guangda Ji and Zhanxing Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. Advances in Neural Information Processing Systems, 33:20823–20833, 2020. Ji et al. [2024] Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak-to-strong correction. arXiv preprint arXiv:2402.02416, 2024. Ji and Telgarsky [2021] Ziwei Ji and Matus Telgarsky. Characterizing the implicit bias via a primal-dual analysis. In Algorithmic Learning Theory, pages 772–804. PMLR, 2021. Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. Lang et al. [2024] Hunter Lang, David Sontag, and Aravindan Vijayaraghavan. Theoretical analysis of weak-to-strong generalization. arXiv preprint arXiv:2405.16043, 2024. Lee et al. [2013] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896. Atlanta, 2013. Lester et al. [2021] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021. Liu et al. [2022] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965, 2022. Liu et al. [2024] Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. Best practices and lessons learned on synthetic data for language models. arXiv preprint arXiv:2404.07503, 2024. Liu and Alahi [2024] Yuejiang Liu and Alexandre Alahi. Co-supervised learning: Improving weak-to-strong generalization with hierarchical mixture of experts. arXiv preprint arXiv:2402.15505, 2024. Lopes and Yao [2022] Miles E Lopes and Junwen Yao. A sharp lower-tail bound for gaussian maxima with application to bootstrap methods in high dimensions. Electronic Journal of Statistics, 16(1):58–83, 2022. Marks and Tegmark [2023] Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023. Mei and Montanari [2022] Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667–766, 2022. Mobahi et al. [2020] Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351–3361, 2020. Muthukumar et al. [2020] Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpolation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, 1(1):67–83, 2020. Muthukumar et al. [2021] Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai. Classification vs regression in overparameterized regimes: Does the loss function matter? Journal of Machine Learning Research, 22(222):1–69, 2021. Nagarajan et al. [2023] Vaishnavh Nagarajan, Aditya K Menon, Srinadh Bhojanapalli, Hossein Mobahi, and Sanjiv Kumar. On student-teacher deviations in distillation: does it pay to disobey? In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 5961–6000. Curran Associates, Inc., 2023. URL https://proceedings.neurips.c/paper_files/paper/2023/file/12d286282e1be5431ea05262a21f415c-Paper-Conference.pdf. Nanda et al. [2023] Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941, 2023. Nikolenko [2021] Sergey I Nikolenko. Synthetic data for deep learning, volume 174. Springer, 2021. Ojha et al. [2023] Utkarsh Ojha, Yuheng Li, Anirudh Sundara Rajan, Yingyu Liang, and Yong Jae Lee. What knowledge gets distilled in knowledge distillation? In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 11037–11048. Curran Associates, Inc., 2023. URL https://proceedings.neurips.c/paper_files/paper/2023/file/2433fec2144ccf5fea1c9c5ebdbc3924-Paper-Conference.pdf. Phuong and Lampert [2019] Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In International conference on machine learning, pages 5142–5151. PMLR, 2019. Pinasco et al. [2021] Damián Pinasco, Ezequiel Smucler, and Ignacio Zalduendo. Orthant probabilities and the attainment of maxima on a vertex of a simplex. Linear Algebra and its Applications, 610:785–803, 2021. Rizve et al. [2021] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arXiv preprint arXiv:2101.06329, 2021. Safaryan et al. [2023] Mher Safaryan, Alexandra Peste, and Dan Alistarh. Knowledge distillation performs partial variance reduction. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 75229–75258. Curran Associates, Inc., 2023. URL https://proceedings.neurips.c/paper_files/paper/2023/file/e1f0da706829d7f198eac0edaacc338-Paper-Conference.pdf. Sarnthein et al. [2023] Felix Sarnthein, Gregor Bachmann, Sotiris Anagnostidis, and Thomas Hofmann. Random teachers are good teachers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 30022–30041. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/sarnthein23a.html. Schwarzschild et al. [2021] Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances in Neural Information Processing Systems, 34:6695–6706, 2021. Shen et al. [2013] Dan Shen, Haipeng Shen, Hongtu Zhu, and JS Marron. Surprising asymptotic conical structure in critical sample eigen-directions. arXiv preprint arXiv:1303.6171, 2013. Somerstep et al. [2024] Seamus Somerstep, Felipe Maia Polo, Moulinath Banerjee, Ya’acov Ritov, Mikhail Yurochkin, and Yuekai Sun. A statistical framework for weak-to-strong generalization. arXiv preprint arXiv:2405.16236, 2024. Stanton et al. [2021] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34:6906–6919, 2021. Subramanian et al. [2022] Vignesh Subramanian, Rahul Arya, and Anant Sahai. Generalization for multiclass classification with overparameterized linear models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=ikWvMRVQBWW. Tao and Li [2024] Leitian Tao and Yixuan Li. Your weak llm is secretly a strong teacher for alignment. arXiv preprint arXiv:2409.08813, 2024. Thrampoulidis [2024] Christos Thrampoulidis. Implicit bias of next-token prediction. arXiv preprint arXiv:2402.18551, 2024. Vershynin [2010] Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027, 2010. Wang et al. [2022a] Guillaume Wang, Konstantin Donhauser, and Fanny Yang. Tight bounds for minimum ℓ1subscriptℓ1 _1ℓ1-norm interpolation of noisy data. In International Conference on Artificial Intelligence and Statistics, pages 10572–10602. PMLR, 2022a. Wang and Thrampoulidis [2021] Ke Wang and Christos Thrampoulidis. Benign overfitting in binary classification of Gaussian mixtures. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4030–4034. IEEE, 2021. Wang et al. [2021] Ke Wang, Vidya Muthukumar, and Christos Thrampoulidis. Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation. arXiv e-prints, art. arXiv:2106.10865, June 2021. Wang et al. [2022b] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022b. Wei et al. [2022] Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models predict how real-world neural representations generalize. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23549–23588. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/wei22a.html. Wei et al. [2021] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. Wu and Sahai [2024] David Wu and Anant Sahai. Precise asymptotic generalization for multiclass classification with overparameterized linear models. Advances in Neural Information Processing Systems, 36, 2024. Xu et al. [2024] Liuchi Xu, Jin Ren, Zhenhua Huang, Weishi Zheng, and Yunwen Chen. Improving knowledge distillation via head and tail categories. IEEE Transactions on Circuits and Systems for Video Technology, 34(5):3465–3480, 2024. doi: 10.1109/TCSVT.2023.3325814. Yang et al. [2024] Wenkai Yang, Shiqi Shen, Guangyao Shen, Zhi Gong, and Yankai Lin. Super (ficial)-alignment: Strong models may deceive weak models in weak-to-strong generalization. arXiv preprint arXiv:2406.11431, 2024. Yuan et al. [2024] Hua Yuan, Yu Shi, Ning Xu, Xu Yang, Xin Geng, and Yong Rui. Learning from biased soft labels. Advances in Neural Information Processing Systems, 36, 2024. Zhang et al. [2021] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34:18408–18419, 2021. Zhang et al. [2024] Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin L Edelman, Milind Tambe, Sham M Kakade, and Eran Malach. Transcendence: Generative models can outperform the experts that train them. arXiv preprint arXiv:2406.11741, 2024. Zhao and Zhu [2023] Qingyue Zhao and Banghua Zhu. Towards the fundamental limits of knowledge transfer over finite domains, 2023. Zou et al. [2023] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023. Appendix A Preliminaries In this section, we lay out some preliminaries for the rigorous justification of our main results, and give a technical overview in Section 3.1 which motivates the study of the survival and contamination of the signal, defined formally in Definition 3. In Appendix B, we will introduce tools from high dimensional probability which can rigorously control the survival and contamination. In Appendix C, we will then use these tools to prove the main results of the paper for binary classification. Then, in Appendix D, we will discuss how the same exact proofs extend to the multilabel setting, and in Appendix E we will prove a tight lower tail inequality for correlated Gaussians which yields Theorem 3.1. In Appendix F we include some simulations validating in our theory, and finally in Appendix G we give a heuristic calculation which arrives at the same predictions for the weak-to-strong regimes which should lead to rigorous proofs even without the 1111-sparse assumption. Notation. For positive integers n, we use the shorthand [n]≜1,…,n≜delimited-[]1…[n] 1,…,n[ n ] ≜ start_ARG 1 , … , n end_ARG . For a vector ∈ℝnsuperscriptℝ v ^nitalic_v ∈ blackboard_Rn, ‖2subscriptnorm2 v_2∥ start_ARG italic_v end_ARG ∥2 always denotes the Euclidean norm. We index entries by using square brackets or subscripts, whichever is clearer, so ⁢[j]delimited-[] v[j]italic_v [ j ] and vjsubscriptv_jvitalic_j both denote the jjjth entry of vitalic_v. For any matrix ∈ℝm×nsuperscriptℝ M ^m× nitalic_M ∈ blackboard_Rm × n, we denote its i⁢jiji jth entry by mi⁢jsubscriptm_ijmitalic_i j, ‖2subscriptnorm2 M_2∥ start_ARG italic_M end_ARG ∥2 denotes the spectral norm, and ‖F=Tr⁡(⊤⁢)subscriptnormtracesuperscripttop M_F= ( M M)∥ start_ARG italic_M end_ARG ∥F = Tr ( start_ARG italic_M⊤ italic_M end_ARG ) denotes the Frobenius norm. If ∈ℝn×nsuperscriptℝ M ^n× nitalic_M ∈ blackboard_Rn × n is symmetric, we write μ1⁢()⩾μ2⁢()⩾…⩾μn⁢()subscript1subscript2…subscript _1( M) _2( M) … _n(% M)μ1 ( italic_M ) ⩾ μ2 ( italic_M ) ⩾ … ⩾ μitalic_n ( italic_M ) to denote the ordered eigenvalues of Mitalic_M. We make extensive use of asymptotic notation. We say f⁢(n)≍g⁢(n)asymptotically-equalsf(n) g(n)f ( n ) ≍ g ( n ) if f⁢(n)=Θ⁢(g⁢(n))Θf(n)= (g(n))f ( n ) = Θ ( g ( n ) ), f⁢(n)≲g⁢(n)less-than-or-similar-tof(n) g(n)f ( n ) ≲ g ( n ) if f⁢(n)=O⁢(g⁢(n))f(n)=O(g(n))f ( n ) = O ( g ( n ) ), and f⁢(n)=O~⁢(g⁢(n))~f(n)= O(g(n))f ( n ) = over~ start_ARG O end_ARG ( g ( n ) ) if f⁢(n)=O⁢(g⁢(n)⁢log⁡(n)c)superscriptf(n)=O(g(n) (n)^c)f ( n ) = O ( g ( n ) log ( start_ARG n end_ARG )c ) for some constant c⩾00c 0c ⩾ 0. When we write f⁢(n)≫g⁢(n)much-greater-thanf(n) g(n)f ( n ) ≫ g ( n ), we mean that f⁢(n)g⁢(n)⩾nεsuperscript f(n)g(n) n divide start_ARG f ( n ) end_ARG start_ARG g ( n ) end_ARG ⩾ nitalic_ε for some constant ε>00 >0ε > 0. An event ℰEE is said to hold with high probability if it holds with probability 1−1/nc11superscript1-1/n^c1 - 1 / nitalic_c for some constant c>00c>0c > 0, and very high probability if it holds with probability 1−exp⁡(−nc)1superscript1- (-n^c)1 - exp ( start_ARG - nitalic_c end_ARG ) for some constant c>00c>0c > 0. For us, high probability statements will be taken with respect to n, the number of datapoints we scale the weak and strong features with respect to, but under our terminology it makes no difference whether we use n or m because they are polynomially related. In the weak-to-strong ensemble analysis, many of the expressions will depend on q+rq+rq + r, but the number of datapoints involved will change. In an effort to reduce confusion, we introduce the following notation. Definition 4 (Bi-level prefactor). If the covariance Σ=Σ⁢(p,q,r)Σ = (p,q,r)Σ = Σ ( p , q , r ) is drawn from the bi-level ensemble scaled with respect to n, then we define μn≜a⁢ns=n1−q−r.≜subscriptsuperscript1 _n ans=n^1-q-r.μitalic_n ≜ divide start_ARG a n end_ARG start_ARG s end_ARG = n1 - q - r . (9) In particular, if q+r>11q+r>1q + r > 1 then μn≪1much-less-thansubscript1 _n 1μitalic_n ≪ 1. During weak supervision, the strong learner has bi-level features with covariance Σ⁢(p,q,r)Σ (p,q,r)Σ ( p , q , r ) scaled with respect to n and receives m=nusuperscriptm=n^um = nitalic_u weakly labeled examples. Hence, we define μm≜a⁢ms=nu−q−r.≜subscriptsuperscript _m ams=n^u-q-r.μitalic_m ≜ divide start_ARG a m end_ARG start_ARG s end_ARG = nitalic_u - q - r . (10) Remark A.1. Note that once a,sa,sa , s are fixed, μnsubscript _nμitalic_n is linear in n. Appendix B Tools for analyzing survival and contamination In this section, we introduce the basic machinery which can justify the heuristic calculations carried out in Appendix A. We will heavily rely on the analysis of Wu and Sahai [2024], so in an attempt to not replicate effort we will comment on how the proofs change in the weak-to-strong setting. Basis change. Since we have bi-level data and the true direction ∗subscript v_*italic_v∗ is 1111-sparse (Assumption 1), there are convenient basis changes which will greatly simplify the calculations. In particular, let S be the spiked subspace of Σ Σ, and witalic_w be any unit vector. Then any vectors in S are eigenvectors of Σ Σ, so we can pick an eigenbasis U∗subscriptU_*U∗ so that subscript x_ strongitalic_xsansserif_strong has independent coordinates, ∗subscript v_*italic_v∗ is 1-sparse, and witalic_w is 3-sparse. We state this more formally in the following definition. Definition 5 (Strong eigenbasis). Given Σ Σ, ∗subscript v_*italic_v∗, and witalic_w as above, let U∗subscriptU_*U∗ be the distinguished eigenbasis of Σ Σ such that after rotating to U∗subscriptU_*U∗, we have ∼N⁢(0,Λ)similar-tosubscript0Λ x_ strong N(0, )italic_xsansserif_strong ∼ N ( 0 , Λ ), ∗=1subscriptsubscript1 v_*= e_1italic_v∗ = italic_e1, and =w1⁢1+w2⁢2+ws+1⁢s+1subscript1subscript1subscript2subscript2subscript1subscript1 w=w_1 e_1+w_2 e_2+w_s+1 e_s+1italic_w = w1 italic_e1 + w2 italic_e2 + witalic_s + 1 italic_eitalic_s + 1. Hanson Wright inequality. Now, we recall a few versions of the Hanson-Wright inequality which handles bilinear forms between subgaussian vectors. For the multiclass case, we also need to handle potentially sparse vectors; this version was proved in Wu and Sahai [2024]. Theorem B.1 (Hanson-Wright for bilinear forms with soft sparsity). Let =(X1,…,Xn)∈ℝnsubscript1…subscriptsuperscriptℝ x=(X_1,…,X_n) ^nitalic_x = ( X1 , … , Xitalic_n ) ∈ blackboard_Rn and ∈(Y1,…,Yn)∈ℝnsubscript1…subscriptsuperscriptℝ y∈(Y_1,…,Y_n) ^nitalic_y ∈ ( Y1 , … , Yitalic_n ) ∈ blackboard_Rn be random vectors such that (Xi,Yi)subscriptsubscript(X_i,Y_i)( Xitalic_i , Yitalic_i ) are independent pairs of (possibly correlated) centered random variables such that ‖Xi‖ψ2⩽Ksubscriptnormsubscriptsubscript2 X_i_ _2 K∥ start_ARG Xitalic_i end_ARG ∥ψ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ⩽ K and YisubscriptY_iYitalic_i has soft sparsity at level σ, i.e. |Yi|⩽1subscript1 Y_i 1| start_ARG Yitalic_i end_ARG | ⩽ 1 almost surely, and ⁢[Yi2]⩽σdelimited-[]superscriptsubscript2E[Y_i^2] _E [ Yitalic_i2 ] ⩽ σ. Assume that conditioned on YjsubscriptY_jYitalic_j, ‖Xj‖ψ2⩽Ksubscriptnormsubscriptsubscript2 X_j_ _2 K∥ start_ARG Xitalic_j end_ARG ∥ψ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ⩽ K. Then there exists an absolute constant c>00c>0c > 0 such that for all ∈ℝn×nsuperscriptℝ M ^n× nitalic_M ∈ blackboard_Rn × n and ϵ⩾0italic-ϵ0ε 0ϵ ⩾ 0 we have ⁢[|⊤⁢−⁢[⊤⁢]|>ϵ]⩽2⁢exp⁡(−c⁢min⁡ϵ2K2⁢σ⁢‖F2,ϵK⁢‖2).superscripttopdelimited-[]superscripttopitalic-ϵ2superscriptitalic-ϵ2superscript2superscriptsubscriptnorm2italic-ϵsubscriptnorm2 Pr [| x M y-% E[ x M y]|>ε] 2 (-c % ε^2K^2σ M_F^2, % εK M_2).Pr [ start_ARG | italic_x⊤ italic_M italic_y - blackboard_E [ italic_x⊤ italic_M italic_y ] | > ϵ end_ARG ] ⩽ 2 exp ( start_ARG - c min start_ARG divide start_ARG ϵ2 end_ARG start_ARG K2 σ ∥ start_ARG italic_M end_ARG ∥F2 end_ARG , divide start_ARG ϵ end_ARG start_ARG K ∥ start_ARG italic_M end_ARG ∥2 end_ARG end_ARG end_ARG ) . (11) Because we are working with MNI, in a typical application we hope to set =−1superscript1 M= A^-1italic_M = italic_A- 1. However, in order to apply Hanson-Wright, one needs to ensure that the matrix Mitalic_M is independent of xitalic_x and yitalic_y. Decorrelated Gram matrix. As we are working with Gaussian covariates, one way to achieve this is to decorrelate Aitalic_A with xitalic_x and yitalic_y. Definition 6 (Decorrelated Gram matrix). Let 1,…,n∼N⁢(0,Λ)similar-tosubscript1…subscript0Λ x_1,…, x_n N(0, )italic_x1 , … , italic_xitalic_n ∼ N ( 0 , Λ ) be iid samples from a Gaussian distribution in ℝdsuperscriptℝR^dblackboard_Rd. Let R=(1),…,(ℓ)superscript1…superscriptℓR= v^(1),…, v^( )R = start_ARG italic_v( 1 ) , … , italic_v( ℓ ) end_ARG be an arbitrary set of orthonormal vectors in ℝdsuperscriptℝR^dblackboard_Rd and for each j∈[ℓ]delimited-[]ℓj∈[ ]j ∈ [ ℓ ], let (j)=(h1,…,hn)∈ℝnsuperscriptsubscriptℎ1…subscriptℎsuperscriptℝ h^(j)=(h_1,…,h_n) ^nitalic_h( j ) = ( h1 , … , hitalic_n ) ∈ blackboard_Rn be the corresponding Gaussian observations, where hi∼⟨Λ1/2⁢i,(j)⟩similar-tosubscriptℎexpectation-valuesuperscriptΛ12subscriptsuperscripth_i ^1/2 g_i, v^(j)hitalic_i ∼ ⟨ start_ARG Λ1 / 2 italic_gitalic_i , italic_v( j ) end_ARG ⟩. Define the projection matrix onto R by ΠR=∑∈R⁢⊤subscriptΠsubscriptsuperscripttop _R= _ v∈ R v v Πitalic_R = ∑italic_v ∈ R italic_v italic_v⊤. The R-projection of the data matrix R∈ℝn×dsubscriptsuperscriptℝ X_R ^n× ditalic_Xitalic_R ∈ blackboard_Rn × d is defined by R≜⁢ΠR,≜subscriptsubscriptΠ X_R X _R,italic_Xitalic_R ≜ italic_X Πitalic_R , and the R-decorrelated data matrix −R∈ℝn×dsubscriptsuperscriptℝ X_-R ^n× ditalic_X- R ∈ blackboard_Rn × d is defined by −R≜⁢(I−ΠR)=−R.≜subscriptsubscriptΠsubscript X_-R X(I- _R)= X- X_R.italic_X- R ≜ italic_X ( I - Πitalic_R ) = italic_X - italic_Xitalic_R . The decorrelated Gram matrix is defined by −R≜−R⁢−R⊤=⁢(I−ΠR)⁢⊤.≜subscriptsubscriptsuperscriptsubscripttopsubscriptΠsuperscripttop A_-R X_-R X_-R = X(I- _R)% X .italic_A- R ≜ italic_X- R italic_X- R⊤ = italic_X ( I - Πitalic_R ) italic_X⊤ . When R=R= vR = start_ARG italic_v end_ARG , we will simply abbreviate v start_ARG italic_v end_ARG with vitalic_v. Remark B.2. If R does not consist of orthonormal vectors, one can apply Gram-Schmidt first so that the above definition still makes sense. It is not hard to show that as defined above, −Rsubscript A_-Ritalic_A- R is mutually independent of (j)j∈[ℓ]subscriptsuperscriptdelimited-[]ℓ h^(j)_j∈[ ] start_ARG italic_h( j ) end_ARG j ∈ [ ℓ ], Fact B.3. In the same setting as Definition 6, −Rsubscript A_-Ritalic_A- R is mutually independent of (j)j∈[ℓ]subscriptsuperscriptdelimited-[]ℓ h^(j)_j∈[ ] start_ARG italic_h( j ) end_ARG j ∈ [ ℓ ]. Proof. The fact that −Rsubscript X_-Ritalic_X- R is independent of the hisubscriptℎh_ihitalic_i’s is due to the fact that if g,hℎg,hg , h are jointly Gaussian with zero mean and unit variance, then g−⁢[g⁢h]⁢hdelimited-[]ℎg- E[gh]hg - E [ g h ] h is independent of hℎh. This immediately generalizes to each row of Xitalic_X, since the entries are all independent for any row. Finally, as the rows are independent, the statement is proved. ∎ To relate a bilinear form against −R−1superscriptsubscript1 A_-R^-1italic_A- R- 1 to the original bilinear form against −1superscript1 A^-1italic_A- 1, we will also need the Woodbury inversion formula for inverting the Gram matrix after decorrelating it with various features. Define the hat matrix for R by Rsubscript H_Ritalic_Hitalic_R ≜R⊤⁢−R−1⁢R.≜absentsuperscriptsubscripttopsuperscriptsubscript1subscript X_R A_-R^-1 X_R.≜ italic_Xitalic_R⊤ italic_A- R- 1 italic_Xitalic_R . (12) These hat matrices appear in the Woodbury inversion formula. For the sake of notational compactness, define Rsubscript M_Ritalic_Mitalic_R ≜R⁢(I+R)−1⁢R⊤.≜absentsubscriptsuperscriptsubscript1superscriptsubscripttop X_R(I+ H_R)^-1 X_R \,.≜ italic_Xitalic_R ( I + italic_Hitalic_R )- 1 italic_Xitalic_R⊤ . (13) Fact B.4 (Woodbury inversion formula). We have −1superscript1 A^-1italic_A- 1 =(R⁢R⊤+−R)−1absentsuperscriptsubscriptsuperscriptsubscripttopsubscript1 =( X_R X_R + A_-R)^-1= ( italic_Xitalic_R italic_Xitalic_R⊤ + italic_A- R )- 1 =−R−1−R−1⁢R⁢(I+R)−1⁢R⊤⁢−R−1absentsuperscriptsubscript1superscriptsubscript1subscriptsuperscriptsubscript1superscriptsubscripttopsuperscriptsubscript1 = A_-R^-1- A_-R^-1 X_R(I+ H_R% )^-1 X_R A_-R^-1= italic_A- R- 1 - italic_A- R- 1 italic_Xitalic_R ( I + italic_Hitalic_R )- 1 italic_Xitalic_R⊤ italic_A- R- 1 (14) =−R−1−R−1⁢R⁢−R−1.absentsuperscriptsubscript1superscriptsubscript1subscriptsuperscriptsubscript1 = A_-R^-1- A_-R^-1 M_R A_-R^% -1.= italic_A- R- 1 - italic_A- R- 1 italic_Mitalic_R italic_A- R- 1 . (15) Concentration of spectrum. Finally, in view of Hanson-Wright, we will need to control ‖−R−1‖2subscriptnormsuperscriptsubscript12 A_-R^-1_2∥ start_ARG italic_A- R- 1 end_ARG ∥2 and Tr⁡(−R−1)tracesuperscriptsubscript1 ( A_-R^-1)Tr ( start_ARG italic_A- R- 1 end_ARG ). To this end, we control the spectrum of −Rsubscript A_-Ritalic_A- R. For any PSD matrix Σ Σ with eigenvalues Λ=diag(λi)i∈[d] =diag( _i)_i∈[d]Λ = diag ( λitalic_i )i ∈ [ d ] sorted in descending order and k∈[d]delimited-[]k∈[d]k ∈ [ d ], define the effective rank rk⁢(Σ)≜∑i⩾kλiλk≜subscriptΣsubscriptsubscriptsubscriptr_k( ) _i k _i _kritalic_k ( Σ ) ≜ divide start_ARG ∑i ⩾ k λitalic_i end_ARG start_ARG λitalic_k end_ARG. We will use the following bounds adapted from Bartlett et al. [2020], which show that the effective rank of the covariance matrix controls the spectrum of the Gram matrix (and hence the sample covariance). Lemma B.5 (Eigenvalue bounds from Bartlett et al. [2020]). Suppose that ∼N⁢(0,Σ)similar-to0Σ x N(0, )italic_x ∼ N ( 0 , Σ ), where Σ=U⁢Λ⁢U⊤ΣΛsuperscripttop =U U Σ = U Λ U⊤. Write Λ=diag⁡(λi)Λdiagsubscript =diag( _i)Λ = diag ( λitalic_i ), and let =⁢⊤∈ℝn×nsuperscripttopsuperscriptℝ A= X X ^n× nitalic_A = italic_X italic_X⊤ ∈ blackboard_Rn × n denote the Gram matrix for n iid observations of xitalic_x. If =U⊤⁢Λ1/2⁢superscripttopsuperscriptΛ12 x=U ^1/2 zitalic_x = U⊤ Λ1 / 2 italic_z, where zitalic_z has independent, unit variance, O⁢(1)1O(1)O ( 1 )-subgaussian coordinates, the following holds. If r1⁢(Σ)=ω⁢(n)subscript1Σr_1( )=ω(n)r1 ( Σ ) = ω ( n ), then with very high probability Tr⁡(Λ)⁢(1−o⁢(1))⩽μn⁢()⩽μ1⁢()⩽Tr⁡(Λ)⁢(1+o⁢(1))traceΛ11subscriptsubscript1traceΛ11 ( )(1-o(1)) _n( A) _1( A)% ( )(1+o(1))Tr ( start_ARG Λ end_ARG ) ( 1 - o ( 1 ) ) ⩽ μitalic_n ( italic_A ) ⩽ μ1 ( italic_A ) ⩽ Tr ( start_ARG Λ end_ARG ) ( 1 + o ( 1 ) ) As a corollary, we show that in the regime q+r>11q+r>1q + r > 1 in the bi-level ensemble, decorrelating the Gram matrix with any sublinear number of jointly Gaussian random variables will yield a flat matrix. Corollary B.6. Suppose that ∼N⁢(0,Σ)similar-to0Σ x N(0, )italic_x ∼ N ( 0 , Σ ), where Σ∈ℝd×dΣsuperscriptℝ ^d× dΣ ∈ blackboard_Rd × d satisfies μ1⁢(Σ)=o⁢(dn)subscript1Σ _1( )=o( dn)μ1 ( Σ ) = o ( divide start_ARG d end_ARG start_ARG n end_ARG ) and for any ℓ=o⁢(d)ℓ =o(d)ℓ = o ( d ), we have μℓ⁢(Σ)⁢rℓ⁢(Σ)=d⁢(1−o⁢(1))subscriptℓΣsubscriptℓΣ11 _ ( )r_ ( )=d(1-o(1))μroman_ℓ ( Σ ) rroman_ℓ ( Σ ) = d ( 1 - o ( 1 ) ) (in particular, this holds if Σ Σ follows the bi-level ensemble and q+r>11q+r>1q + r > 1). Let R be an arbitrary set of orthonormal vectors in ℝdsuperscriptℝR^dblackboard_Rd such that |R|=o⁢(d) R=o(d)| start_ARG R end_ARG | = o ( d ), and let −Rsubscript A_-Ritalic_A- R be the corresponding decorrelated Gram matrix for n iid observations of xitalic_x as in Definition 6. Then with very high probability, d⁢(1−on⁢(1))⩽μn⁢(−R)⩽μ1⁢(−R)⩽d⁢(1+on⁢(1))1subscript1subscriptsubscriptsubscript1subscript1subscript1d(1-o_n(1)) _n( A_-R) _1( A_-R)% d(1+o_n(1))d ( 1 - oitalic_n ( 1 ) ) ⩽ μitalic_n ( italic_A- R ) ⩽ μ1 ( italic_A- R ) ⩽ d ( 1 + oitalic_n ( 1 ) ) Proof. First, note that the rows of −Rsubscript X_-Ritalic_X- R are iid samples from N⁢(0,Σ−R)0subscriptΣN(0, _-R)N ( 0 , Σ- R ), where Σ−R=(I−ΠR)⁢Σ⁢(I−ΠR)subscriptΣsubscriptΠΣsubscriptΠ _-R=(I- _R) (I- _R)Σ- R = ( I - Πitalic_R ) Σ ( I - Πitalic_R ). Let ℓ=|R|ℓ = Rℓ = | start_ARG R end_ARG |; by assumption ℓ=o⁢(d)ℓ =o(d)ℓ = o ( d ). Since I−ΠRsubscriptΠI- _RI - Πitalic_R is a projection matrix, we conclude by Cauchy Interlacing that μ1⁢(Σ−R)⩽μ1⁢(Σ)subscript1subscriptΣsubscript1Σ _1( _-R) _1( )μ1 ( Σ- R ) ⩽ μ1 ( Σ ) and for i∈ℓ+1,…,dℓ1…i∈ +1,…,di ∈ start_ARG ℓ + 1 , … , d end_ARG , μi⁢(Σ)⩽μi⁢(Σ−R)subscriptΣsubscriptsubscriptΣ _i( ) _i( _-R)μitalic_i ( Σ ) ⩽ μitalic_i ( Σ- R ). It follows that r1⁢(Σ−R)=Tr⁡(Σ−R)μ1⁢(Σ−R)subscript1subscriptΣtracesubscriptΣsubscript1subscriptΣ r_1( _-R)= ( _-R) _1( _-R)r1 ( Σ- R ) = divide start_ARG Tr ( start_ARG Σ- R end_ARG ) end_ARG start_ARG μ1 ( Σ- R ) end_ARG ⩾μℓ⁢(Σ)⁢rℓ⁢(Σ)μ1⁢(Σ)absentsubscriptℓΣsubscriptℓΣsubscript1Σ _ ( )r_ ( ) _1( )⩾ divide start_ARG μroman_ℓ ( Σ ) rroman_ℓ ( Σ ) end_ARG start_ARG μ1 ( Σ ) end_ARG ⩾ω⁢(d⁢(1−o⁢(1))d/n)absent11 ω ( d(1-o(1))d/n)⩾ ω ( start_ARG divide start_ARG d ( 1 - o ( 1 ) ) end_ARG start_ARG d / n end_ARG end_ARG ) (Assumption on Σ Σ) ⩾ω⁢(n).absent ω(n).⩾ ω ( n ) . Thus Lemma B.5 implies that μi⁢(−R)=d⁢(1±o⁢(1))subscriptsubscriptplus-or-minus11 _i( A_-R)=d(1± o(1))μitalic_i ( italic_A- R ) = d ( 1 ± o ( 1 ) ) with high probability. ∎ This leads to the following bounds on the spectra of −R−1superscriptsubscript1 A_-R^-1italic_A- R- 1 and the hat matrix Rsubscript H_Ritalic_Hitalic_R; this generalizes Wu and Sahai [2024, Corollary B.5, Proposition B.6]. Lemma B.7. In the bi-level model, for any set of orthonormal vectors R with |R|=o⁢(d) R=o(d)| start_ARG R end_ARG | = o ( d ), if q+r>11q+r>1q + r > 1, then with very high probability we have Tr⁡(−R−1)tracesuperscriptsubscript1 ( A_-R^-1)Tr ( start_ARG italic_A- R- 1 end_ARG ) =n1−p⁢(1±o⁢(1))absentsuperscript1plus-or-minus11 =n^1-p(1± o(1))= n1 - p ( 1 ± o ( 1 ) ) ‖−R−1‖2subscriptnormsuperscriptsubscript12 A_-R^-1_2∥ start_ARG italic_A- R- 1 end_ARG ∥2 ⩽(1+o⁢(1))⁢n−pabsent11superscript (1+o(1))n^-p⩽ ( 1 + o ( 1 ) ) n- p ‖R⁢R⊤‖2subscriptnormsubscriptsuperscriptsubscripttop2 X_R X_R _2∥ start_ARG italic_Xitalic_R italic_Xitalic_R⊤ end_ARG ∥2 ⩽o⁢(np)absentsuperscript o(n^p)⩽ o ( nitalic_p ) μi⁢(R)subscriptsubscript _i( H_R)μitalic_i ( italic_Hitalic_R ) =1±o⁢(1),i∈[d]formulae-sequenceabsentplus-or-minus11delimited-[] =1± o(1), i∈[d]= 1 ± o ( 1 ) , i ∈ [ d ] Proof. The first two statements are an immediate consequence of Corollary B.6. For the third, let S,T⊆ℝdsuperscriptℝS,T ^dS , T ⊆ blackboard_Rd be the spiked and unspiked subspaces of Σ Σ. Since R=(S+T)⁢ΠRsubscriptsubscriptsubscriptsubscriptΠ X_R=( X_S+ X_T) _Ritalic_Xitalic_R = ( italic_Xitalic_S + italic_Xitalic_T ) Πitalic_R and using (A+B)⁢M⁢(A+B)⪯2⁢A⁢M⁢A+2⁢B⁢M⁢Bprecedes-or-equals22(A+B)M(A+B) 2AMA+2BMB( A + B ) M ( A + B ) ⪯ 2 A M A + 2 B M B, we have R⁢R⊤⪯2⁢S⁢S⊤+2⁢T⁢ΠR⁢T⊤.precedes-or-equalssubscriptsuperscriptsubscripttop2subscriptsuperscriptsubscripttop2subscriptsubscriptΠsuperscriptsubscripttop X_R X_R 2 X_S X_S^% +2 X_T _R X_T .italic_Xitalic_R italic_Xitalic_R⊤ ⪯ 2 italic_Xitalic_S italic_Xitalic_S⊤ + 2 italic_Xitalic_T Πitalic_R italic_Xitalic_T⊤ . Wu and Sahai [2024, Lemma I.1] implies that ‖S⁢S⊤‖2⩽n1−q−r⋅np=o⁢(np)subscriptnormsubscriptsuperscriptsubscripttop2⋅superscript1superscriptsuperscript X_S X_S _2 n^1-q-r· n^p=o(n^% p)∥ start_ARG italic_Xitalic_S italic_Xitalic_S⊤ end_ARG ∥2 ⩽ n1 - q - r ⋅ nitalic_p = o ( nitalic_p ) since q+r>11q+r>1q + r > 1. On the other hand, since the law of Tsubscript X_Titalic_Xitalic_T is rotationally invariant, one can view T⁢ΠR⁢T⊤subscriptsubscriptΠsuperscriptsubscripttop X_T _R X_T italic_Xitalic_T Πitalic_R italic_Xitalic_T⊤ as a Wishart matrix ⁢⊤superscripttop Y Y italic_Y italic_Y⊤ where ∈ℝn×|R|superscriptℝ Y ^n× Ritalic_Y ∈ blackboard_Rn × | start_ARG R end_ARG |. Hence the concentration of singular values for a matrix with independent subgaussian rows (see, e.g. Vershynin [2010, Theorem 5.39]) implies that with probability 1−e−n1superscript1-e^- n1 - e- square-root start_ARG n end_ARG, ‖T⁢ΠR⁢T⊤‖2⩽(|R|+n+n1/4)2=O⁢(R+n)=o⁢(np).subscriptnormsubscriptsubscriptΠsuperscriptsubscripttop2superscriptsuperscript142superscript X_T _R X_T _2 ( % R+ n+n^1/4)^2=O(R+n)=o(n^p).∥ start_ARG italic_Xitalic_T Πitalic_R italic_Xitalic_T⊤ end_ARG ∥2 ⩽ ( square-root start_ARG | start_ARG R end_ARG | end_ARG + square-root start_ARG n end_ARG + n1 / 4 )2 = O ( R + n ) = o ( nitalic_p ) . Combining this with the bound on ‖S⁢S⊤‖2subscriptnormsubscriptsuperscriptsubscripttop2 X_S X_S _2∥ start_ARG italic_Xitalic_S italic_Xitalic_S⊤ end_ARG ∥2, we conclude that ‖R⁢R⊤‖2=o⁢(np)subscriptnormsubscriptsuperscriptsubscripttop2superscript X_R X_R _2=o(n^p)∥ start_ARG italic_Xitalic_R italic_Xitalic_R⊤ end_ARG ∥2 = o ( nitalic_p ). For the last statement, since R=I+R⊤⁢−R−1⁢Rsubscriptsuperscriptsubscripttopsuperscriptsubscript1subscript H_R=I+ X_R A_-R^-1 X_Ritalic_Hitalic_R = I + italic_Xitalic_R⊤ italic_A- R- 1 italic_Xitalic_R, by applying the earlier bounds that μi⁢(−R−1)=np⁢(1±o⁢(1))subscriptsuperscriptsubscript1superscriptplus-or-minus11 _i( A_-R^-1)=n^p(1± o(1))μitalic_i ( italic_A- R- 1 ) = nitalic_p ( 1 ± o ( 1 ) ) and 0⪯R⊤⁢R⪯o⁢(np)precedes-or-equals0superscriptsubscripttopsubscriptprecedes-or-equalssuperscript0 X_R X_R o(n^p)0 ⪯ italic_Xitalic_R⊤ italic_Xitalic_R ⪯ o ( nitalic_p ), we conclude that μi⁢(R)=1±o⁢(1)subscriptsubscriptplus-or-minus11 _i( H_R)=1± o(1)μitalic_i ( italic_Hitalic_R ) = 1 ± o ( 1 ). ∎ Hanson-Wright calculations. Combining the above results, we can establish a concentration inequality for the bilinear form i⊤⁢−1⁢superscriptsubscripttopsuperscript1 z_i A^-1 yitalic_zitalic_i⊤ italic_A- 1 italic_y. Proposition B.8. Let =sgn⁢(⟨,⟩)sgnexpectation-value y=sgn( g, w)italic_y = sgn ( ⟨ start_ARG italic_g , italic_w end_ARG ⟩ ) for some unit vector ∈ℝdsuperscriptℝ w ^ditalic_w ∈ blackboard_Rd and R be any set of orthonormal vectors such that span⁢(R)⊇i,subscriptspanspan(R) e_i, wspan ( R ) ⊇ start_ARG italic_eitalic_i , italic_w end_ARG and |R|=o⁢(d) R=o(d)| start_ARG R end_ARG | = o ( d ). Then conditioned on −R−1superscriptsubscript1 A_-R^-1italic_A- R- 1, with probability 1−1n111- 1n1 - divide start_ARG 1 end_ARG start_ARG n end_ARG, |i⊤⁢−R−1⁢−⁢[i⊤⁢−R−1⁢]|⩽cB.8⁢‖−R−1‖F⁢log⁡n,superscriptsubscripttopsuperscriptsubscript1delimited-[]superscriptsubscripttopsuperscriptsubscript1subscriptB.8subscriptnormsuperscriptsubscript1 z_i A_-R^-1 y- % E[ z_i A_-R^-1 y] c_ const% _hw A_-R^-1_F n,| start_ARG italic_zitalic_i⊤ italic_A- R- 1 italic_y - E [ italic_zitalic_i⊤ italic_A- R- 1 italic_y ] end_ARG | ⩽ c ∥ start_ARG italic_A- R- 1 end_ARG ∥F square-root start_ARG log n end_ARG , where c1subscript1c_1c1 is a positive constant. Furthermore, under the bi-level scaling, if q+r>11q+r>1q + r > 1, then with very high probability over −R−1superscriptsubscript1 A_-R^-1italic_A- R- 1 we have ⁢[i⊤⁢−R−1⁢|−R−1]delimited-[]conditionalsuperscriptsubscripttopsuperscriptsubscript1superscriptsubscript1 E[ z_i A_-R^-1 y| A% _-R^-1]E [ italic_zitalic_i⊤ italic_A- R- 1 italic_y | italic_A- R- 1 ] =(2π)3/2⁢(1±on⁢(1))⁢arcsin⁡(wi)⋅nd,absent⋅superscript232plus-or-minus1subscript1arcsinesubscript = ( 2π)^3/2(1± o_n(1)) (w_i)·% nd,= ( start_ARG divide start_ARG 2 end_ARG start_ARG π end_ARG end_ARG )3 / 2 ( 1 ± oitalic_n ( 1 ) ) arcsin ( start_ARG witalic_i end_ARG ) ⋅ divide start_ARG n end_ARG start_ARG d end_ARG , ‖−R−1‖Fsubscriptnormsuperscriptsubscript1 A_-R^-1_F∥ start_ARG italic_A- R- 1 end_ARG ∥F =(1±on⁢(1))⁢nd.absentplus-or-minus1subscript1 =(1± o_n(1)) nd.= ( 1 ± oitalic_n ( 1 ) ) divide start_ARG square-root start_ARG n end_ARG end_ARG start_ARG d end_ARG . Proof. Since −R−1superscriptsubscript1 A_-R^-1italic_A- R- 1 is independent of (i,)subscript( z_i, y)( italic_zitalic_i , italic_y ), we can apply Theorem B.1 to obtain the first statement. For the other two expressions, we can apply Lemma B.7 since we assumed q+r>11q+r>1q + r > 1. We now compute the expectation using the noise stability formula: ⁢[i⊤⁢−R−1⁢|−R−1]delimited-[]conditionalsuperscriptsubscripttopsuperscriptsubscript1superscriptsubscript1 E[ z_i A_-R^-1 y| A% _-R^-1]E [ italic_zitalic_i⊤ italic_A- R- 1 italic_y | italic_A- R- 1 ] =Tr⁡(−R−1⁢[⁢i⊤])absenttracesuperscriptsubscript1delimited-[]superscriptsubscripttop = ( A_-R^-1 E[ y z_i ])= Tr ( start_ARG italic_A- R- 1 E [ italic_y italic_zitalic_i⊤ ] end_ARG ) =(2π)3/2⁢arcsin⁡(wi)⁢Tr⁡(−R−1)absentsuperscript232arcsinesubscripttracesuperscriptsubscript1 = ( 2π)^3/2 (w_i) ( A_-R^-% 1)= ( start_ARG divide start_ARG 2 end_ARG start_ARG π end_ARG end_ARG )3 / 2 arcsin ( start_ARG witalic_i end_ARG ) Tr ( start_ARG italic_A- R- 1 end_ARG ) (Noise stability) =(2π)3/2⁢(1±on⁢(1))⁢arcsin⁡(wi)⋅nd,absent⋅superscript232plus-or-minus1subscript1arcsinesubscript = ( 2π)^3/2(1± o_n(1)) (w_i)·% nd,= ( start_ARG divide start_ARG 2 end_ARG start_ARG π end_ARG end_ARG )3 / 2 ( 1 ± oitalic_n ( 1 ) ) arcsin ( start_ARG witalic_i end_ARG ) ⋅ divide start_ARG n end_ARG start_ARG d end_ARG , (Lemma B.7) where the last line holds with very high probability over −R−1superscriptsubscript1 A_-R^-1italic_A- R- 1. Finally, for the last statement we have ‖−R−1‖Fsubscriptnormsuperscriptsubscript1 A_-R^-1_F∥ start_ARG italic_A- R- 1 end_ARG ∥F =∑i∈[n]μi⁢(−R−1)2absentsubscriptdelimited-[]subscriptsuperscriptsuperscriptsubscript12 = _i∈[n] _i( A_-R^-1)^2= square-root start_ARG ∑i ∈ [ n ] μitalic_i ( italic_A- R- 1 )2 end_ARG =∑i∈[n]d2⁢(1±o⁢(1))absentsubscriptdelimited-[]superscript2plus-or-minus11 = _i∈[n]d^2(1± o(1))= square-root start_ARG ∑i ∈ [ n ] d2 ( 1 ± o ( 1 ) ) end_ARG (Lemma B.7) =(1±o⁢(1))⁢nd.absentplus-or-minus11 =(1± o(1)) nd.= ( 1 ± o ( 1 ) ) divide start_ARG square-root start_ARG n end_ARG end_ARG start_ARG d end_ARG . ∎ B.1 Controlling the error from Woodbury In the previous subsection, we introduced some tools for obtaining concentration of the coefficients of the MNI classifier subscript f_ strongitalic_fsansserif_strong. The remaining survival and contamination reduce to understanding the typical behavior of bilinear forms i⊤⁢−1⁢superscriptsubscripttopsuperscript1 z_i A^-1 yitalic_zitalic_i⊤ italic_A- 1 italic_y. To do so, we work in the distinguished basis U∗subscriptU_*U∗ introduced in Definition 5. We can thus pick R=1,2,s+1subscript1subscript2subscript1R= e_1, e_2, e_s+1R = start_ARG italic_e1 , italic_e2 , italic_eitalic_s + 1 end_ARG and apply Hanson-Wright to the bilinear form i⊤⁢−R−1⁢superscriptsubscripttopsuperscriptsubscript1 z_i A_-R^-1 yitalic_zitalic_i⊤ italic_A- R- 1 italic_y. The next lemma bounds the error from replacing −1superscript1 A^-1italic_A- 1 with −R−1superscriptsubscript1 A_-R^-1italic_A- R- 1. Lemma B.9. Suppose we are in the bi-level model, =sgn⁢(⟨,⟩)sgnexpectation-value y=sgn( g, w)italic_y = sgn ( ⟨ start_ARG italic_g , italic_w end_ARG ⟩ ), and q+r>11q+r>1q + r > 1. Suppose in the basis U∗subscriptU_*U∗, =w1⁢1+w2⁢2+ws+1⁢s+1subscript1subscript1subscript2subscript2subscript1subscript1 w=w_1 e_1+w_2 e_2+w_s+1 e_s+1italic_w = w1 italic_e1 + w2 italic_e2 + witalic_s + 1 italic_eitalic_s + 1. Let R=1,2,s+1subscript1subscript2subscript1R= e_1, e_2, e_s+1R = start_ARG italic_e1 , italic_e2 , italic_eitalic_s + 1 end_ARG . With probability 1−O⁢(1/n)111-O(1/n)1 - O ( 1 / n ), we have uniformly over i∈[d]delimited-[]i∈[d]i ∈ [ d ] that |i⊤⁢−1⁢−i⊤⁢−R−1⁢|≲‖i⊤⁢−R−1⁢R‖2⁢‖R⊤⁢−R−1⁢‖2.less-than-or-similar-tosuperscriptsubscripttopsuperscript1superscriptsubscripttopsuperscriptsubscript1subscriptnormsuperscriptsubscripttopsuperscriptsubscript1subscript2subscriptnormsuperscriptsubscripttopsuperscriptsubscript12 z_i A^-1 y- z_i^% A_-R^-1 y z_i A_-R^-1% X_R_2 X_R A_-R^-1 y_2.| start_ARG italic_zitalic_i⊤ italic_A- 1 italic_y - italic_zitalic_i⊤ italic_A- R- 1 italic_y end_ARG | ≲ ∥ start_ARG italic_zitalic_i⊤ italic_A- R- 1 italic_Xitalic_R end_ARG ∥2 ∥ start_ARG italic_Xitalic_R⊤ italic_A- R- 1 italic_y end_ARG ∥2 . Furthermore, in the bi-level scaling we have with probability 1−O⁢(1/n)111-O(1/n)1 - O ( 1 / n ) that ‖i⊤⁢−R−1⁢R‖2subscriptnormsuperscriptsubscripttopsuperscriptsubscript1subscript2 z_i A_-R^-1 X_R_2∥ start_ARG italic_zitalic_i⊤ italic_A- R- 1 italic_Xitalic_R end_ARG ∥2 =O~⁢(μn⁢n1−p2)i∈[2]O~⁢(μn⁢n−p2+n1−p)i=s+1O~⁢(μn⁢n−p2)otherwiseabsentcases~subscriptsuperscript12delimited-[]2~subscriptsuperscript2superscript11~subscriptsuperscript2otherwise = cases* O( _nn 1-p2)&$i% ∈[2]$\\ O( _nn^- p2+n^1-p)&$i=s+1$\\ O( _nn^- p2)&otherwise cases*= start_ROW start_CELL over~ start_ARG O end_ARG ( square-root start_ARG μitalic_n end_ARG ndivide start_ARG 1 - p end_ARG start_ARG 2 end_ARG ) end_CELL start_CELL i ∈ [ 2 ] end_CELL end_ROW start_ROW start_CELL over~ start_ARG O end_ARG ( square-root start_ARG μitalic_n end_ARG n- divide start_ARG p end_ARG start_ARG 2 end_ARG + n1 - p ) end_CELL start_CELL i = s + 1 end_CELL end_ROW start_ROW start_CELL over~ start_ARG O end_ARG ( square-root start_ARG μitalic_n end_ARG n- divide start_ARG p end_ARG start_ARG 2 end_ARG ) end_CELL start_CELL otherwise end_CELL end_ROW and ‖R⊤⁢−R−1⁢‖2subscriptnormsuperscriptsubscripttopsuperscriptsubscript12 X_R A_-R^-1 y_2∥ start_ARG italic_Xitalic_R⊤ italic_A- R- 1 italic_y end_ARG ∥2 ≲μn⁢((|w1|+|w2|)⁢n1−p2+O~⁢(n−p2))+|ws+1|⁢n1−p+O~⁢(n12−p).less-than-or-similar-toabsentsubscriptsubscript1subscript2superscript12~superscript2subscript1superscript1~superscript12 _n(( w_1+ w_2% )n 1-p2+ O(n^- p2))+ w_s+1n% ^1-p+ O(n 12-p).≲ square-root start_ARG μitalic_n end_ARG ( ( | start_ARG w1 end_ARG | + | start_ARG w2 end_ARG | ) ndivide start_ARG 1 - p end_ARG start_ARG 2 end_ARG + over~ start_ARG O end_ARG ( n- divide start_ARG p end_ARG start_ARG 2 end_ARG ) ) + | start_ARG witalic_s + 1 end_ARG | n1 - p + over~ start_ARG O end_ARG ( ndivide start_ARG 1 end_ARG start_ARG 2 end_ARG - p ) . Proof. By relating these two bilinear forms using Woodbury, we have |i⊤⁢−1⁢−i⊤⁢−R−1⁢|superscriptsubscripttopsuperscript1superscriptsubscripttopsuperscriptsubscript1 z_i A^-1 y- z_i^% A_-R^-1 y| start_ARG italic_zitalic_i⊤ italic_A- 1 italic_y - italic_zitalic_i⊤ italic_A- R- 1 italic_y end_ARG | =|i⊤⁢−R−1⁢R⁢−R−1⁢|absentsuperscriptsubscripttopsuperscriptsubscript1subscriptsuperscriptsubscript1 = z_i A_-R^-1 M_R% A_-R^-1 y= | start_ARG italic_zitalic_i⊤ italic_A- R- 1 italic_Mitalic_R italic_A- R- 1 italic_y end_ARG | (B.4) ⩽‖R‖2⁢‖i⊤⁢−R−1⁢R‖2⁢‖R⊤⁢−R−1⁢‖2absentsubscriptnormsubscript2subscriptnormsuperscriptsubscripttopsuperscriptsubscript1subscript2subscriptnormsuperscriptsubscripttopsuperscriptsubscript12 H_R_2 z_i A_% -R^-1 X_R_2 X_R A_-R^-1 y_2⩽ ∥ start_ARG italic_Hitalic_R end_ARG ∥2 ∥ start_ARG italic_zitalic_i⊤ italic_A- R- 1 italic_Xitalic_R end_ARG ∥2 ∥ start_ARG italic_Xitalic_R⊤ italic_A- R- 1 italic_y end_ARG ∥2 (Cauchy-Schwarz) ⩽(1+o⁢(1))⁢‖i⊤⁢−R−1⁢R‖2⁢‖R⊤⁢−R−1⁢‖2.absent11subscriptnormsuperscriptsubscripttopsuperscriptsubscript1subscript2subscriptnormsuperscriptsubscripttopsuperscriptsubscript12 (1+o(1)) z_i A_-R^-1 X% _R_2 X_R A_-R^-1 y_2\,.⩽ ( 1 + o ( 1 ) ) ∥ start_ARG italic_zitalic_i⊤ italic_A- R- 1 italic_Xitalic_R end_ARG ∥2 ∥ start_ARG italic_Xitalic_R⊤ italic_A- R- 1 italic_y end_ARG ∥2 . (Lemma B.7) It remains to control the above norms, which we achieve through another application of Hanson-Wright. Indeed, we can compute ‖i⊤⁢−R−1⁢R‖22superscriptsubscriptnormsuperscriptsubscripttopsuperscriptsubscript1subscript22 z_i A_-R^-1 X_R_2^2∥ start_ARG italic_zitalic_i⊤ italic_A- R- 1 italic_Xitalic_R end_ARG ∥22 =∑j∈1,2,s+1λj⁢(i⊤⁢−R−1⁢j)2absentsubscript121subscriptsuperscriptsuperscriptsubscripttopsuperscriptsubscript1subscript2 = _j∈ 1,2,s+1 _j( z_i A% _-R^-1 z_j)^2= ∑j ∈ start_ARG 1 , 2 , s + 1 end_ARG λitalic_j ( italic_zitalic_i⊤ italic_A- R- 1 italic_zitalic_j )2 Now, by Hanson-Wright, we have with probability 1−O⁢(1n)111-O( 1n)1 - O ( divide start_ARG 1 end_ARG start_ARG n end_ARG ) that λU⁢(i⊤⁢−R−1⁢s+1)2=O~⁢(n2−2⁢p)i=s+1O~⁢(n1−2⁢p)otherwisesubscriptsuperscriptsuperscriptsubscripttopsuperscriptsubscript1subscript12cases~superscript221~superscript12otherwise _U( z_i A_-R^-1 z_s+1)^2=% cases* O(n^2-2p)&$i=s+1$\\ O(n^1-2p)&otherwise cases*λitalic_U ( italic_zitalic_i⊤ italic_A- R- 1 italic_zitalic_s + 1 )2 = start_ROW start_CELL over~ start_ARG O end_ARG ( n2 - 2 p ) end_CELL start_CELL i = s + 1 end_CELL end_ROW start_ROW start_CELL over~ start_ARG O end_ARG ( n1 - 2 p ) end_CELL start_CELL otherwise end_CELL end_ROW Similarly we have with probability 1−O⁢(1n)111-O( 1n)1 - O ( divide start_ARG 1 end_ARG start_ARG n end_ARG ) that λF⁢∑j∈[2](i⊤⁢−R−1⁢j)2=O~⁢(np−q−r⋅n2−2⁢p)i∈[2]O~⁢(np−q−r⋅n1−2⁢p)otherwisesubscriptsubscriptdelimited-[]2superscriptsuperscriptsubscripttopsuperscriptsubscript1subscript2cases~⋅superscriptsuperscript22delimited-[]2~⋅superscriptsuperscript12otherwise _F _j∈[2]( z_i A_-R^-1 % z_j)^2= cases* O(n^p-q-r· n^2-2p)&$i∈[2]$\\ O(n^p-q-r· n^1-2p)&otherwise cases*λitalic_F ∑j ∈ [ 2 ] ( italic_zitalic_i⊤ italic_A- R- 1 italic_zitalic_j )2 = start_ROW start_CELL over~ start_ARG O end_ARG ( nitalic_p - q - r ⋅ n2 - 2 p ) end_CELL start_CELL i ∈ [ 2 ] end_CELL end_ROW start_ROW start_CELL over~ start_ARG O end_ARG ( nitalic_p - q - r ⋅ n1 - 2 p ) end_CELL start_CELL otherwise end_CELL end_ROW Since np−q−r≫1much-greater-thansuperscript1n^p-q-r 1nitalic_p - q - r ≫ 1 as p>q+rp>q+rp > q + r and np−q−r⋅n1−p=μn⋅superscriptsuperscript1subscriptn^p-q-r· n^1-p= _nnitalic_p - q - r ⋅ n1 - p = μitalic_n, it follows that with probability 1−O⁢(1n)111-O( 1n)1 - O ( divide start_ARG 1 end_ARG start_ARG n end_ARG ) we have simultaneously for i∈[d]delimited-[]i∈[d]i ∈ [ d ] that ‖i⊤⁢−R−1⁢R‖2subscriptnormsuperscriptsubscripttopsuperscriptsubscript1subscript2 z_i A_-R^-1 X_R_2∥ start_ARG italic_zitalic_i⊤ italic_A- R- 1 italic_Xitalic_R end_ARG ∥2 =O~⁢(μn⁢n1−p2)i∈[2]O~⁢(μn⁢n−p2+n1−p)i=s+1O~⁢(μn⁢n−p2)otherwiseabsentcases~subscriptsuperscript12delimited-[]2~subscriptsuperscript2superscript11~subscriptsuperscript2otherwise = cases* O( _nn 1-p2)&$i% ∈[2]$\\ O( _nn^- p2+n^1-p)&$i=s+1$\\ O( _nn^- p2)&otherwise cases*= start_ROW start_CELL over~ start_ARG O end_ARG ( square-root start_ARG μitalic_n end_ARG ndivide start_ARG 1 - p end_ARG start_ARG 2 end_ARG ) end_CELL start_CELL i ∈ [ 2 ] end_CELL end_ROW start_ROW start_CELL over~ start_ARG O end_ARG ( square-root start_ARG μitalic_n end_ARG n- divide start_ARG p end_ARG start_ARG 2 end_ARG + n1 - p ) end_CELL start_CELL i = s + 1 end_CELL end_ROW start_ROW start_CELL over~ start_ARG O end_ARG ( square-root start_ARG μitalic_n end_ARG n- divide start_ARG p end_ARG start_ARG 2 end_ARG ) end_CELL start_CELL otherwise end_CELL end_ROW A similar calculation, applying Proposition B.8, yields that (i⊤⁢−R−1⁢)2=1d2⁢[(2π)3/2⁢arcsin⁡(wi)⁢n±O~⁢(n)]2i∈1,2,s+1O~⁢(nd2)otherwisesuperscriptsuperscriptsubscripttopsuperscriptsubscript12cases1superscript2superscriptplus-or-minussuperscript232arcsinesubscript~2121~superscript2otherwise ( z_i A_-R^-1 y)^2= cases*% 1d^2 [ ( 2π)^3/2 (w_i)n±% O( n)]^2&$i∈ 1,2,s+1$\\ O( nd^2)&otherwise cases*( italic_zitalic_i⊤ italic_A- R- 1 italic_y )2 = start_ROW start_CELL divide start_ARG 1 end_ARG start_ARG d2 end_ARG [ start_ARG ( start_ARG divide start_ARG 2 end_ARG start_ARG π end_ARG end_ARG )3 / 2 arcsin ( start_ARG witalic_i end_ARG ) n ± over~ start_ARG O end_ARG ( square-root start_ARG n end_ARG ) end_ARG ]2 end_CELL start_CELL i ∈ start_ARG 1 , 2 , s + 1 end_ARG end_CELL end_ROW start_ROW start_CELL over~ start_ARG O end_ARG ( divide start_ARG n end_ARG start_ARG d2 end_ARG ) end_CELL start_CELL otherwise end_CELL end_ROW Hence, ‖R⊤⁢−R−1⁢‖2subscriptnormsuperscriptsubscripttopsuperscriptsubscript12 X_R A_-R^-1 y_2∥ start_ARG italic_Xitalic_R⊤ italic_A- R- 1 italic_y end_ARG ∥2 ≲∑i∈1,2,s+1λid⁢(|wi|⁢n+O~⁢(n))less-than-or-similar-toabsentsubscript121subscriptsubscript~ _i∈ 1,2,s+1 _id% ( w_in+ O( n))≲ ∑i ∈ start_ARG 1 , 2 , s + 1 end_ARG divide start_ARG square-root start_ARG λitalic_i end_ARG end_ARG start_ARG d end_ARG ( | start_ARG witalic_i end_ARG | n + over~ start_ARG O end_ARG ( square-root start_ARG n end_ARG ) ) ≲μn⁢(‖S‖1⁢n1−p2+O~⁢(n−p2))+|ws+1|⁢n1−p+O~⁢(n12−p).less-than-or-similar-toabsentsubscriptsubscriptnormsubscript1superscript12~superscript2subscript1superscript1~superscript12 _n( w_S_1n 1-p2+% O(n^- p2))+ w_s+1n^1-p+ O(% n 12-p).≲ square-root start_ARG μitalic_n end_ARG ( ∥ start_ARG italic_witalic_S end_ARG ∥1 ndivide start_ARG 1 - p end_ARG start_ARG 2 end_ARG + over~ start_ARG O end_ARG ( n- divide start_ARG p end_ARG start_ARG 2 end_ARG ) ) + | start_ARG witalic_s + 1 end_ARG | n1 - p + over~ start_ARG O end_ARG ( ndivide start_ARG 1 end_ARG start_ARG 2 end_ARG - p ) . This concludes the proof. ∎ We can simplify the above error bound greatly if witalic_w satisfies the 1-sparse assumption. Corollary B.10 (Woodbury error under 1111-sparse assumption). Suppose, in addition to the assumptions of Lemma B.9, we have =∗subscript w= v_*italic_w = italic_v∗, where ∗subscript v_*italic_v∗ satisfies the 1-sparse assumption. In other words, in the U∗subscriptU_*U∗ basis we have ∗=1subscriptsubscript1 v_*= e_1italic_v∗ = italic_e1. Then we pick R=1subscript1R= e_1R = start_ARG italic_e1 end_ARG , and with probability 1−O⁢(1n)111-O( 1n)1 - O ( divide start_ARG 1 end_ARG start_ARG n end_ARG ) we have |i⊤⁢−1⁢−i⊤⁢−R−1⁢|⩽O~⁢(μn⋅n1−p)i=1O~⁢(μn⋅n12−p)otherwisesuperscriptsubscripttopsuperscript1superscriptsubscripttopsuperscriptsubscript1cases~⋅subscriptsuperscript11~⋅subscriptsuperscript12otherwise z_i A^-1 y- z_i^% A_-R^-1 y cases* O( _n% · n^1-p)&$i=1$\\ O( _n· n 12-p)&otherwise cases*| start_ARG italic_zitalic_i⊤ italic_A- 1 italic_y - italic_zitalic_i⊤ italic_A- R- 1 italic_y end_ARG | ⩽ start_ROW start_CELL over~ start_ARG O end_ARG ( μitalic_n ⋅ n1 - p ) end_CELL start_CELL i = 1 end_CELL end_ROW start_ROW start_CELL over~ start_ARG O end_ARG ( μitalic_n ⋅ ndivide start_ARG 1 end_ARG start_ARG 2 end_ARG - p ) end_CELL start_CELL otherwise end_CELL end_ROW B.2 Survival analysis Let us first establish a general survival bound using Lemma B.9. Afterward, we present special cases under additional assumptions. Proposition B.11. Let Σ=Σ⁢(p,q,r)Σ = (p,q,r)Σ = Σ ( p , q , r ) be drawn from the bi-level ensemble scaled with respect to n with q+r>11q+r>1q + r > 1. Suppose in the distinguished basis U∗subscriptU_*U∗, we have =w1⁢1+w2⁢2+ws+1⁢s+1subscript1subscript1subscript2subscript2subscript1subscript1 w=w_1 e_1+w_2 e_2+w_s+1 e_s+1italic_w = w1 italic_e1 + w2 italic_e2 + witalic_s + 1 italic_eitalic_s + 1, and set R=1,2,s+1subscript1subscript2subscript1R= e_1, e_2, e_s+1R = start_ARG italic_e1 , italic_e2 , italic_eitalic_s + 1 end_ARG . Then, given n labels generated by witalic_w, we have |1⊤⁢−1⁢−(2π)3/2⋅nd⁢(1+o⁢(1))⁢arcsin⁡w1|superscriptsubscript1topsuperscript1⋅superscript23211arcsinesubscript1 z_1 A^-1 y- ( % 2π)^3/2· nd(1+o(1)) w_1| start_ARG italic_z1⊤ italic_A- 1 italic_y - ( start_ARG divide start_ARG 2 end_ARG start_ARG π end_ARG end_ARG )3 / 2 ⋅ divide start_ARG n end_ARG start_ARG d end_ARG ( 1 + o ( 1 ) ) arcsin w1 end_ARG | ≲O~⁢(nd)+‖i⊤⁢−R−1⁢R‖2⁢‖R⊤⁢−R−1⁢‖2.less-than-or-similar-toabsent~subscriptnormsuperscriptsubscripttopsuperscriptsubscript1subscript2subscriptnormsuperscriptsubscripttopsuperscriptsubscript12 O ( nd)+ z_% i A_-R^-1 X_R_2 X_R A_% -R^-1 y_2.≲ over~ start_ARG O end_ARG ( start_ARG divide start_ARG square-root start_ARG n end_ARG end_ARG start_ARG d end_ARG end_ARG ) + ∥ start_ARG italic_zitalic_i⊤ italic_A- R- 1 italic_Xitalic_R end_ARG ∥2 ∥ start_ARG italic_Xitalic_R⊤ italic_A- R- 1 italic_y end_ARG ∥2 . Proof. Recall our expressions for the survival: SUn⁢(|)subscriptSUnconditional SU_n( v| w)SUn ( v | w ) =∑i∈[d]λi⁢i⊤⁢−1⁢⟨i,⟩,absentsubscriptdelimited-[]subscriptsuperscriptsubscripttopsuperscript1expectation-valuesubscript = _i∈[d] _i z_i A^-1 y% v_i, v,= ∑i ∈ [ d ] λitalic_i italic_zitalic_i⊤ italic_A- 1 italic_y ⟨ start_ARG italic_vitalic_i , italic_v end_ARG ⟩ , (MNI) where each label is generated by yi=sgn⁢(⟨i,⟩)subscriptsgnexpectation-valuesubscripty_i=sgn( g_i, w)yitalic_i = sgn ( ⟨ start_ARG italic_gitalic_i , italic_w end_ARG ⟩ ). Setting =∗subscript v= v_*italic_v = italic_v∗, in the basis U∗subscriptU_*U∗ from Definition 5, the above expression simplifies to SUn⁢(∗|)subscriptSUnconditionalsubscript SU_n( v_*| w)SUn ( v∗ | w ) =λ1⁢1⊤⁢−1⁢absentsubscript1superscriptsubscript1topsuperscript1 = _1 z_1 A^-1 y= λ1 italic_z1⊤ italic_A- 1 italic_y Applying Woodbury (B.4) with R=,R= v, wR = start_ARG italic_v , italic_w end_ARG , we have 1⊤⁢−1⁢=1⊤⁢−R−1⁢−1⊤⁢−R−1⁢R⁢−R−1⁢.superscriptsubscript1topsuperscript1superscriptsubscript1topsuperscriptsubscript1superscriptsubscript1topsuperscriptsubscript1subscriptsuperscriptsubscript1 z_1 A^-1 y= z_1 A_-R% ^-1 y- z_1 A_-R^-1 M_R A_-R^-1% y.italic_z1⊤ italic_A- 1 italic_y = italic_z1⊤ italic_A- R- 1 italic_y - italic_z1⊤ italic_A- R- 1 italic_Mitalic_R italic_A- R- 1 italic_y . Now we can apply Proposition B.8 to conclude that |1⊤⁢−R−1⁢−(2π)3/2⋅nd⁢(1+o⁢(1))⁢arcsin⁡(w1)|⩽O~⁢(nd)superscriptsubscript1topsuperscriptsubscript1⋅superscript23211arcsinesubscript1~ z_1 A_-R^-1 y-% ( 2π)^3/2· nd(1+o(1)) (w_1) % O ( nd)| start_ARG italic_z1⊤ italic_A- R- 1 italic_y - ( start_ARG divide start_ARG 2 end_ARG start_ARG π end_ARG end_ARG )3 / 2 ⋅ divide start_ARG n end_ARG start_ARG d end_ARG ( 1 + o ( 1 ) ) arcsin ( start_ARG w1 end_ARG ) end_ARG | ⩽ over~ start_ARG O end_ARG ( start_ARG divide start_ARG square-root start_ARG n end_ARG end_ARG start_ARG d end_ARG end_ARG ) with probability 1−1n111- 1n1 - divide start_ARG 1 end_ARG start_ARG n end_ARG. Now, triangle inequality and Lemma B.9 shows that |1⊤⁢−1⁢−(2π)3/2⋅nd⁢(1+o⁢(1))⁢arcsin⁡w1|superscriptsubscript1topsuperscript1⋅superscript23211arcsinesubscript1 z_1 A^-1 y- ( % 2π)^3/2· nd(1+o(1)) w_1| start_ARG italic_z1⊤ italic_A- 1 italic_y - ( start_ARG divide start_ARG 2 end_ARG start_ARG π end_ARG end_ARG )3 / 2 ⋅ divide start_ARG n end_ARG start_ARG d end_ARG ( 1 + o ( 1 ) ) arcsin w1 end_ARG | ≲O~⁢(nd)+‖i⊤⁢−R−1⁢R‖2⁢‖R⊤⁢−R−1⁢‖2.less-than-or-similar-toabsent~subscriptnormsuperscriptsubscripttopsuperscriptsubscript1subscript2subscriptnormsuperscriptsubscripttopsuperscriptsubscript12 O ( nd)+ z_% i A_-R^-1 X_R_2 X_R A_% -R^-1 y_2.≲ over~ start_ARG O end_ARG ( start_ARG divide start_ARG square-root start_ARG n end_ARG end_ARG start_ARG d end_ARG end_ARG ) + ∥ start_ARG italic_zitalic_i⊤ italic_A- R- 1 italic_Xitalic_R end_ARG ∥2 ∥ start_ARG italic_Xitalic_R⊤ italic_A- R- 1 italic_y end_ARG ∥2 . Thus the statement follows. ∎ By way of Corollary B.10, we get a clean expression for the survival if =∗subscript w= v_*italic_w = italic_v∗. Corollary B.12. If q+r>11q+r>1q + r > 1, ∗subscript v_*italic_v∗ satisfies the 1-sparse assumption, and =sgn⁢(⟨,∗⟩)sgnexpectation-valuesubscript y=sgn( g, v_*)italic_y = sgn ( ⟨ start_ARG italic_g , italic_v∗ end_ARG ⟩ ), then with probability 1−O⁢(1/n)111-O(1/n)1 - O ( 1 / n ) 1⊤⁢−1⁢superscriptsubscript1topsuperscript1 z_1 A^-1 yitalic_z1⊤ italic_A- 1 italic_y ≍ndasymptotically-equalsabsent nd≍ divide start_ARG n end_ARG start_ARG d end_ARG |i⊤⁢−1⁢|superscriptsubscripttopsuperscript1 z_i A^-1 y| start_ARG italic_zitalic_i⊤ italic_A- 1 italic_y end_ARG | ⩽O~⁢(nd)∀i>1formulae-sequenceabsent~for-all1 O ( nd) ∀ i>1⩽ over~ start_ARG O end_ARG ( start_ARG divide start_ARG square-root start_ARG n end_ARG end_ARG start_ARG d end_ARG end_ARG ) ∀ i > 1 SUn⁢(∗|∗)subscriptSUnconditionalsubscriptsubscript SU_n( v_*| v_*)SUn ( v∗ | v∗ ) ≍μnasymptotically-equalsabsentsubscript _n≍ μitalic_n Proof. If =∗subscript w= v_*italic_w = italic_v∗, then by Corollary B.10 the error bound simplifies to |i⊤⁢−1⁢−i⊤⁢−R−1⁢|⩽O~⁢(μn⋅nd)i=1O~⁢(μn⋅nd)otherwisesuperscriptsubscripttopsuperscript1superscriptsubscripttopsuperscriptsubscript1cases~⋅subscript1~⋅subscriptotherwise z_i A^-1 y- z_i^% A_-R^-1 y cases* O( _n% · nd)&$i=1$\\ O( _n· nd)&otherwise cases*| start_ARG italic_zitalic_i⊤ italic_A- 1 italic_y - italic_zitalic_i⊤ italic_A- R- 1 italic_y end_ARG | ⩽ start_ROW start_CELL over~ start_ARG O end_ARG ( μitalic_n ⋅ divide start_ARG n end_ARG start_ARG d end_ARG ) end_CELL start_CELL i = 1 end_CELL end_ROW start_ROW start_CELL over~ start_ARG O end_ARG ( μitalic_n ⋅ divide start_ARG n end_ARG start_ARG d end_ARG ) end_CELL start_CELL otherwise end_CELL end_ROW Since q+r>11q+r>1q + r > 1, these error terms are lower order, and the results follow. ∎ We can get a similarly clean bound under a few additional assumptions on witalic_w and q+rq+rq + r. Corollary B.13. If, in addition to the assumptions of Proposition B.11, we have (1) |wi|⩾ω⁢(log⁡n)subscript w_i ω( nn)| start_ARG witalic_i end_ARG | ⩾ ω ( square-root start_ARG divide start_ARG log n end_ARG start_ARG n end_ARG end_ARG ) for i∈1,2,s+1121i∈ 1,2,s+1i ∈ start_ARG 1 , 2 , s + 1 end_ARG (2) |w2|=o⁢(|w1|)subscript2subscript1 w_2=o( w_1)| start_ARG w2 end_ARG | = o ( | start_ARG w1 end_ARG | ) (3) μn⁢n1−p2≪|w1|much-less-thansubscriptsuperscript12subscript1 _nn 1-p2 w_1square-root start_ARG μitalic_n end_ARG ndivide start_ARG 1 - p end_ARG start_ARG 2 end_ARG ≪ | start_ARG w1 end_ARG | then with yi=sgn⁢(⟨i,⟩)subscriptsgnexpectation-valuesubscripty_i=sgn( g_i, w)yitalic_i = sgn ( ⟨ start_ARG italic_gitalic_i , italic_w end_ARG ⟩ ), we have 1⊤⁢−1⁢≍nd⁢arcsin⁡w1.asymptotically-equalssuperscriptsubscript1topsuperscript1arcsinesubscript1 z_1 A^-1 y nd w_1.italic_z1⊤ italic_A- 1 italic_y ≍ divide start_ARG n end_ARG start_ARG d end_ARG arcsin w1 . Furthermore, the survival of ∗subscript v_*italic_v∗ given n labels generated by witalic_w is with high probability SUn⁢(∗|)≍μn⁢w1asymptotically-equalssubscriptSUnconditionalsubscriptsubscriptnsubscriptw1 SU_n( v_*| w) _nw_1SUn ( v∗ | w ) ≍ μroman_n w1 (16) Proof. Under the additional assumptions, then from Lemma B.9 ‖R⊤⁢−R−1⁢‖2subscriptnormsuperscriptsubscripttopsuperscriptsubscript12 X_R A_-R^-1 y_2∥ start_ARG italic_Xitalic_R⊤ italic_A- R- 1 italic_y end_ARG ∥2 ≲μn⁢(|w1|+|w2|)⁢n1−p2+n1−pless-than-or-similar-toabsentsubscriptsubscript1subscript2superscript12superscript1 _n( w_1+ w_2% )n 1-p2+n^1-p≲ square-root start_ARG μitalic_n end_ARG ( | start_ARG w1 end_ARG | + | start_ARG w2 end_ARG | ) ndivide start_ARG 1 - p end_ARG start_ARG 2 end_ARG + n1 - p (|wi|=ω⁢(log⁡n/n)subscript w_i=ω( n/n)| start_ARG witalic_i end_ARG | = ω ( square-root start_ARG log n / n end_ARG )) ≲μn⁢|w1|⁢n1−p2+n1−pless-than-or-similar-toabsentsubscriptsubscript1superscript12superscript1 _n w_1n 1-p2+n^1% -p≲ square-root start_ARG μitalic_n end_ARG | start_ARG w1 end_ARG | ndivide start_ARG 1 - p end_ARG start_ARG 2 end_ARG + n1 - p (|w2|=o⁢(|w1|)subscript2subscript1 w_2=o( w_1)| start_ARG w2 end_ARG | = o ( | start_ARG w1 end_ARG | )) Since i=11i=1i = 1 here, we can apply Lemma B.9 to see that ‖1⊤⁢−R−1⁢R‖2=O~⁢(μn⁢n1−p2)subscriptnormsuperscriptsubscript1topsuperscriptsubscript1subscript2~subscriptsuperscript12 z_1 A_-R^-1 X_R_2= O( % _nn 1-p2)∥ start_ARG italic_z1⊤ italic_A- R- 1 italic_Xitalic_R end_ARG ∥2 = over~ start_ARG O end_ARG ( square-root start_ARG μitalic_n end_ARG ndivide start_ARG 1 - p end_ARG start_ARG 2 end_ARG ). Thus, the Woodbury error term is upper bounded by O~⁢(μn⁢n1−p2)⁢(μn⁢|w1|⁢n1−p2+n1−p)~subscriptsuperscript12subscriptsubscript1superscript12superscript1 O( _nn 1-p2)( _n% w_1n 1-p2+n^1-p)over~ start_ARG O end_ARG ( square-root start_ARG μitalic_n end_ARG ndivide start_ARG 1 - p end_ARG start_ARG 2 end_ARG ) ( square-root start_ARG μitalic_n end_ARG | start_ARG w1 end_ARG | ndivide start_ARG 1 - p end_ARG start_ARG 2 end_ARG + n1 - p ) ⩽O~⁢(μn⁢|w1|⁢n1−p+μn⁢n3−3⁢p2)absent~subscriptsubscript1superscript1subscriptsuperscript332 O( _n w_1n^1-p+ % _nn 3-3p2)⩽ over~ start_ARG O end_ARG ( μitalic_n | start_ARG w1 end_ARG | n1 - p + square-root start_ARG μitalic_n end_ARG ndivide start_ARG 3 - 3 p end_ARG start_ARG 2 end_ARG ) ≲O~⁢(μn⁢|w1|⁢n1−p)+o⁢(n1−p⁢|w1|)less-than-or-similar-toabsent~subscriptsubscript1superscript1superscript1subscript1 O( _n w_1n^1-p)+o(n^1% -p w_1)≲ over~ start_ARG O end_ARG ( μitalic_n | start_ARG w1 end_ARG | n1 - p ) + o ( n1 - p | start_ARG w1 end_ARG | ) ⩽o⁢(n1−p⁢|w1|)absentsuperscript1subscript1 o(n^1-p w_1)⩽ o ( n1 - p | start_ARG w1 end_ARG | ) where the second line follows because we assumed μn⁢n1−p2≪|w1|much-less-thansubscriptsuperscript12subscript1 _nn 1-p2 w_1square-root start_ARG μitalic_n end_ARG ndivide start_ARG 1 - p end_ARG start_ARG 2 end_ARG ≪ | start_ARG w1 end_ARG |, and the last line follows because we assumed q+r>11q+r>1q + r > 1 so μn≪1much-less-thansubscript1 _n 1μitalic_n ≪ 1. Therefore, Proposition B.8 implies that |1⊤⁢−1⁢−(2π)3/2⁢n1−p⁢(1+o⁢(1))⁢arcsin⁡w1|superscriptsubscript1topsuperscript1superscript232superscript111arcsinesubscript1 z_1 A^-1 y- ( % 2π)^3/2n^1-p(1+o(1)) w_1| start_ARG italic_z1⊤ italic_A- 1 italic_y - ( start_ARG divide start_ARG 2 end_ARG start_ARG π end_ARG end_ARG )3 / 2 n1 - p ( 1 + o ( 1 ) ) arcsin w1 end_ARG | ≲O~(n12−p)+o(n1−parcsin⁡(|w1|) O(n 12-p)+o(n^1-p% ( w_1)≲ over~ start_ARG O end_ARG ( ndivide start_ARG 1 end_ARG start_ARG 2 end_ARG - p ) + o ( n1 - p arcsin ( start_ARG | start_ARG w1 end_ARG | end_ARG ) ⩽o⁢(n1−p⁢arcsin⁡w1),absentsuperscript1arcsinesubscript1 o (n^1-p w_1),⩽ o ( start_ARG n1 - p arcsin w1 end_ARG ) , (|w1|⩾ω⁢(log⁡n)subscript1 w_1 ω( nn)| start_ARG w1 end_ARG | ⩾ ω ( square-root start_ARG divide start_ARG log n end_ARG start_ARG n end_ARG end_ARG )) which recovers the stated result. ∎ B.3 Contamination analysis We now move onto the contamination. Proposition B.14. Suppose ∗subscript v_*italic_v∗ satisfies the 1-sparse assumption, and in the distinguished basis U∗subscriptU_*U∗ we have ∗=1subscriptsubscript1 v_*= e_1italic_v∗ = italic_e1 and =w1⁢1+w2⁢2+ws+1⁢s+1subscript1subscript1subscript2subscript2subscript1subscript1 w=w_1 e_1+w_2 e_2+w_s+1 e_s+1italic_w = w1 italic_e1 + w2 italic_e2 + witalic_s + 1 italic_eitalic_s + 1. Under the same assumptions as Corollary B.13, if there are n datapoints, and q+r>11q+r>1q + r > 1, then n⁢(∗|)2≲o⁢(μn2⁢|w1|2)+μn2⁢nr−1⁢log⁡(n)2+n1−p⁢log⁡(n)less-than-or-similar-tosubscriptsuperscriptconditionalsubscript2superscriptsubscript2superscriptsubscript12superscriptsubscript2superscript1superscript2superscript1 CN_n( v_*| w)^2 o( _n^2% w_1^2)+ _n^2n^r-1 (n)^2+n^1-p (n)sansserif_CNn ( italic_v∗ | italic_w )2 ≲ o ( μitalic_n2 | start_ARG w1 end_ARG |2 ) + μitalic_n2 nitalic_r - 1 log ( start_ARG n end_ARG )2 + n1 - p log ( start_ARG n end_ARG ) n⁢(∗|)2≳μn2⁢nr−1+n1−p.greater-than-or-equivalent-tosubscriptsuperscriptconditionalsubscript2superscriptsubscript2superscript1superscript1 CN_n( v_*| w)^2 _n^2n^r-1+n% ^1-p.sansserif_CNn ( italic_v∗ | italic_w )2 ≳ μitalic_n2 nitalic_r - 1 + n1 - p . Furthermore, the lower bound holds even without the additional assumptions from Corollary B.13. Proof. Since ∗subscript v_*italic_v∗ is 1-sparse, we have ⁢(∗|)2superscriptconditionalsubscript2 CN( v_*| w)^2sansserif_CN ( italic_v∗ | italic_w )2 =∑i∈2,s+1(λi⁢i⊤⁢−1⁢)2+∑i∉1,2,s+1(λi⁢i⊤⁢−1⁢)2.absentsubscript21superscriptsubscriptsuperscriptsubscripttopsuperscript12subscript121superscriptsubscriptsuperscriptsubscripttopsuperscript12 = _i∈ 2,s+1( _i z_i A^% -1 y)^2+ _i ∈ 1,2,s+1( _i z_i % A^-1 y)^2\,.= ∑i ∈ start_ARG 2 , s + 1 end_ARG ( λitalic_i italic_zitalic_i⊤ italic_A- 1 italic_y )2 + ∑i ∉ start_ARG 1 , 2 , s + 1 end_ARG ( λitalic_i italic_zitalic_i⊤ italic_A- 1 italic_y )2 . (17) Since by definition wi=0subscript0w_i=0witalic_i = 0 for i∉1,2,s+1121i ∈ 1,2,s+1i ∉ start_ARG 1 , 2 , s + 1 end_ARG , the second term in Equation 17 can be upper bounded up to constant factors using Wu and Sahai [2024, Proposition A.2] by μn2⁢sn⁢log⁡(n)2+nd⁢log⁡(n)=μn2⁢nr−1⁢log⁡(n)2+n1−p⁢log⁡(n).superscriptsubscript2superscript2superscriptsubscript2superscript1superscript2superscript1 _n^2 sn (n)^2+ nd (n)= _n^2n% ^r-1 (n)^2+n^1-p (n).μitalic_n2 divide start_ARG s end_ARG start_ARG n end_ARG log ( start_ARG n end_ARG )2 + divide start_ARG n end_ARG start_ARG d end_ARG log ( start_ARG n end_ARG ) = μitalic_n2 nitalic_r - 1 log ( start_ARG n end_ARG )2 + n1 - p log ( start_ARG n end_ARG ) . Now, set R=1,2,s+1subscript1subscript2subscript1R= e_1, e_2, e_s+1R = start_ARG italic_e1 , italic_e2 , italic_eitalic_s + 1 end_ARG . For the first term, we know from Lemma B.9 that |i⊤⁢−1⁢−i⊤⁢−R−1⁢|≲‖i⊤⁢−R−1⁢R‖2⁢‖R⊤⁢−R−1⁢‖2,less-than-or-similar-tosuperscriptsubscripttopsuperscript1superscriptsubscripttopsuperscriptsubscript1subscriptnormsuperscriptsubscripttopsuperscriptsubscript1subscript2subscriptnormsuperscriptsubscripttopsuperscriptsubscript12 z_i A^-1 y- z_i^% A_-R^-1 y z_i A_-R^-1% X_R_2 X_R A_-R^-1 y_2,| start_ARG italic_zitalic_i⊤ italic_A- 1 italic_y - italic_zitalic_i⊤ italic_A- R- 1 italic_y end_ARG | ≲ ∥ start_ARG italic_zitalic_i⊤ italic_A- R- 1 italic_Xitalic_R end_ARG ∥2 ∥ start_ARG italic_Xitalic_R⊤ italic_A- R- 1 italic_y end_ARG ∥2 , where ‖i⊤⁢−R−1⁢R‖2subscriptnormsuperscriptsubscripttopsuperscriptsubscript1subscript2 z_i A_-R^-1 X_R_2∥ start_ARG italic_zitalic_i⊤ italic_A- R- 1 italic_Xitalic_R end_ARG ∥2 =O~⁢(μn⁢n1−p2)i=2O~⁢(μn⁢n−p2+n1−p)i=s+1absentcases~subscriptsuperscript122~subscriptsuperscript2superscript11 = cases* O( _nn 1-p2)&$i=% 2$\\ O( _nn^- p2+n^1-p)&$i=s+1$ cases*= start_ROW start_CELL over~ start_ARG O end_ARG ( square-root start_ARG μitalic_n end_ARG ndivide start_ARG 1 - p end_ARG start_ARG 2 end_ARG ) end_CELL start_CELL i = 2 end_CELL end_ROW start_ROW start_CELL over~ start_ARG O end_ARG ( square-root start_ARG μitalic_n end_ARG n- divide start_ARG p end_ARG start_ARG 2 end_ARG + n1 - p ) end_CELL start_CELL i = s + 1 end_CELL end_ROW and from our bounds in Proposition B.11, we have ‖R⊤⁢−R−1⁢‖2subscriptnormsuperscriptsubscripttopsuperscriptsubscript12 X_R A_-R^-1 y_2∥ start_ARG italic_Xitalic_R⊤ italic_A- R- 1 italic_y end_ARG ∥2 ≲μn⁢|w1|⁢n1−p2+n1−p.less-than-or-similar-toabsentsubscriptsubscript1superscript12superscript1 _n w_1n 1-p2+n^1% -p.≲ square-root start_ARG μitalic_n end_ARG | start_ARG w1 end_ARG | ndivide start_ARG 1 - p end_ARG start_ARG 2 end_ARG + n1 - p . Hence we have ‖2⊤⁢−R−1⁢R‖2⁢‖R⊤⁢−R−1⁢‖2subscriptnormsuperscriptsubscript2topsuperscriptsubscript1subscript2subscriptnormsuperscriptsubscripttopsuperscriptsubscript12 z_2 A_-R^-1 X_R_2 % X_R A_-R^-1 y_2∥ start_ARG italic_z2⊤ italic_A- R- 1 italic_Xitalic_R end_ARG ∥2 ∥ start_ARG italic_Xitalic_R⊤ italic_A- R- 1 italic_y end_ARG ∥2 ⩽o⁢(n1−p⁢|w1|).absentsuperscript1subscript1 o(n^1-p w_1).⩽ o ( n1 - p | start_ARG w1 end_ARG | ) . On the other hand, we have ‖s+1⊤⁢−R−1⁢R‖2⁢‖R⊤⁢−R−1⁢‖2subscriptnormsuperscriptsubscript1topsuperscriptsubscript1subscript2subscriptnormsuperscriptsubscripttopsuperscriptsubscript12 z_s+1 A_-R^-1 X_R_2 % X_R A_-R^-1 y_2∥ start_ARG italic_zitalic_s + 1⊤ italic_A- R- 1 italic_Xitalic_R end_ARG ∥2 ∥ start_ARG italic_Xitalic_R⊤ italic_A- R- 1 italic_y end_ARG ∥2 ≲μn⁢|w1|⁢n12−p+μn⁢|w1|⁢n3−3⁢p2+μn⁢n1−3⁢p2+n2−2⁢pless-than-or-similar-toabsentsubscriptsubscript1superscript12subscriptsubscript1superscript332subscriptsuperscript132superscript22 _n w_1n 12-p+ _% n w_1n 3-3p2+ _nn^1- 3p2+n^% 2-2p≲ μitalic_n | start_ARG w1 end_ARG | ndivide start_ARG 1 end_ARG start_ARG 2 end_ARG - p + square-root start_ARG μitalic_n end_ARG | start_ARG w1 end_ARG | ndivide start_ARG 3 - 3 p end_ARG start_ARG 2 end_ARG + square-root start_ARG μitalic_n end_ARG n1 - divide start_ARG 3 p end_ARG start_ARG 2 end_ARG + n2 - 2 p ≲o⁢(n12−p)+o⁢(n1−p⁢|w1|)+n2−2⁢p,less-than-or-similar-toabsentsuperscript12superscript1subscript1superscript22 o(n 12-p)+o(n^1-p w_1)+n^% 2-2p,≲ o ( ndivide start_ARG 1 end_ARG start_ARG 2 end_ARG - p ) + o ( n1 - p | start_ARG w1 end_ARG | ) + n2 - 2 p , where the last line follows from the assumptions and p>11p>1p > 1. Putting these together with Hanson-Wright, we have |2⊤⁢−1⁢|superscriptsubscript2topsuperscript1 z_2 A^-1 y| start_ARG italic_z2⊤ italic_A- 1 italic_y end_ARG | ⩽O~⁢(n12−p)+o⁢(n1−p⁢|w1|)absent~superscript12superscript1subscript1 O(n 12-p)+o(n^1-p % w_1)⩽ over~ start_ARG O end_ARG ( ndivide start_ARG 1 end_ARG start_ARG 2 end_ARG - p ) + o ( n1 - p | start_ARG w1 end_ARG | ) |s+1⊤⁢−1⁢|superscriptsubscript1topsuperscript1 z_s+1 A^-1 y| start_ARG italic_zitalic_s + 1⊤ italic_A- 1 italic_y end_ARG | ⩽O~⁢(n12−p)+o⁢(n1−p⁢|w1|)+n2−2⁢pabsent~superscript12superscript1subscript1superscript22 O(n 12-p)+o(n^1-p % w_1)+n^2-2p⩽ over~ start_ARG O end_ARG ( ndivide start_ARG 1 end_ARG start_ARG 2 end_ARG - p ) + o ( n1 - p | start_ARG w1 end_ARG | ) + n2 - 2 p This yields an upper bound of o⁢(μn2⁢|w1|2)+O~⁢(n1−2⁢p)+o⁢(n2−2⁢p⁢|w1|2)+n4−4⁢psuperscriptsubscript2superscriptsubscript12~superscript12superscript22superscriptsubscript12superscript44 o( _n^2 w_1^2)+ O(n^1-2p)+% o(n^2-2p w_1^2)+n^4-4po ( μitalic_n2 | start_ARG w1 end_ARG |2 ) + over~ start_ARG O end_ARG ( n1 - 2 p ) + o ( n2 - 2 p | start_ARG w1 end_ARG |2 ) + n4 - 4 p ⩽o⁢(μn2⁢|w1|2)+n1−p,absentsuperscriptsubscript2superscriptsubscript12superscript1 o( _n^2 w_1^2)+n^1-p,⩽ o ( μitalic_n2 | start_ARG w1 end_ARG |2 ) + n1 - p , as p>11p>1p > 1 and |w1|⩽1subscript11 w_1 1| start_ARG w1 end_ARG | ⩽ 1. This completes the proof of the upper bound. For the lower bound, a careful inspection of the proof of Wu and Sahai [2024, Proposition A.2] reveals that, for q+r>11q+r>1q + r > 1, the lower bound holds for any multiclass problem with t⩾00t 0t ⩾ 0. This recovers the desired bound by simply lower bounding the second term in the expansion Equation 17, completing the proof. ∎ Appendix C Analyzing the subset ensemble In this section, we prove Theorem 3.2, which establishes weak-to-strong generalization in the simple weak-to-strong subset ensemble. We restate it below for convenience. See 3.2 Proof. The key idea of the proof is that, despite the error rate of the weak classifier being 12−o⁢(1)121 12-o(1)divide start_ARG 1 end_ARG start_ARG 2 end_ARG - o ( 1 ), the weak classifier still has a noticeable amount of mass on the true label defining direction. So long as the error rate is not 12−O⁢(1m)121 12-O( 1 m)divide start_ARG 1 end_ARG start_ARG 2 end_ARG - O ( divide start_ARG 1 end_ARG start_ARG square-root start_ARG m end_ARG end_ARG ), which random guessing can achieve, the strong learner can still pick up on the signal contained in the weak labels. To make this precise, we will control the coordinates of subscript f_ weakitalic_fsansserif_weak in the distinguished basis (Definition 5). Quantitatively, for the weak learner subscript f_ weakitalic_fsansserif_weak, since we have clean labels and q+r>1subscriptsubscript1q_ weak+r_ weak>1qsansserif_weak + rsansserif_weak > 1, we can apply Corollary B.12. Let μn,≜a⁢ns≜subscriptsubscriptsubscript _n, weak a_ weakns_ weakμitalic_n , sansserif_weak ≜ divide start_ARG asansserif_weak n end_ARG start_ARG ssansserif_weak end_ARG denote the bi-level prefactor for the weak model. Then the corollary shows that in the basis where ∗=1subscriptsubscript1 v_*= e_1italic_v∗ = italic_e1, we have with high probability ⁢[1]subscriptdelimited-[]1 f_ weak[1]italic_fsansserif_weak [ 1 ] =λF,⁢1⊤⁢−1⁢absentsubscriptsuperscriptsubscript1topsuperscript1 = _F, weak z_1 A^-1 y= square-root start_ARG λitalic_F , sansserif_weak end_ARG italic_z1⊤ italic_A- 1 italic_y ≍a⁢ds⁢ndasymptotically-equalsabsentsubscriptsubscriptsubscriptsubscript a_ weakd_ weaks_ % weak nd_ weak≍ square-root start_ARG divide start_ARG asansserif_weak dsansserif_weak end_ARG start_ARG ssansserif_weak end_ARG end_ARG divide start_ARG n end_ARG start_ARG dsansserif_weak end_ARG =μn,⁢n1−p2,absentsubscriptsuperscript1subscript2 = _n, weakn 1-p_ weak2,= square-root start_ARG μitalic_n , sansserif_weak end_ARG ndivide start_ARG 1 - psansserif_weak end_ARG start_ARG 2 end_ARG , whereas for all other coordinates we have with high probability |⁢[i]|subscriptdelimited-[] f_ weak[i]| start_ARG italic_fsansserif_weak [ i ] end_ARG | ≲λF,⋅nd⁢log⁡ni∈2,…,sλU,⋅nd⁢log⁡ni∈s+1,…,dless-than-or-similar-toabsentcases⋅subscriptsubscript2…subscript⋅subscriptsubscriptsubscript1…subscript cases* _F, weak· % nd_ weak n&$i∈ 2,…,s_ % weak$\\ _U, weak· nd_ weak % n&$i∈ s_ weak+1,…,d_ weak$ % cases*≲ start_ROW start_CELL square-root start_ARG λitalic_F , sansserif_weak end_ARG ⋅ divide start_ARG square-root start_ARG n end_ARG end_ARG start_ARG dsansserif_weak end_ARG square-root start_ARG log n end_ARG end_CELL start_CELL i ∈ start_ARG 2 , … , ssansserif_weak end_ARG end_CELL end_ROW start_ROW start_CELL square-root start_ARG λitalic_U , sansserif_weak end_ARG ⋅ divide start_ARG square-root start_ARG n end_ARG end_ARG start_ARG dsansserif_weak end_ARG square-root start_ARG log n end_ARG end_CELL start_CELL i ∈ start_ARG ssansserif_weak + 1 , … , dsansserif_weak end_ARG end_CELL end_ROW ≲μn,⁢n−p2⁢log⁡ni∈2,…,sn12−p⁢log⁡ni∈s+1,…,dless-than-or-similar-toabsentcasessubscriptsuperscriptsubscript22…subscriptsuperscript12subscriptsubscript1…subscript cases* _n, weakn^- p_% weak2 n&$i∈ 2,…,s_ weak$\\ n 12-p_ weak n&$i∈ s_ weak% +1,…,d_ weak$ cases*≲ start_ROW start_CELL square-root start_ARG μitalic_n , sansserif_weak end_ARG n- divide start_ARG psansserif_weak end_ARG start_ARG 2 end_ARG square-root start_ARG log n end_ARG end_CELL start_CELL i ∈ start_ARG 2 , … , ssansserif_weak end_ARG end_CELL end_ROW start_ROW start_CELL ndivide start_ARG 1 end_ARG start_ARG 2 end_ARG - psansserif_weak square-root start_ARG log n end_ARG end_CELL start_CELL i ∈ start_ARG ssansserif_weak + 1 , … , dsansserif_weak end_ARG end_CELL end_ROW Hence, to obtain subscript w_ weakitalic_wsansserif_weak in U∗subscriptU_*U∗, we first normalize subscript f_ weakitalic_fsansserif_weak and then do the basis change. Some calculation yields ‖2superscriptnormsubscript2 f_ weak^2∥ start_ARG italic_fsansserif_weak end_ARG ∥2 ≲μn,⁢n1−p+n1−p⁢log⁡n⋅s⁢λF,+(d−s)⁢λU,dless-than-or-similar-toabsentsubscriptsuperscript1subscriptsuperscript1subscript⋅subscriptsubscriptsubscriptsubscriptsubscriptsubscript _n, weakn^1-p_ weak+n^1-p_% weak n· s_ weak _F, weak+(d_% weak-s_ weak) _U, weakd_ weak≲ μitalic_n , sansserif_weak n1 - psansserif_weak + n1 - psansserif_weak log n ⋅ divide start_ARG ssansserif_weak λitalic_F , sansserif_weak + ( dsansserif_weak - ssansserif_weak ) λitalic_U , sansserif_weak end_ARG start_ARG dsansserif_weak end_ARG =(μn,+log⁡n)⁢n1−pabsentsubscriptsuperscript1subscript =( _n, weak+ n)n^1-p_ weak= ( μitalic_n , sansserif_weak + log n ) n1 - psansserif_weak ≲n1−p⁢log⁡nless-than-or-similar-toabsentsuperscript1subscript n^1-p_ weak n≲ n1 - psansserif_weak log n (q+r>1subscriptsubscript1q_ weak+r_ weak>1qsansserif_weak + rsansserif_weak > 1) where the second line follows because s⁢λF,+(d−s)⁢λU,=dsubscriptsubscriptsubscriptsubscriptsubscriptsubscripts_ weak _F, weak+(d_ weak-s_ weak% ) _U, weak=d_ weakssansserif_weak λitalic_F , sansserif_weak + ( dsansserif_weak - ssansserif_weak ) λitalic_U , sansserif_weak = dsansserif_weak. Similarly, we can compute a simple lower bound: ‖2⩾∑i>s+1(i⊤⁢−1⁢)2superscriptnormsubscript2subscript1superscriptsuperscriptsubscripttopsuperscript12 f_ weak^2 _i>s+1( z_% i A^-1 y)^2∥ start_ARG italic_fsansserif_weak end_ARG ∥2 ⩾ ∑i > s + 1 ( italic_zitalic_i⊤ italic_A- 1 italic_y )2 =⊤⁢−1⁢∑i>s+1i⁢i⊤⁢−1⁢absentsuperscripttopsuperscript1subscript1subscriptsuperscriptsubscripttopsuperscript1 = y A^-1 _i>s+1 z_i z_i % A^-1 y= italic_y⊤ italic_A- 1 ∑i > s + 1 italic_zitalic_i italic_zitalic_i⊤ italic_A- 1 italic_y ≳n1−p.greater-than-or-equivalent-toabsentsuperscript1subscript n^1-p_ weak.≳ n1 - psansserif_weak . (Lemma B.7) We therefore obtain the sandwiching bounds n1−p2≲‖≲n1−p2⁢log⁡n.less-than-or-similar-tosuperscript1subscript2normsubscriptless-than-or-similar-tosuperscript1subscript2 n 1-p_ weak2 f_ % weak n 1-p_ weak2 n.ndivide start_ARG 1 - psansserif_weak end_ARG start_ARG 2 end_ARG ≲ ∥ start_ARG italic_fsansserif_weak end_ARG ∥ ≲ ndivide start_ARG 1 - psansserif_weak end_ARG start_ARG 2 end_ARG square-root start_ARG log n end_ARG . (18) By rotating to the distinguished basis, since q+r>1subscriptsubscript1q_ weak+r_ weak>1qsansserif_weak + rsansserif_weak > 1, this yields =‖=w1⁢1+w2⁢2+ws+1⁢s+1,subscriptsubscriptnormsubscriptsubscript1subscript1subscript2subscript2subscript1subscript1 w_ weak= f_ weak % f_ weak=w_1 e_1+w_2 e_2+w_s+1 e_s+1,italic_wsansserif_weak = divide start_ARG italic_fsansserif_weak end_ARG start_ARG ∥ start_ARG italic_fsansserif_weak end_ARG ∥ end_ARG = w1 italic_e1 + w2 italic_e2 + witalic_s + 1 italic_eitalic_s + 1 , where 1log⁡n⁢μn,≲less-than-or-similar-to1subscriptabsent 1 n _n, weak start_ARG 1 end_ARG start_ARG square-root start_ARG log n end_ARG end_ARG square-root start_ARG μitalic_n , sansserif_weak end_ARG ≲ w1≲μn,less-than-or-similar-tosubscript1subscript \,w_1 _n, weakw1 ≲ square-root start_ARG μitalic_n , sansserif_weak end_ARG (19) 1log⁡n⁢n−q2≲less-than-or-similar-to1superscriptsubscript2absent 1 nn^- q_ weak2 start_ARG 1 end_ARG start_ARG log n end_ARG n- divide start_ARG qsansserif_weak end_ARG start_ARG 2 end_ARG ≲ |w2|≲n−q2⁢log⁡nless-than-or-similar-tosubscript2superscriptsubscript2 \, w_2 n^- q_ weak2% n| start_ARG w2 end_ARG | ≲ n- divide start_ARG qsansserif_weak end_ARG start_ARG 2 end_ARG square-root start_ARG log n end_ARG (20) 1≲less-than-or-similar-to1absent 1 1 ≲ |ws+1|.subscript1 \, w_s+1\,.| start_ARG witalic_s + 1 end_ARG | . (21) Let us now check the conditions to apply Corollary B.13, keeping in mind we need to scale with respect to m instead of n. In particular, we require • wi=ω⁢(log⁡m)subscriptw_i=ω( mm)witalic_i = ω ( square-root start_ARG divide start_ARG log m end_ARG start_ARG m end_ARG end_ARG ) for all i∈1,2,s+1121i∈ 1,2,s+1i ∈ start_ARG 1 , 2 , s + 1 end_ARG . • |w2|≪w1much-less-thansubscript2subscript1 w_2 w_1| start_ARG w2 end_ARG | ≪ w1 • μm⁢nu−p2≪|w1|much-less-thansubscriptsuperscript2subscript1 _mn u-p2 w_1square-root start_ARG μitalic_m end_ARG ndivide start_ARG u - p end_ARG start_ARG 2 end_ARG ≪ | start_ARG w1 end_ARG | Since we are going to check the second condition, and clearly the first condition is satisfied for i=s+11i=s+1i = s + 1, it suffices to check the first condition for i=11i=1i = 1. By plugging in the scaling for w1subscript1w_1w1 in Equation 19, we have w1=ω⁢(log⁡m)subscript1w_1=ω( mm)w1 = ω ( square-root start_ARG divide start_ARG log m end_ARG start_ARG m end_ARG end_ARG ) if 1−q−r>−u⇔q+r<u+1.iff1subscriptsubscriptsubscriptsubscript1 1-q_ weak-r_ weak>-u q_ weak+r% _ weak<u+1.1 - qsansserif_weak - rsansserif_weak > - u ⇔ qsansserif_weak + rsansserif_weak < u + 1 . (22) Let us now verify the second condition. It turns out that the second condition always holds under the bi-level parameterization. Indeed, from Equations 19 and 20, we always have |w2|⩽n−q2⁢log⁡n≪1log⁡n⁢n1−q−r2⩽w1,subscript2superscriptsubscript2much-less-than1superscript1subscriptsubscript2subscript1 w_2 n^- q_ weak2% n 1 nn 1-q_ weak-r_ % weak2 w_1,| start_ARG w2 end_ARG | ⩽ n- divide start_ARG qsansserif_weak end_ARG start_ARG 2 end_ARG square-root start_ARG log n end_ARG ≪ divide start_ARG 1 end_ARG start_ARG square-root start_ARG log n end_ARG end_ARG ndivide start_ARG 1 - qsansserif_weak - rsansserif_weak end_ARG start_ARG 2 end_ARG ⩽ w1 , as r<1subscript1r_ weak<1rsansserif_weak < 1. Finally, for the third condition, we have μm⁢nu−p2=nu−q−r2+u−p2≪n1−q−r2subscriptsuperscript2superscript22much-less-thansuperscript1subscriptsubscript2 _mn u-p2=n u-q-r2+ u-p2% n 1-q_ weak-r_ weak2square-root start_ARG μitalic_m end_ARG ndivide start_ARG u - p end_ARG start_ARG 2 end_ARG = ndivide start_ARG u - q - r end_ARG start_ARG 2 end_ARG + divide start_ARG u - p end_ARG start_ARG 2 end_ARG ≪ ndivide start_ARG 1 - qsansserif_weak - rsansserif_weak end_ARG start_ARG 2 end_ARG ⇔2⁢u−(p+q+r)<1−(q+r)iffabsent21subscriptsubscript 2u-(p+q+r)<1-(q_ weak+r_ weak)⇔ 2 u - ( p + q + r ) < 1 - ( qsansserif_weak + rsansserif_weak ) ⇔q+r>2⁢u−(p+1)+q+riffabsent21subscriptsubscript q+r>2u-(p+1)+q_ weak+r_ weak⇔ q + r > 2 u - ( p + 1 ) + qsansserif_weak + rsansserif_weak (23) With these preparations in hand, we now prove the positive side of the result. Sufficient condition for weak-to-strong generalization. First we lower bound SUm⁢(∗|)subscriptSUmconditionalsubscriptsubscript SU_m( v_*| w_ weak)SUm ( v∗ | wsansserif_weak ). Under Equations 22 and 23, we conclude that SUm,⁢(∗|)≳μm⁢w1⩾1log⁡n⁢μm⋅n1−q−r2greater-than-or-equivalent-tosubscriptSUmconditionalsubscriptsubscriptsubscriptmsubscriptw1⋅1nsubscriptmsuperscriptn1subscriptqsubscriptr2 SU_m, strong( v_*| w_ weak% ) _mw_1 1 n _m· n 1-% q_ weak-r_ weak2SUm , sansserif_strong ( v∗ | wsansserif_weak ) ≳ μroman_m w1 ⩾ divide start_ARG 1 end_ARG start_ARG square-root start_ARG log n end_ARG end_ARG μroman_m ⋅ ndivide start_ARG 1 - qsansserif_weak - rsansserif_weak end_ARG start_ARG 2 end_ARG On the other hand, by Proposition B.14 we have m,⁢(∗|)2subscriptsuperscriptconditionalsubscript2 CN_m, strong( v_*| w)^2sansserif_CNm , sansserif_strong ( italic_v∗ | italic_w )2 ⩽o⁢(μm2⁢|w1|2)+μm2⁢nr−u⁢log⁡n+nu−p⁢log⁡n.absentsuperscriptsubscript2superscriptsubscript12superscriptsubscript2superscriptsuperscript o( _m^2 w_1^2)+ _m^2n^r% -u n+n^u-p n.⩽ o ( μitalic_m2 | start_ARG w1 end_ARG |2 ) + μitalic_m2 nitalic_r - u log n + nitalic_u - p log n . We will now analyze the survival-to-contamination ratio for the strong learner. The first term is clearly at least ω⁢(1)1ω(1)ω ( 1 ). For the second term, we get 1log⁡n⁢n1−q−r2−r−u2,1superscript1subscriptsubscript22 1 nn 1-q_ weak-r_ weak2% - r-u2,divide start_ARG 1 end_ARG start_ARG log n end_ARG ndivide start_ARG 1 - qsansserif_weak - rsansserif_weak end_ARG start_ARG 2 end_ARG - divide start_ARG r - u end_ARG start_ARG 2 end_ARG , which is ω⁢(1)1ω(1)ω ( 1 ) if q+r<u+1−rsubscriptsubscript1 q_ weak+r_ weak<u+1-rqsansserif_weak + rsansserif_weak < u + 1 - r (24) Finally, for the third term, we get 1log⁡n⁢nu−q−r+1−q−r2−u−p2,1superscript1subscriptsubscript22 1 nn^u-q-r+ 1-q_ weak-r_ % weak2- u-p2,divide start_ARG 1 end_ARG start_ARG log n end_ARG nitalic_u - q - r + divide start_ARG 1 - qsansserif_weak - rsansserif_weak end_ARG start_ARG 2 end_ARG - divide start_ARG u - p end_ARG start_ARG 2 end_ARG , which is ω⁢(1)1ω(1)ω ( 1 ) if q+r<u+p+1−2⁢(q+r).subscriptsubscript12 q_ weak+r_ weak<u+p+1-2(q+r).qsansserif_weak + rsansserif_weak < u + p + 1 - 2 ( q + r ) . (25) Collecting the conditions Equations 22, 24 and 25 yields u u >q+r−min⁡1,1−r,p+1−2⁢(q+r)absentsubscriptsubscript1112 >q_ weak+r_ weak- 1,1-r,p+1-2(q+% r)> qsansserif_weak + rsansserif_weak - min start_ARG 1 , 1 - r , p + 1 - 2 ( q + r ) end_ARG =q+r−min⁡1−r,p+1−2⁢(q+r),absentsubscriptsubscript112 =q_ weak+r_ weak- 1-r,p+1-2(q+r),= qsansserif_weak + rsansserif_weak - min start_ARG 1 - r , p + 1 - 2 ( q + r ) end_ARG , which establishes the positive side of the result. Sufficient condition for failure of weak-to-strong generalization. To get the negative result, we need to upper bound SUm,⁢(∗|)subscriptSUmconditionalsubscriptsubscript SU_m, strong( v_*| w_ weak)SUm , sansserif_strong ( v∗ | wsansserif_weak ) and lower bound m,⁢(∗|)subscriptconditionalsubscriptsubscript CN_m, strong( v_*| w_ weak)sansserif_CNm , sansserif_strong ( italic_v∗ | italic_wsansserif_weak ). Our earlier calculations and Proposition B.14 yield |SUm,⁢(∗|)|subscriptSUmconditionalsubscriptsubscript SU_m, strong( v_*| w% _ weak)| start_ARG SUm , sansserif_strong ( v∗ | wsansserif_weak ) end_ARG | ≲μm⋅max⁡w1,nu2−p⁢log⁡nless-than-or-similar-toabsent⋅subscriptsubscript1superscript2 _m· w_1,n u2-p % n≲ μitalic_m ⋅ max start_ARG w1 , ndivide start_ARG u end_ARG start_ARG 2 end_ARG - p square-root start_ARG log n end_ARG end_ARG (26) m,⁢(∗|)subscriptconditionalsubscriptsubscript CN_m, strong( v_*| w_ weak)sansserif_CNm , sansserif_strong ( italic_v∗ | italic_wsansserif_weak ) ≳μm⁢nr−u2+nu−p2.greater-than-or-equivalent-toabsentsubscriptsuperscript2superscript2 _mn r-u2+n u-p2\,.≳ μitalic_m ndivide start_ARG r - u end_ARG start_ARG 2 end_ARG + ndivide start_ARG u - p end_ARG start_ARG 2 end_ARG . (27) The above bound only differ by log factors from the upper bounds. Hence, in this case, the sufficient condition merely flips the direction of the inequality, which recovers the stated result. Otherwise, if u<q+r−1subscriptsubscript1u<q_ weak+r_ weak-1u < qsansserif_weak + rsansserif_weak - 1, then certainly u<q+r−1+rsubscriptsubscript1u<q_ weak+r_ weak-1+ru < qsansserif_weak + rsansserif_weak - 1 + r, and nu2−p⁢log⁡nsuperscript2n u2-p nndivide start_ARG u end_ARG start_ARG 2 end_ARG - p square-root start_ARG log n end_ARG dominates. The latter is in turn dominated by nu−p2superscript2n u-p2ndivide start_ARG u - p end_ARG start_ARG 2 end_ARG, so the survival-to-contamination ratio is o⁢(1)1o(1)o ( 1 ). This completes the proof. ∎ Appendix D Multilabel classification In this section, we analyze multilabel classification with k classes, which is a variant of multiclass classification where a single datapoint can be positive examples of multiple classes. Given n datapoints, we encode this label enformation using k label vectors (1),…,(k)superscript1…superscript y^(1),…, y^(k)italic_y( 1 ) , … , italic_y( k ), where for i∈[k]delimited-[]i∈[k]i ∈ [ k ], we encode the iiith label vector as (i)∈±1nsuperscriptsuperscriptplus-or-minus1 y^(i)∈ ± 1^nitalic_y( i ) ∈ start_ARG ± 1 end_ARG n, with 1111 representing positive examples. We make the following 1111-sparse assumption for multilabel data. Assumption 3 (1-sparse assumption for multilabel). The label defining directions ∗(1),…,∗(k)superscriptsubscript1…superscriptsubscript v_*^(1),…, v_*^(k)italic_v∗( 1 ) , … , italic_v∗( k ) are aligned with a top k eigenbasis of the strong covariance Σ Σ. In a strong eigenbasis where ∗(i)=isuperscriptsubscriptsubscript v_*^(i)= e_iitalic_v∗( i ) = italic_eitalic_i, we have y(j)=sgn⁢(xj),∀j∈[k]formulae-sequencesuperscriptsgnsubscriptfor-alldelimited-[] y^(j)=sgn(x_j), ∀ j∈[k]y( j ) = sgn ( xitalic_j ) , ∀ j ∈ [ k ] Remark D.1. In fact, the analysis below will turn out to work even if the label defining directions are not orthogonal; they could all be the same! The analysis only relies on there existing some basis UisubscriptU_iUitalic_i such that ∗(i)=1superscriptsubscriptsubscript1 v_*^(i)= e_1italic_v∗( i ) = italic_e1. However, the tight misclassification rate would be different depending on the relationship between the label defining directions. The weak-to-strong setup is defined as follows. (1) ∈ℝd×ksubscriptsuperscriptℝsubscript f_ weak ^d_ weak× kitalic_fsansserif_weak ∈ blackboard_Rdsansserif_weak × k: train on n datapoints using weak features and ground-truth labels. (2) ∈ℝd×ksubscriptsuperscriptℝ f_ w2s ^d× kitalic_fsansserif_w2s ∈ blackboard_Rd × k: train on m≫nmuch-greater-thanm nm ≫ n datapoints using strong features and hard pseudolabels generated from subscript f_ weakitalic_fsansserif_weak. As before, we will study weak-to-strong generalization using overparameterized linear classifiers. Hence, subscript f_ weakitalic_fsansserif_weak will now consist of k different linear classifiers (i)∈ℝdsuperscriptsubscriptsuperscriptℝsubscript f_ weak^(i) ^d_ weakitalic_fsansserif_weak( i ) ∈ blackboard_Rdsansserif_weak for i∈[k]delimited-[]i∈[k]i ∈ [ k ], all trained by MNI on n clean multilabel datapoints. To train (i)superscriptsubscript f_ w2s^(i)italic_fsansserif_w2s( i ), we generate hard pseudolabel vectors ^(i)=sgn⁢(⟨(i),⟩)superscript^sgnsuperscriptsubscriptsubscript y^(i)=sgn( f_ weak^(i), % x_ weak )over start_ARG italic_y end_ARG( i ) = sgn ( ⟨ italic_fsansserif_weak( i ) , italic_xsansserif_weak ⟩ ), and then perform MNI on these hard pseudolabel vectors. We deem that a multilabel classifier fitalic_f generalizes if, for a fresh test sample subscript x_ testitalic_xsansserif_test and for every class i∈[k]delimited-[]i∈[k]i ∈ [ k ], fitalic_f correctly labels whether subscript x_ testitalic_xsansserif_test is a positive example of class i. More formally, define for a collection of classifiers =((1),…,(k))superscript1…superscript f=( f^(1),…, f^(k))italic_f = ( italic_f( 1 ) , … , italic_f( k ) ), the loss function ℓ()=[∃i∈[k]:sgn((i)())≠y(i)] ( f)= Pr[∃ i∈[k]:sgn( f^(i)(% x_ test))≠ y_ test^(i)]ℓ ( italic_f ) = Pr [ ∃ i ∈ [ k ] : sgn ( italic_f( i ) ( italic_xsansserif_test ) ) ≠ ysansserif_test( i ) ], where the probability is taken over a fresh test sample (,y)subscriptsubscript( x_ test,y_ test)( italic_xsansserif_test , ysansserif_test ). Remark D.2. The weak model can also be trained on clean multiclass data rather than clean multilabel data. This will only affect the regimes for the weak model to satisfy Item 1.i, based on Theorem 3.1. The subset ensemble definition is essentially the same as before (Assumption 2), with the main difference being that we require all k of the label defining directions to be favored and axis-alignable in both the weak and strong favored feature subspace. Assumption 4 (Subset ensemble for multilabel classification). Let Λ=Λ⁢(p,q,r)∈ℝd×dΛsuperscriptℝ = (p,q,r) ^d× dΛ = Λ ( p , q , r ) ∈ blackboard_Rd × d denote the strong eigenvalues and Λ=Λ⁢(p,q,r)∈ℝd×dsubscriptΛsubscriptsubscriptsubscriptsuperscriptℝsubscriptsubscript _ weak= (p_ weak,q_ weak,r_ % weak) ^d_ weak× d_ weakΛsansserif_weak = Λ ( psansserif_weak , qsansserif_weak , rsansserif_weak ) ∈ blackboard_Rdsansserif_weak × dsansserif_weak denote the weak eigenvalues, both drawn from the bi-level ensemble. Suppose the 1111-sparse assumption (Assumption 1) holds for the strong covariance, with any distinguished eigenbasis U where ∗(i)=isuperscriptsubscriptsubscript v_*^(i)= e_iitalic_v∗( i ) = italic_eitalic_i for i∈[k]delimited-[]i∈[k]i ∈ [ k ]. The following conditions relate the weak and strong features after rotating to U. (1) ∼N⁢(0,Λ)similar-tosubscript0Λ x_ strong N(0, )italic_xsansserif_strong ∼ N ( 0 , Λ ), where Λ=λF⁢I[s]+λU⁢I[d]∖[s]Λsubscriptsubscriptdelimited-[]subscriptsubscriptdelimited-[]delimited-[] = _FI_[s]+ _UI_[d] [s]Λ = λitalic_F I[ s ] + λitalic_U I[ d ] ∖ [ s ]. (2) There exists subsets of coordinates S⊆[s],T⊆[d]∖[s]formulae-sequencedelimited-[]delimited-[]delimited-[]S [s],T [d] [s]S ⊆ [ s ] , T ⊆ [ d ] ∖ [ s ], with [k]⊆Sdelimited-[][k] S[ k ] ⊆ S and |S|=ssubscript S=s_ weak| start_ARG S end_ARG | = ssansserif_weak, such that =(λF,λF⁢ΠS+λU,λU⁢ΠT)⁢=dN⁢(0,λF,⁢IS+λU,⁢IT).subscriptsubscriptsubscriptsubscriptΠsubscriptsubscriptsubscriptΠsubscriptsuperscript0subscriptsubscriptsubscriptsubscript x_ weak=( _F, weak _F% _S+ _U, weak _U _T) x% _ strong d=N(0, _F, weak% I_S+ _U, weakI_T)\,.italic_xsansserif_weak = ( square-root start_ARG divide start_ARG λitalic_F , sansserif_weak end_ARG start_ARG λitalic_F end_ARG end_ARG Πitalic_S + square-root start_ARG divide start_ARG λitalic_U , sansserif_weak end_ARG start_ARG λitalic_U end_ARG end_ARG Πitalic_T ) italic_xsansserif_strong start_RELOP SUPERSCRIPTOP start_ARG = end_ARG start_ARG d end_ARG end_RELOP N ( 0 , λitalic_F , sansserif_weak Iitalic_S + λitalic_U , sansserif_weak Iitalic_T ) . The crucial observation is that multilabel training boils down to k (nearly) independent binary classification problems. The main difference to establish successful generalization is that we now need to union bound over all k multilabel classifiers. The high probability bounds from the binary analysis come from two sources: (1) applying spectral bounds (Lemma B.5), which holds with very high probability (exp⁡(−n1/2)superscript12 (-n^1/2)exp ( start_ARG - n1 / 2 end_ARG )), and (2) Hanson-Wright calculations (Proposition B.8), where bounds that hold with probability δ have deviation poly⁢log⁡(1/δ)poly1poly (1/δ)poly log ( start_ARG 1 / δ end_ARG ). Hence, for k a constant, δ is only affected by a constant, and this will only change the bounds in our analysis by a constant, which does not shift the regimes. Furthermore, even if we allow k to scale with n as in Definition 2, since k is polynomial in n, the dependence for the high probability bounds on k is at most polylogarithmic in k, so again nothing changes with the analysis, which looks at polynomial regime shifts. For the converse direction, one can use a crude bound of just analyzing the probability of one classifier failing, which gives a error rate bounded from below by 12−on⁢(1)12subscript1 12-o_n(1)divide start_ARG 1 end_ARG start_ARG 2 end_ARG - oitalic_n ( 1 ). However, we expect that a more refined analysis would give the expected error rate of 1−O⁢(2−k)1superscript21-O(2^-k)1 - O ( 2- k ); we sketch an argument for how to get this improved error rate after the theorem statement. With these minor modifications, the formal details go through unchanged, and we arrive at our main theorem for weak-to-strong multilabel classification using hard pseudo-multilabels. Theorem D.3 (Weak-to-strong generalization for multilabel subset ensemble). Consider the setup where the weak model subscript f_ weakitalic_fsansserif_weak is trained on n correctly labeled examples and the strong model subscript f_ w2sitalic_fsansserif_w2s is trained on m=nusuperscriptm=n^um = nitalic_u weakly labeled examples using MNI, where u<pu<pu < p, q+r>uq+r>uq + r > u, and q+r>1subscriptsubscript1q_ weak+r_ weak>1qsansserif_weak + rsansserif_weak > 1. Assume the following: (1) The true multilabels satisfy the 1111-sparse assumption (Assumption 3) for the strong covariance. (2) The weak and strong features follow the subset ensemble (Assumption 2) with bi-level eigenvalues Λ⁢(p,q,r)Λ (p,q,r)Λ ( p , q , r ) and Λ⁢(p,q,r)Λsubscriptsubscriptsubscript (p_ weak,q_ weak,r_ weak)Λ ( psansserif_weak , qsansserif_weak , rsansserif_weak ), respectively, scaled relative to n. (3) There are not too many weakly labeled examples: u<p+1+q+r−(q+r)21subscriptsubscript2u< p+1+q+r-(q_ weak+r_ weak)2u < divide start_ARG p + 1 + q + r - ( qsansserif_weak + rsansserif_weak ) end_ARG start_ARG 2 end_ARG. Let τ≜p+1−2⁢(q+r)≜subscript12 _ strong p+1-2(q+r)τsansserif_strong ≜ p + 1 - 2 ( q + r ). Then, the resulting test error for subscript f_ w2sitalic_fsansserif_w2s satisfies ⁢[ℓ⁢()]delimited-[]ℓsubscript E[ ( f_ w2s)]E [ ℓ ( italic_fsansserif_w2s ) ] =on⁢(1),if⁢u>q+r−min⁡1−r,τΩ⁢(1),if⁢u<q+r−min⁡1−r,τ.absentcasessubscript1ifsubscriptsubscript1subscriptΩ1ifsubscriptsubscript1subscript = caseso_n(1),&\ if\ u>q_ weak+r_% weak- 1-r, _ strong\\ (1),&\ if\ u<q_ weak+r_ weak- % 1-r, _ strong. cases= start_ROW start_CELL oitalic_n ( 1 ) , end_CELL start_CELL if u > qsansserif_weak + rsansserif_weak - min start_ARG 1 - r , τsansserif_strong end_ARG end_CELL end_ROW start_ROW start_CELL Ω ( 1 ) , end_CELL start_CELL if u < qsansserif_weak + rsansserif_weak - min start_ARG 1 - r , τsansserif_strong end_ARG . end_CELL end_ROW (28) We now sketch out an approach which should give the correct error rate for constant k (or even k growing slowly with n). Note that the complement of the failure event is ∀i∈[k]:sgn(⟨(i),)⟩=y(i).∀ i∈[k]:sgn( f_ w2s^(i), x_% test) =y_ test^(i).∀ i ∈ [ k ] : sgn ( ⟨ italic_fsansserif_w2s( i ) , italic_xsansserif_test ) ⟩ = ysansserif_test( i ) . From the Gram-Schmidt decomposition Equation 8 and the noise stability formula, we can decompose this event as ∀i∈[k]:sgn⁢(SU⁢(∗(i)|)⁢(∗(i)|)⁢x∗(i)+g(i))=sgn⁢(x∗(i)),:for-alldelimited-[]sgnSUconditionalsuperscriptsubscripticonditionalsuperscriptsubscriptsuperscriptsubscriptsuperscriptsgnsuperscriptsubscript∀ i∈[k]:sgn( SU( v_*^(i)| w)% CN( v_*^(i)| w)x_*^(i)+g^(i))=sgn(x_*^% (i)),∀ i ∈ [ k ] : sgn ( divide start_ARG SU ( v∗( i ) | w ) end_ARG start_ARG sansserif_CN ( italic_v∗( i ) | italic_w ) end_ARG x∗( i ) + g( i ) ) = sgn ( x∗( i ) ) , where x∗(i)=⟨,∗(i)⟩superscriptsubscriptexpectation-valuesubscriptsuperscriptsubscriptx_*^(i)= g_ test, v_*^(i)x∗( i ) = ⟨ start_ARG italic_gsansserif_test , italic_v∗( i ) end_ARG ⟩ and g(i)superscriptg^(i)g( i ) are iid standard Gaussians. In the converse regime, the survival to contamination ratio is polynomially decaying, so for typical g(i)superscriptg^(i)g( i ), with probability exp⁡(−nc)superscript (-n^c)exp ( start_ARG - nitalic_c end_ARG ) we have sgn⁢(SU⁢(∗(i)|)⁢(∗(i)|)⁢x∗(i)+g(i))=sgn⁢(g(i))sgnSUconditionalsuperscriptsubscripticonditionalsuperscriptsubscriptsuperscriptsubscriptsuperscriptsgnsuperscriptsgn( SU( v_*^(i)| w) CN( v% _*^(i)| w)x_*^(i)+g^(i))=sgn(g^(i))sgn ( divide start_ARG SU ( v∗( i ) | w ) end_ARG start_ARG sansserif_CN ( italic_v∗( i ) | italic_w ) end_ARG x∗( i ) + g( i ) ) = sgn ( g( i ) ). Furthermore, by unpacking on the analysis of (i)superscriptsubscript f_ w2s^(i)italic_fsansserif_w2s( i ) (specifically, see Equation 20) for i≠ji≠ ji ≠ j, g(i)superscriptg^(i)g( i ) and g(j)superscriptg^(j)g( j ) only differ by a Gaussian with polynomially decaying variance, so up to a failure event with probability exp⁡(−nc)superscript (-n^c)exp ( start_ARG - nitalic_c end_ARG ), we can replace these with the same Gaussian g~~ gover~ start_ARG g end_ARG which is independent of both x∗(i)superscriptsubscriptx_*^(i)x∗( i ) and x∗(j)superscriptsubscriptx_*^(j)x∗( j ). We can do this for all pairs (i,j)(i,j)( i , j ) and union bound over k (this is where we use the slow growing condition on k). Hence, up to these error terms, which are negligible, the complementary event occurs with probability 2−ksuperscript22^-k2- k, so the test error will indeed be 1−O⁢(2−k)1superscript21-O(2^-k)1 - O ( 2- k ). D.1 Multilabel supervision for multiclass classification In this section, we expand upon the arguments to use weak multilabel supervision to train a weak-to-strong multiclass classifier subscript f_ w2sitalic_fsansserif_w2s. The difficulty of the multiclass analysis is studying the MNI behavior for weak supervision. However, since the multilabel MNI behavior is well under control by the arguments above, it is tractable to study this setup instead. The key insight is that, for multiclass classification to succeed, it suffices to look at pairwise comparisons between the score functions for the k different classes (see Wu and Sahai [2024] for more justification). In particular, one studies the relative survival given different class label vectors: SU⁢(|^(i))−SU⁢(|^(j)),SUconditionalsuperscriptsubscript^iSUconditionalsuperscriptsubscript^j SU( v| y_ weak^(i))- SU( % v| y_ weak^(j)),SU ( v | over start_ARG y end_ARGsansserif_weak( i ) ) - SU ( v | over start_ARG y end_ARGsansserif_weak( j ) ) , with the signal component for class i being the above quantity with =∗(i)superscriptsubscript v= v_*^(i)italic_v = italic_v∗( i ). One can then define the relative contamination in analogous way. However, the path towards studying the survival comes from understanding the coefficients of subscript f_ weakitalic_fsansserif_weak and subscript f_ w2sitalic_fsansserif_w2s, which is feasible. Thus, to determine whether subscript f_ w2sitalic_fsansserif_w2s generalizes for multiclass classification, it reduces back down to the multilabel/binary behavior. As argued in Wu and Sahai [2024], because of the margin between the features and the survivals, it suffices to get a polynomially increasing SU/SU SU/ CNSU / sansserif_CN ratio. Hence, in the successful regime for weak-to-strong multilabel generalization, this relative SU/SU SU/ CNSU / sansserif_CN ratio is polynomially increasing, which implies that multiclass classification succeeds. Appendix E Improving the bounds for misclassification rate In this section we tighten the bounds on misclassification rate for the multiclass setting. Theorem E.1 (Tightening of Proposition A.7). WS24] Assume we are in the bi-level ensemble model (Definition 1), the true data generating process is 1-sparse (Assumption 1), and the number of classes follows the scaling defined in Definition 2. Then, in the negative regime where the model does not achieve vanishing error, we have ⁢[ℓ⁢()]=1−Θ⁢(1k),delimited-[]ℓsubscript1Θ1 E[ ( f_ strong)]=1- % ( 1k),E [ ℓ ( italic_fsansserif_strong ) ] = 1 - Θ ( start_ARG divide start_ARG 1 end_ARG start_ARG k end_ARG end_ARG ) , (29) where the expectation is taken over the randomness of the training data and the test point. At a high level, the misclassification event ℰerrsubscriptℰerrE_ errEerr is governed by the following inequality holding: SUnn⁢maxj∈[k]⁡|xj|⩽maxi∈[k]⁡gi,subscriptSUnsubscriptsubscriptdelimited-[]subscriptsubscriptdelimited-[]subscript SU_n CN_n _j∈[k]% x_j _i∈[k]g_i,divide start_ARG SUn end_ARG start_ARG sansserif_CNn end_ARG maxitalic_j ∈ [ k ] | start_ARG xitalic_j end_ARG | ⩽ maxitalic_i ∈ [ k ] gitalic_i , where (xj)j∈[k]subscriptsubscriptdelimited-[](x_j)_j∈[k]( xitalic_j )j ∈ [ k ] are iid standard normals, and (gi)i∈[k]subscriptsubscriptdelimited-[](g_i)_i∈[k]( gitalic_i )i ∈ [ k ] are jointly gaussian with some correlation structure. Now, [Wu and Sahai, 2024, Proposition A.3] implies that SUnn⩽n−usubscriptSUnsubscriptsuperscript SU_n CN_n n^-udivide start_ARG SUn end_ARG start_ARG sansserif_CNn end_ARG ⩽ n- u for some constant u>00u>0u > 0 with probability 1−O⁢(1/n)111-O(1/n)1 - O ( 1 / n ). Also, we can upper bound maxj∈[k]⁡|xj|⩽O⁢(log⁡k)subscriptdelimited-[]subscript _j∈[k] x_j O( k)maxitalic_j ∈ [ k ] | start_ARG xitalic_j end_ARG | ⩽ O ( square-root start_ARG log k end_ARG ) with probability at least 1−O⁢(1/k)111-O(1/k)1 - O ( 1 / k ). Moreover, in the regime where classification fails, we know that from [Wu and Sahai, 2024, Proposition F.1], we have that ⁢[gi⁢gj]⩽12+n−δdelimited-[]subscriptsubscript12superscript E[g_ig_j] 12+n^-δE [ gitalic_i gitalic_j ] ⩽ divide start_ARG 1 end_ARG start_ARG 2 end_ARG + n- δ for some constant δ>00δ>0δ > 0 with probability at least 1−O⁢(1/n)111-O(1/n)1 - O ( 1 / n ). Since k=o⁢(n)k=o(n)k = o ( n ), when we union bound over the above events, they get absorbed by the O⁢(⋅)⋅O(·)O ( ⋅ ) and Ω⁢(⋅)Ω⋅ (·)Ω ( ⋅ ) terms. To tighten the misclassification rate, we will improve upon [Lopes and Yao, 2022, Theorem 2.1]. In particular, we shows the following lower tail inequality for the maximum of correlated gaussians. We will prove the following theorem, which captures the lower tail behavior of the maximum of correalted Gaussians very far from its expectation. This result complements the lower tail bound of Lopes and Yao [2022], which holds for more moderate deviations. See 3.6 Before we prove the theorem, let us see how it tightens the misclassiifcation rate. Proof of Theorem E.1. We will apply Theorem 3.6 with N=k−11N=k-1N = k - 1, ρ0=12+n−δsubscript012superscript _0= 12+n^-δρ0 = divide start_ARG 1 end_ARG start_ARG 2 end_ARG + n- δ, and δ0=1log⁡ksubscript01 _0= 1 kδ0 = divide start_ARG 1 end_ARG start_ARG log k end_ARG. Noting that δ0⁢2⁢(1−ρ0)⁢log⁡k⩾n−usubscript021subscript0superscript _0 2(1- _0) k n^-uδ0 square-root start_ARG 2 ( 1 - ρ0 ) log k end_ARG ⩾ n- u if k=ck⁢ntsubscriptsuperscriptk=c_kn^tk = citalic_k nitalic_t for any t∈[0,1)01t∈[0,1)t ∈ [ 0 , 1 ), we see that ⁢[maxi∈[k]⁡gi⩽n−u]delimited-[]subscriptdelimited-[]subscriptsuperscript Pr[ _i∈[k]g_i n^-u]Pr [ maxitalic_i ∈ [ k ] gitalic_i ⩽ n- u ] ⩽⁢[maxi∈[k]⁡gi⩽δ0⁢2⁢(1−ρ0)⁢log⁡k]absentdelimited-[]subscriptdelimited-[]subscriptsubscript021subscript0 Pr[ _i∈[k]g_i _0% 2(1- _0) k]⩽ Pr [ maxitalic_i ∈ [ k ] gitalic_i ⩽ δ0 square-root start_ARG 2 ( 1 - ρ0 ) log k end_ARG ] (δ0⁢2⁢(1−ρ0)⁢log⁡k⩾n−usubscript021subscript0superscript _0 2(1- _0) k n^-uδ0 square-root start_ARG 2 ( 1 - ρ0 ) log k end_ARG ⩾ n- u) ⩽O⁢(k−(1−δ0)2⁢(1+n−δ)⁢(log⁡k)n−δ−δ02)absentsuperscriptsuperscript1subscript021superscriptsuperscriptsuperscriptsubscript02 O (k^-(1- _0)^2(1+n^-δ)( k% )^n^-δ- _02)⩽ O ( start_ARG k- ( 1 - δ0 ) start_POSTSUPERSCRIPT 2 ( 1 + n- δ ) end_POSTSUPERSCRIPT ( log k )n start_POSTSUPERSCRIPT - δ - divide start_ARG δ0 end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT end_ARG ) (Theorem 3.6) ⩽O⁢(k−(1−δ0)2⁢(log⁡k)−δ02)⁢(1+on⁢(1))absentsuperscriptsuperscript1subscript02superscriptsubscript021subscript1 O (k^-(1- _0)^2( k)^- % _02)(1+o_n(1))⩽ O ( start_ARG k- ( 1 - δ0 ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( log k )- divide start_ARG δ0 end_ARG start_ARG 2 end_ARG end_ARG ) ( 1 + oitalic_n ( 1 ) ) (kn−δ=1+on⁢(1)superscriptsuperscript1subscript1k^n^-δ=1+o_n(1)kitalic_n start_POSTSUPERSCRIPT - δ end_POSTSUPERSCRIPT = 1 + oitalic_n ( 1 )) ⩽O⁢(k−(1−δ0)2)absentsuperscriptsuperscript1subscript02 O(k^-(1- _0)^2)⩽ O ( k- ( 1 - δ0 ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ) ((log⁡k)−δ0/2=1−ok⁢(1)superscriptsubscript021subscript1( k)^- _0/2=1-o_k(1)( log k )- δ0 / 2 = 1 - oitalic_k ( 1 )) ⩽O⁢(1k).absent1 O ( 1k)\,.⩽ O ( start_ARG divide start_ARG 1 end_ARG start_ARG k end_ARG end_ARG ) . (δ0=1/log⁡ksubscript01 _0=1/ kδ0 = 1 / log k) Inverting this bound, we see that ⁢[maxi∈[k]⁡gi>n−u]⩾1−O⁢(1k).delimited-[]subscriptdelimited-[]subscriptsuperscript11 Pr[ _i∈[k]g_i>n^-u] 1-O% ( 1k)\,.Pr [ maxitalic_i ∈ [ k ] gitalic_i > n- u ] ⩾ 1 - O ( start_ARG divide start_ARG 1 end_ARG start_ARG k end_ARG end_ARG ) . For the upper bound on the above probability, we can use Slepian’s lemma and then compare to Gaussians gi¯subscript g_iunder¯ start_ARG gitalic_i end_ARG which have correlation 12−n−δ12superscript 12-n^-δdivide start_ARG 1 end_ARG start_ARG 2 end_ARG - n- δ. Writing it all out, we have ⁢[maxi∈[k]⁡gi⩽n−u]delimited-[]subscriptdelimited-[]subscriptsuperscript Pr[ _i∈[k]g_i n^-u]Pr [ maxitalic_i ∈ [ k ] gitalic_i ⩽ n- u ] ⩾⁢[maxi∈[k]⁡gi¯⩽n−u]absentdelimited-[]subscriptdelimited-[]¯subscriptsuperscript Pr[ _i∈[k] g_i% n^-u]⩾ Pr [ maxitalic_i ∈ [ k ] under¯ start_ARG gitalic_i end_ARG ⩽ n- u ] (Slepian’s lemma) ⩾⁢[maxi∈[k]⁡gi¯⩽0]absentdelimited-[]subscriptdelimited-[]¯subscript0 Pr[ _i∈[k] g_i% 0]⩾ Pr [ maxitalic_i ∈ [ k ] under¯ start_ARG gitalic_i end_ARG ⩽ 0 ] ⩾Ω⁢(1k1+n−δ)absentΩ1superscript1superscript ( 1k^1+n^-δ)⩾ Ω ( start_ARG divide start_ARG 1 end_ARG start_ARG k1 + n start_POSTSUPERSCRIPT - δ end_POSTSUPERSCRIPT end_ARG end_ARG ) ([Pinasco et al., 2021, Theorem 2.1]) ⩾Ω⁢(1k).absentΩ1 ( 1k)\,.⩾ Ω ( start_ARG divide start_ARG 1 end_ARG start_ARG k end_ARG end_ARG ) . (kn−δ=1+on⁢(1)superscriptsuperscript1subscript1k^n^-δ=1+o_n(1)kitalic_n start_POSTSUPERSCRIPT - δ end_POSTSUPERSCRIPT = 1 + oitalic_n ( 1 )) from which we get a nearly matching upper bound on the probability of 1−Ω⁢(1k)1Ω11- ( 1k)1 - Ω ( divide start_ARG 1 end_ARG start_ARG k end_ARG ). ∎ We return to the proof of Theorem 3.6. Proof of Theorem 3.6. The lower bound directly follows from the proof of Lopes and Yao [2022], so we focus on proving the upper bound. We remark that the constant hidden by Θ Θ can be pinpointed to (1+oN⁢(1))⁢ρ01−ρ01subscript1subscript01subscript0(1+o_N(1)) _01- _0( 1 + oitalic_N ( 1 ) ) square-root start_ARG divide start_ARG ρ0 end_ARG start_ARG 1 - ρ0 end_ARG end_ARG, but we will not discuss this further. To reduce confusion, we will attempt to follow the notation and treatment from Lopes and Yao [2022]. To prove the upper bound, we can use Slepian’s lemma to reduce to the case where ⁢[gi⁢gj]=ρ0delimited-[]subscriptsubscriptsubscript0 E[g_ig_j]= _0E [ gitalic_i gitalic_j ] = ρ0 for all i≠ji≠ ji ≠ j. Then, we can explicitly decompose gi=ρ0⁢x+1−ρ0⁢hi,subscriptsubscript01subscript0subscriptℎg_i= _0x+ 1- _0h_i,gitalic_i = square-root start_ARG ρ0 end_ARG x + square-root start_ARG 1 - ρ0 end_ARG hitalic_i , where x,hisubscriptℎx,h_ix , hitalic_i are iid standard Gaussians. Via this decomposition, we can write an integral representation for the desired probability: ⁢[maxi∈[N]⁡gi⩽tN]delimited-[]subscriptdelimited-[]subscriptsubscript Pr[ _i∈[N]g_i t_N]Pr [ maxitalic_i ∈ [ N ] gitalic_i ⩽ titalic_N ] =∫−∞ψ⁢(s)⁢s,absentsuperscriptsubscriptdifferential-d = _-∞^∞ψ(s)ds,= ∫- ∞ ψ ( s ) d s , (30) ψ⁢(s) ψ(s)ψ ( s ) ≜ϕ⁢(s)⁢ΦN⁢(tN−ρ0⁢s1−ρ0)⁢d⁢s,≜absentitalic-ϕsuperscriptΦsubscriptsubscript01subscript0 φ(s) ^N ( t_N- _0s% 1- _0)ds,≜ ϕ ( s ) Φitalic_N ( start_ARG divide start_ARG titalic_N - square-root start_ARG ρ0 end_ARG s end_ARG start_ARG square-root start_ARG 1 - ρ0 end_ARG end_ARG end_ARG ) d s , (31) where ϕ⁢(⋅)italic-ϕ⋅φ(·)ϕ ( ⋅ ) and Φ⁢(⋅)Φ⋅ (·)Φ ( ⋅ ) are the standard Gaussian density and CDF, respectively. We will estimate this integral by splitting it into a couple pieces. To this end, we will bound the Gaussian CDF using Mills’ inequality. For any t>00t>0t > 0, we have t1+t2⁢ϕ⁢(t)⩽1−Φ⁢(t)=Φ⁢(−t)⩽1t⁢ϕ⁢(t).1superscript2italic-ϕ1Φ1italic-ϕ t1+t^2φ(t) 1- (t)= (-t) % 1tφ(t)\,.divide start_ARG t end_ARG start_ARG 1 + t2 end_ARG ϕ ( t ) ⩽ 1 - Φ ( t ) = Φ ( - t ) ⩽ divide start_ARG 1 end_ARG start_ARG t end_ARG ϕ ( t ) . We will need the following multiplicative estimate on ΦN⁢(⋅)superscriptΦ⋅ ^N(·)Φitalic_N ( ⋅ ), which holds for any δ∈[0,1]01δ∈[0,1]δ ∈ [ 0 , 1 ]: ΦN⁢(2⁢log⁡N⁢(1−δ))superscriptΦ21 ^N( 2 N(1-δ))Φitalic_N ( square-root start_ARG 2 log N end_ARG ( 1 - δ ) ) =(1−Φ⁢(−2⁢log⁡N⁢(1−δ)))Nabsentsuperscript1Φ21 =(1- (- 2 N(1-δ)))^N= ( 1 - Φ ( - square-root start_ARG 2 log N end_ARG ( 1 - δ ) ) )N ⩽(1−12⁢π⁢N(1−δ)2⁢(2⁢log⁡N⁢(1−δ)+1))Nabsentsuperscript112superscriptsuperscript12211 (1- 1 2πN^(1-δ)^2(% 2 N(1-δ)+1))^N⩽ ( start_ARG 1 - divide start_ARG 1 end_ARG start_ARG square-root start_ARG 2 π end_ARG N( 1 - δ ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( square-root start_ARG 2 log N end_ARG ( 1 - δ ) + 1 ) end_ARG end_ARG )N ⩽exp⁡(−N1−(1−δ)22⁢π⁢(2⁢log⁡N+1)).absentsuperscript1superscript12221 (- N^1-(1-δ)^2 2π( 2% N+1))\,.⩽ exp ( start_ARG - divide start_ARG N1 - ( 1 - δ ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG square-root start_ARG 2 π end_ARG ( square-root start_ARG 2 log N end_ARG + 1 ) end_ARG end_ARG ) . (32) The above bound (32) yields nontrivial bounds only when δ>c⁢log⁡log⁡Nlog⁡Nδ> c N Nδ > divide start_ARG c log log N end_ARG start_ARG log N end_ARG for a constant c>1414c> 14c > divide start_ARG 1 end_ARG start_ARG 4 end_ARG, and decays superpolynomially if c>3434c> 34c > divide start_ARG 3 end_ARG start_ARG 4 end_ARG. These bounds motivate splitting (31) into a few different pieces, based on which term dominates the behavior of the integral ψ. To this end, let cN,dN>0subscriptsubscript0c_N,d_N>0citalic_N , ditalic_N > 0 be parameters we specify shortly. Then we split the integral into three pieces: ∫−∞−cNψ⁢(s)⁢ds+∫−cN−dNψ⁢(s)⁢ds+∫−dN∞ψ⁢(s)⁢ds.superscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsuperscriptsubscriptsubscript _-∞^-c_Nψ(s) s+ _-c_N^-d_% Nψ(s) s+ _-d_N^∞ψ(s) s.∫- ∞- citalic_N ψ ( s ) d start_ARG s end_ARG + ∫- c start_POSTSUBSCRIPT N end_POSTSUBSCRIPT- ditalic_N ψ ( s ) d start_ARG s end_ARG + ∫- d start_POSTSUBSCRIPT N end_POSTSUBSCRIPT∞ ψ ( s ) d start_ARG s end_ARG . As in Lopes and Yao [2022], we define α0≜(1ρ0−1)⁢(1−δ0)2≜subscript01subscript01superscript1subscript02 _0 ( 1 _0-1)(1- _0)^2α0 ≜ ( divide start_ARG 1 end_ARG start_ARG ρ0 end_ARG - 1 ) ( 1 - δ0 )2 and β0≜α01−δ0≜subscript0subscript01subscript0 _0 _01- _0β0 ≜ divide start_ARG α0 end_ARG start_ARG 1 - δ0 end_ARG. Then the ultimate bound we want to prove is ∫−∞ψ⁢(s)⁢ds⩽O⁢(N−α0⁢(log⁡N)β0−12)superscriptsubscriptsuperscriptsubscript0superscriptsubscript012 _-∞^∞ψ(s) s O (N^- _% 0( N) _0-12)∫- ∞ ψ ( s ) d start_ARG s end_ARG ⩽ O ( start_ARG N- α0 ( log N )divide start_ARG β0 - 1 end_ARG start_ARG 2 end_ARG end_ARG ). We explain the choice of cN,dN>0subscriptsubscript0c_N,d_N>0citalic_N , ditalic_N > 0 as follows. For succinctness, we introduce the following two functions on ℝRblackboard_R: sN⁢(u)subscript s_N(u)sitalic_N ( u ) ≜−2⁢(1−ρ0)⁢log⁡Nρ0⁢u=−2⁢α0⁢log⁡N1−δ0⁢u≜absent21subscript0subscript02subscript01subscript0 - 2(1- _0) N _0u=- % 2 _0 N1- _0u≜ - square-root start_ARG divide start_ARG 2 ( 1 - ρ0 ) log N end_ARG start_ARG ρ0 end_ARG end_ARG u = - divide start_ARG square-root start_ARG 2 α0 log N end_ARG end_ARG start_ARG 1 - δ0 end_ARG u fN⁢(s)subscript f_N(s)fitalic_N ( s ) ≜tN−ρ0⁢s1−ρ0,≜absentsubscriptsubscript01subscript0 t_N- _0s 1- _0,≜ divide start_ARG titalic_N - square-root start_ARG ρ0 end_ARG s end_ARG start_ARG square-root start_ARG 1 - ρ0 end_ARG end_ARG , and reparameterize the interval [−cN,−dN]subscriptsubscript[-c_N,-d_N][ - citalic_N , - ditalic_N ] as another interval ℐII on u below. 1. We want cNsubscriptc_Ncitalic_N to satisfy Φ⁢(−cN)=O⁢(N−α0⁢(log⁡N)β0−12)Φsubscriptsuperscriptsubscript0superscriptsubscript012 (-c_N)=O (N^- _0( N) _0-12)Φ ( - citalic_N ) = O ( start_ARG N- α0 ( log N )divide start_ARG β0 - 1 end_ARG start_ARG 2 end_ARG end_ARG ). Given this, the first term can be bounded by ∫−∞−cNψ⁢(s)⁢ds⩽∫−∞−cNϕ⁢(s)⁢ds=Φ⁢(−cN)⩽O⁢(1Nα0⁢(log⁡N)β0−12).superscriptsubscriptsubscriptsuperscriptsubscriptsubscriptitalic-ϕΦsubscript1superscriptsubscript0superscriptsubscript012 _-∞^-c_Nψ(s) s _-∞^-c_N% φ(s) s= (-c_N) O ( 1N _0% ( N) _0-12).∫- ∞- citalic_N ψ ( s ) d start_ARG s end_ARG ⩽ ∫- ∞- citalic_N ϕ ( s ) d start_ARG s end_ARG = Φ ( - citalic_N ) ⩽ O ( start_ARG divide start_ARG 1 end_ARG start_ARG Nitalic_α0 end_ARG ( log N )divide start_ARG β0 - 1 end_ARG start_ARG 2 end_ARG end_ARG ) . By inverting Mills’ inequality, we see that it suffices to pick cNsubscript c_Ncitalic_N ≜2⁢(1−ρ0)⁢log⁡Nρ0⁢(1−δ0−log⁡log⁡N4⁢log⁡N)≜absent21subscript0subscript01subscript04 2(1- _0) N _0 (1-% _0- N4 N)≜ square-root start_ARG divide start_ARG 2 ( 1 - ρ0 ) log N end_ARG start_ARG ρ0 end_ARG end_ARG ( start_ARG 1 - δ0 - divide start_ARG log log N end_ARG start_ARG 4 log N end_ARG end_ARG ) (33) =sN⁢(1−δ0−log⁡log⁡N4⁢log⁡N).absentsubscript1subscript04 =s_N (1- _0- N4 N)\,.= sitalic_N ( start_ARG 1 - δ0 - divide start_ARG log log N end_ARG start_ARG 4 log N end_ARG end_ARG ) . Indeed, we have for sufficiently large N that Φ⁢(−cN)⩽ϕ⁢(cN)cNΦsubscriptitalic-ϕsubscriptsubscript (-c_N) φ(c_N)c_NΦ ( - citalic_N ) ⩽ divide start_ARG ϕ ( citalic_N ) end_ARG start_ARG citalic_N end_ARG ⩽exp⁡(−α0⁢log⁡N⁢(1−log⁡log⁡N4⁢(1−δ0)⁢log⁡N2))2⁢π⁢2⁢(1ρ0−1)⁢log⁡N⁢(1−oN⁢(1))absentsubscript01superscript41subscript02221subscript011subscript1 (- _0 N(1- N4(1-% _0) N^2)) 2π 2( 1 _0-1) N(1-o% _N(1))⩽ divide start_ARG exp ( start_ARG - α0 log N ( 1 - divide start_ARG log log N end_ARG start_ARG 4 ( 1 - δ0 ) log N end_ARG2 ) end_ARG ) end_ARG start_ARG square-root start_ARG 2 π end_ARG square-root start_ARG 2 ( divide start_ARG 1 end_ARG start_ARG ρ0 end_ARG - 1 ) log N end_ARG ( 1 - oitalic_N ( 1 ) ) end_ARG ⩽ρ01−ρ0⁢N−α0⁢(log⁡N)β0−12absentsubscript01subscript0superscriptsubscript0superscriptsubscript012 _01- _0N^- _0( N% ) _0-12⩽ square-root start_ARG divide start_ARG ρ0 end_ARG start_ARG 1 - ρ0 end_ARG end_ARG N- α0 ( log N )divide start_ARG β0 - 1 end_ARG start_ARG 2 end_ARG (β0=α01−δ0subscript0subscript01subscript0 _0= _01- _0β0 = divide start_ARG α0 end_ARG start_ARG 1 - δ0 end_ARG) 2. We pick dNsubscriptd_Nditalic_N such that ΦN⁢(fN⁢(−dN))≪1Nα0⁢(log⁡N)β0−12much-less-thansuperscriptΦsubscriptsubscript1superscriptsubscript0superscriptsubscript012 ^N(f_N(-d_N)) 1N _0( N) _0-1% 2Φitalic_N ( fitalic_N ( - ditalic_N ) ) ≪ divide start_ARG 1 end_ARG start_ARG Nitalic_α0 end_ARG ( log N )divide start_ARG β0 - 1 end_ARG start_ARG 2 end_ARG. Given this, by monotonicity we evidently have ∫−dN∞ψ⁢(s)⁢ds⩽ΦN⁢(fN⁢(−dN))≪1Nα0⁢(log⁡N)β0−12.superscriptsubscriptsubscriptsuperscriptΦsubscriptsubscriptmuch-less-than1superscriptsubscript0superscriptsubscript012 _-d_N^∞ψ(s) s ^N(f_N(-d_N))% 1N _0( N) _0-12.∫- d start_POSTSUBSCRIPT N end_POSTSUBSCRIPT∞ ψ ( s ) d start_ARG s end_ARG ⩽ Φitalic_N ( fitalic_N ( - ditalic_N ) ) ≪ divide start_ARG 1 end_ARG start_ARG Nitalic_α0 end_ARG ( log N )divide start_ARG β0 - 1 end_ARG start_ARG 2 end_ARG . Owing to (32), to achieve the desired superpolynomial decay it suffices to pick dN⩾0subscript0d_N 0ditalic_N ⩾ 0 such that tN+ρ0⁢dN1−ρ0⩽2⁢log⁡N⁢(1−(34+ε)⁢log⁡log⁡Nlog⁡N)subscriptsubscript0subscript1subscript02134 t_N+ _0d_N 1- _0 2 N(1-% ( 34+ ) N N)divide start_ARG titalic_N + square-root start_ARG ρ0 end_ARG ditalic_N end_ARG start_ARG square-root start_ARG 1 - ρ0 end_ARG end_ARG ⩽ square-root start_ARG 2 log N end_ARG ( 1 - ( divide start_ARG 3 end_ARG start_ARG 4 end_ARG + ε ) divide start_ARG log log N end_ARG start_ARG log N end_ARG ), where ε>00 >0ε > 0 is a constant. Recalling that tN=δ0⁢2⁢(1−ρ0)⁢log⁡Nsubscriptsubscript021subscript0t_N= _0 2(1- _0) Ntitalic_N = δ0 square-root start_ARG 2 ( 1 - ρ0 ) log N end_ARG, we see that fN⁢(−dN)=2⁢log⁡N⁢δ0+ρ01−ρ0⁢dNsubscriptsubscript2subscript0subscript01subscript0subscriptf_N(-d_N)= 2 N _0+ _0 1- _0% d_Nfitalic_N ( - ditalic_N ) = square-root start_ARG 2 log N end_ARG δ0 + divide start_ARG square-root start_ARG ρ0 end_ARG end_ARG start_ARG square-root start_ARG 1 - ρ0 end_ARG end_ARG ditalic_N. Hence the desired inequality holds by picking dNsubscript d_Nditalic_N ≜2⁢(1−ρ0)⁢log⁡Nρ0⁢(1−δ0−(34+ε)⁢log⁡log⁡Nlog⁡N)≜absent21subscript0subscript01subscript034 2(1- _0) N _0 (1-% _0-( 34+ ) N N)≜ square-root start_ARG divide start_ARG 2 ( 1 - ρ0 ) log N end_ARG start_ARG ρ0 end_ARG end_ARG ( start_ARG 1 - δ0 - ( divide start_ARG 3 end_ARG start_ARG 4 end_ARG + ε ) divide start_ARG log log N end_ARG start_ARG log N end_ARG end_ARG ) (34) =sN⁢(1−δ0−(34+ε)⁢log⁡log⁡Nlog⁡N).absentsubscript1subscript034 =s_N (1- _0-( 34+ ) % N N)\,.= sitalic_N ( start_ARG 1 - δ0 - ( divide start_ARG 3 end_ARG start_ARG 4 end_ARG + ε ) divide start_ARG log log N end_ARG start_ARG log N end_ARG end_ARG ) . Given the above choices of cN,dNsubscriptsubscriptc_N,d_Ncitalic_N , ditalic_N in Eqs. 33 and 34, we see that the interval ℐII that u belongs to is ℐ≜[1−δ0−(34+ε)⁢log⁡log⁡Nlog⁡N,1−δ0−14⋅log⁡log⁡Nlog⁡N]≜ℐ1subscript0341subscript0⋅14I [1- _0-( 34+ ) N% N,1- _0- 14· N N]I ≜ [ 1 - δ0 - ( divide start_ARG 3 end_ARG start_ARG 4 end_ARG + ε ) divide start_ARG log log N end_ARG start_ARG log N end_ARG , 1 - δ0 - divide start_ARG 1 end_ARG start_ARG 4 end_ARG ⋅ divide start_ARG log log N end_ARG start_ARG log N end_ARG ]. Hence, it is natural to reparameterize the integral in terms of η∈[0,12+ε]012η∈[0, 12+ ]η ∈ [ 0 , divide start_ARG 1 end_ARG start_ARG 2 end_ARG + ε ] via the following change of variables uN⁢(η)subscript u_N(η)uitalic_N ( η ) ≜1−δ0−(η+14)⁢log⁡log⁡Nlog⁡N,≜absent1subscript014 1- _0- (η+ 14) N N,≜ 1 - δ0 - divide start_ARG ( η + divide start_ARG 1 end_ARG start_ARG 4 end_ARG ) log log N end_ARG start_ARG log N end_ARG , and an easy computation yields ds=sN′⁢(uN⁢(η))⁢uN′⁢(η)⁢dη=2⁢α01−δ0⋅log⁡log⁡Nlog⁡N⁢dη.superscriptsubscript′subscriptsuperscriptsubscript′⋅2subscript01subscript0 s=s_N (u_N(η))u_N (η) % η= 2 _01- _0· N N% η\,.d start_ARG s end_ARG = sitalic_N′ ( uitalic_N ( η ) ) uitalic_N′ ( η ) d start_ARG η end_ARG = divide start_ARG square-root start_ARG 2 α0 end_ARG end_ARG start_ARG 1 - δ0 end_ARG ⋅ divide start_ARG log log N end_ARG start_ARG log N end_ARG d start_ARG η end_ARG . It is not hard to see that uN⁢([0,12+ε])=[−cN,−dN]subscript012subscriptsubscriptu_N([0, 12+ ])=[-c_N,-d_N]uitalic_N ( [ 0 , divide start_ARG 1 end_ARG start_ARG 2 end_ARG + ε ] ) = [ - citalic_N , - ditalic_N ]. Let us introduce the following abbreviations for the compositions of our changes of variable: s~N⁢(η)subscript~ s_N(η)over~ start_ARG s end_ARGN ( η ) ≜sN⁢(uN⁢(η))≜absentsubscriptsubscript s_N(u_N(η))≜ sitalic_N ( uitalic_N ( η ) ) f~N⁢(η)subscript~ f_N(η)over~ start_ARG f end_ARGN ( η ) ≜fN⁢(s~N⁢(η))=2⁢log⁡N⁢(1−(η+14)⁢log⁡log⁡Nlog⁡N).≜absentsubscriptsubscript~2114 f_N( s_N(η))= 2 N % (1- (η+ 14) N N).≜ fitalic_N ( over~ start_ARG s end_ARGN ( η ) ) = square-root start_ARG 2 log N end_ARG ( start_ARG 1 - divide start_ARG ( η + divide start_ARG 1 end_ARG start_ARG 4 end_ARG ) log log N end_ARG start_ARG log N end_ARG end_ARG ) . In this notation, we have ∫−cN−dNψ⁢(s)⁢ds=2⁢α01−δ0⋅log⁡log⁡Nlog⁡N⁢∫01/2+εψ⁢(s~N⁢(η))⁢dη.superscriptsubscriptsubscriptsubscript⋅2subscript01subscript0superscriptsubscript012subscript~ _-c_N^-d_Nψ(s) s= 2 _01-% _0· N N _0^1/2+ ψ(% s_N(η)) η\,.∫- c start_POSTSUBSCRIPT N end_POSTSUBSCRIPT- ditalic_N ψ ( s ) d start_ARG s end_ARG = divide start_ARG square-root start_ARG 2 α0 end_ARG end_ARG start_ARG 1 - δ0 end_ARG ⋅ divide start_ARG log log N end_ARG start_ARG log N end_ARG ∫01 / 2 + ε ψ ( over~ start_ARG s end_ARGN ( η ) ) d start_ARG η end_ARG . Now the argument as in [Lopes and Yao, 2022, Eq. 4.22] shows that this integral is O⁢(N−α0⁢(log⁡N)β0−12)superscriptsubscript0superscriptsubscript012O(N^- _0( N) _0-12)O ( N- α0 ( log N )divide start_ARG β0 - 1 end_ARG start_ARG 2 end_ARG ), which completes the proof of the first part of the theorem.666To be explicit, their calculation requires us to verify that 2⁢log⁡N⁢(δ0+uN⁢(η))=ω⁢(1)2subscript0subscript1 2 N( _0+u_N(η))=ω(1)square-root start_ARG 2 log N end_ARG ( δ0 + uitalic_N ( η ) ) = ω ( 1 ), which is certainly true. However, let us give an alternative proof which is a bit simpler. We will estimate the integral by performing a Riemann sum with subintervals of width τ>00τ>0τ > 0, which we pick to satisfy β0⁢τ<1subscript01 _0τ<1β0 τ < 1 and such that τ evenly divides 12+ε12 12+ start_ARG 1 end_ARG start_ARG 2 end_ARG + ε. Then, we have ∫01/2+εψ⁢(s~N⁢(η))⁢dη⩽∑k=01/2+ετ∫k⁢τ(k+1)⁢τψ⁢(s~N⁢(η))⁢dη.superscriptsubscript012subscript~superscriptsubscript012superscriptsubscript1subscript~ _0^1/2+ ψ( s_N(η))% η _k=0 1/2+ τ _k% τ^(k+1)τψ( s_N(η)) η\,.∫01 / 2 + ε ψ ( over~ start_ARG s end_ARGN ( η ) ) d start_ARG η end_ARG ⩽ ∑k = 0divide start_ARG 1 / 2 + ε end_ARG start_ARG τ end_ARG ∫k τ( k + 1 ) τ ψ ( over~ start_ARG s end_ARGN ( η ) ) d start_ARG η end_ARG . By monotonicity, for any k we have ∫k⁢τ(k+1)⁢τψ⁢(s~N⁢(η))⁢dηsuperscriptsubscript1subscript~ _kτ^(k+1)τψ( s_N(η))% η∫k τ( k + 1 ) τ ψ ( over~ start_ARG s end_ARGN ( η ) ) d start_ARG η end_ARG ⩽τ⋅ϕ⁢(s~N⁢((k+1)⁢τ))⁢ΦN⁢(f~N⁢(k⁢τ))absent⋅italic-ϕsubscript~1superscriptΦsubscript~ τ·φ( s_N((k+1)τ)) ^N(% f_N(kτ))⩽ τ ⋅ ϕ ( over~ start_ARG s end_ARGN ( ( k + 1 ) τ ) ) Φitalic_N ( over~ start_ARG f end_ARGN ( k τ ) ) ⩽C⋅τ⋅N−α0⁢(log⁡N)β0⋅(12+2⁢(k+1)⁢τ)⁢exp⁡(−(log⁡N)2⁢k⁢τ),absent⋅superscriptsubscript0superscript⋅subscript01221superscript2 C·τ· N^- _0( N) _0·% ( 12+2(k+1)τ) (-( N)^2kτ),⩽ C ⋅ τ ⋅ N- α0 ( log N )β0 ⋅ ( divide start_ARG 1 end_ARG start_ARG 2 end_ARG + 2 ( k + 1 ) τ ) exp ( start_ARG - ( log N )2 k τ end_ARG ) , where C is a universal constant. Here, the last line used ϕ⁢(s~N⁢(η))=O⁢(N−α0⁢(log⁡N)2⁢β0⋅(14+η))italic-ϕsubscript~superscriptsubscript0superscript⋅2subscript014φ( s_N(η))=O(N^- _0( N)^2 _0·(% 14+η))ϕ ( over~ start_ARG s end_ARGN ( η ) ) = O ( N- α0 ( log N )2 β0 ⋅ ( divide start_ARG 1 end_ARG start_ARG 4 end_ARG + η ) ) and ΦN⁢(f~N⁢(η))=O⁢(exp⁡(−log⁡N)2⁢η)superscriptΦsubscript~superscript2 ^N( f_N(η))=O( (- N)^2η)Φitalic_N ( over~ start_ARG f end_ARGN ( η ) ) = O ( exp ( start_ARG - log N end_ARG )2 η ). If 2⁢β0⁢(k+1)⁢τ<122subscript01122 _0(k+1)τ< 122 β0 ( k + 1 ) τ < divide start_ARG 1 end_ARG start_ARG 2 end_ARG, then log⁡log⁡Nlog⁡N⁢(log⁡N)2⁢β0⁢(k+1)⁢τ≪(log⁡N)−12much-less-thansuperscript2subscript01superscript12 N N( N)^2 _0(k+1)τ ( N)^- 1% 2divide start_ARG log log N end_ARG start_ARG log N end_ARG ( log N )2 β0 ( k + 1 ) τ ≪ ( log N )- divide start_ARG 1 end_ARG start_ARG 2 end_ARG, as desired. On the other hand, if 2⁢β0⁢(k+1)⁢τ⩾122subscript01122 _0(k+1)τ 122 β0 ( k + 1 ) τ ⩾ divide start_ARG 1 end_ARG start_ARG 2 end_ARG, then as β0⁢τ<18subscript018 _0τ< 18β0 τ < divide start_ARG 1 end_ARG start_ARG 8 end_ARG, we have 2⁢k⁢τ⩾14⁢β0>0214subscript002kτ 14 _0>02 k τ ⩾ divide start_ARG 1 end_ARG start_ARG 4 β0 end_ARG > 0. Since exp⁡(−(log⁡N)2⁢k⁢τ)superscript2 (-( N)^2kτ)exp ( start_ARG - ( log N )2 k τ end_ARG ) dominates any polylog terms, so it is not hard to see that the total contribution here is ≪N−α0⁢(log⁡N)β0−12much-less-thanabsentsuperscriptsubscript0superscriptsubscript012 N^- _0( N) _0-12≪ N- α0 ( log N )divide start_ARG β0 - 1 end_ARG start_ARG 2 end_ARG. Hence, we conclude that ∫−cN−dNψ⁢(s)⁢ds⩽ρ01−ρ0⁢N−α0⁢(log⁡N)β0−12superscriptsubscriptsubscriptsubscriptsubscript01subscript0superscriptsubscript0superscriptsubscript012 _-c_N^-d_Nψ(s) s _01-% _0N^- _0( N) _0-12∫- c start_POSTSUBSCRIPT N end_POSTSUBSCRIPT- ditalic_N ψ ( s ) d start_ARG s end_ARG ⩽ square-root start_ARG divide start_ARG ρ0 end_ARG start_ARG 1 - ρ0 end_ARG end_ARG N- α0 ( log N )divide start_ARG β0 - 1 end_ARG start_ARG 2 end_ARG, as desired. ∎ Appendix F Experiments In this section, we describe the simulations we conducted to validate the theory. We generated Gaussian data following the subset ensembles specified in the figures, and constructed two linear models from them: the MNI classifier and the simple averaging classifier. In the averaging classifier, we average over the positive examples of a label, which approximates the behavior of the first few iterations of gradient descent. In contrast, the MNI classifier governs the asymptotic behavior of gradient descent. The weak-to-strong behavior for these two learning algorithms was compared to two other baselines: the weak accuracy for subscript f_ weakitalic_fsansserif_weak and the accuracy for the strong model trained on m clean labels: subscript f_ strongitalic_fsansserif_strong. The test accuracies were evaluated on n=100subscript100n_ test=100nsansserif_test = 100 fresh datapoints. We ran 8888 independent trials to train subscript f_ weakitalic_fsansserif_weak with n=5050n=50n = 50 so that we could explore how the weak-to-strong behavior scales with p and u. For each subscript f_ weakitalic_fsansserif_weak, we conducted 16 independent trials to train subscript f_ w2sitalic_fsansserif_w2s. We swept out u using five equally spaced points in [1,1.3]11.3[1,1.3][ 1 , 1.3 ]. In Figures 3 and 4, we show the results of the averaging and MNI experiments, respectively, for four different slices. In the top row, we show two slices where the theory predicts weak-to-strong generalization to occur for MNI, and in the bottom row we show two slices where the theory predicts failure of weak-to-strong generalization. The error bars show the estimated 95% CI over all sources of uncertainty in the inner and outer loop (subscript f_ w2sitalic_fsansserif_w2s and subscript f_ weakitalic_fsansserif_weak). In both the averaging and MNI plots, the theory successfully predicts whether weak-to-strong generalization occurs. Furthermore, in every plot the ground truth trained strong model subscript f_ strongitalic_fsansserif_strong trained on m clean labels has better test accuracy than subscript f_ weakitalic_fsansserif_weak and subscript f_ w2sitalic_fsansserif_w2s, as expected. Another interesting experimental observation is that the averaging classifier does significantly better than MNI in non-asymptotic settings. This corroborates the view of practitioners of the benefits of early-stopping for gradient descent. (a) (p,q,r)=(2,0.6,0.6),(p,q,r)=(1.4,0.9,0.5)formulae-sequence20.60.6subscriptsubscriptsubscript1.40.90.5(p,q,r)=(2,0.6,0.6),(p_ weak,q_ weak,r_ weak)=(1% .4,0.9,0.5)( p , q , r ) = ( 2 , 0.6 , 0.6 ) , ( psansserif_weak , qsansserif_weak , rsansserif_weak ) = ( 1.4 , 0.9 , 0.5 ). (b) (p,q,r)=(2,0.9,0.4)20.90.4(p,q,r)=(2,0.9,0.4)( p , q , r ) = ( 2 , 0.9 , 0.4 ), (p,q,r)=(1.4,0.9,0.4)subscriptsubscriptsubscript1.40.90.4(p_ weak,q_ weak,r_ weak)=(1.4,0.9,0.4)( psansserif_weak , qsansserif_weak , rsansserif_weak ) = ( 1.4 , 0.9 , 0.4 ) (c) (p,q,r)=(1.5,0.6,0.8),(p,q,r)=(1.4,0.9,0.5)formulae-sequence1.50.60.8subscriptsubscriptsubscript1.40.90.5(p,q,r)=(1.5,0.6,0.8),(p_ weak,q_ weak,r_ weak)=% (1.4,0.9,0.5)( p , q , r ) = ( 1.5 , 0.6 , 0.8 ) , ( psansserif_weak , qsansserif_weak , rsansserif_weak ) = ( 1.4 , 0.9 , 0.5 ). (d) (p,q,r)=(2,0.9,0.4)20.90.4(p,q,r)=(2,0.9,0.4)( p , q , r ) = ( 2 , 0.9 , 0.4 ), (p,q,r)=(1.4,0.9,0.4)subscriptsubscriptsubscript1.40.90.4(p_ weak,q_ weak,r_ weak)=(1.4,0.9,0.4)( psansserif_weak , qsansserif_weak , rsansserif_weak ) = ( 1.4 , 0.9 , 0.4 ) Figure 3: Comparison of test accuracies for four different models using averaging training. The x-axis plots m, the number of additional labeled datapoints. The models are trained using class averaging, which approximates the behavior of the initial few gradient descent iterations. Note how the weak model has low accuracy, whereas the weak-to-strong model and ground truth have higher accuracies that increase as m increases. The top row Figures 3(a) and 3(b) are in a regime where we predict MNI weak-to-strong generalization to succeed, whereas the bottom row Figures 3(c) and 3(d) depict regimes where we expect MNI weak-to-strong generalization to fail. (a) (p,q,r)=(2,0.6,0.6),(p,q,r)=(1.4,0.9,0.5)formulae-sequence20.60.6subscriptsubscriptsubscript1.40.90.5(p,q,r)=(2,0.6,0.6),(p_ weak,q_ weak,r_ weak)=(1% .4,0.9,0.5)( p , q , r ) = ( 2 , 0.6 , 0.6 ) , ( psansserif_weak , qsansserif_weak , rsansserif_weak ) = ( 1.4 , 0.9 , 0.5 ). (b) (p,q,r)=(2,0.9,0.5)20.90.5(p,q,r)=(2,0.9,0.5)( p , q , r ) = ( 2 , 0.9 , 0.5 ), (p,q,r)=(1.4,0.9,0.5)subscriptsubscriptsubscript1.40.90.5(p_ weak,q_ weak,r_ weak)=(1.4,0.9,0.5)( psansserif_weak , qsansserif_weak , rsansserif_weak ) = ( 1.4 , 0.9 , 0.5 ) (c) (p,q,r)=(1.5,0.6,0.8),(p,q,r)=(1.4,0.9,0.5)formulae-sequence1.50.60.8subscriptsubscriptsubscript1.40.90.5(p,q,r)=(1.5,0.6,0.8),(p_ weak,q_ weak,r_ weak)=% (1.4,0.9,0.5)( p , q , r ) = ( 1.5 , 0.6 , 0.8 ) , ( psansserif_weak , qsansserif_weak , rsansserif_weak ) = ( 1.4 , 0.9 , 0.5 ). (d) (p,q,r)=(2,0.9,0.4)20.90.4(p,q,r)=(2,0.9,0.4)( p , q , r ) = ( 2 , 0.9 , 0.4 ), (p,q,r)=(1.4,0.9,0.4)subscriptsubscriptsubscript1.40.90.4(p_ weak,q_ weak,r_ weak)=(1.4,0.9,0.4)( psansserif_weak , qsansserif_weak , rsansserif_weak ) = ( 1.4 , 0.9 , 0.4 ) Figure 4: Comparison of MNI test accuracies for four different models. Observe how the weak-to-strong accuracy increases as m increases. Again, the top row Figures 4(a) and 4(b) are in a regime where we predict MNI weak-to-strong generalization to succeed, whereas the bottom row Figures 4(c) and 4(d) depict regimes where we expect MNI weak-to-strong generalization to fail. The plots corroborate these theoretical predictions. Appendix G Heuristic calculations Recall from the definition of MNI that =⊤⁢−1⁢,subscriptsuperscripttopsuperscript1 f_ strong= X A^-1 y,italic_fsansserif_strong = italic_X⊤ italic_A- 1 italic_y , where ∈±1nsuperscriptplus-or-minus1 y∈ ± 1^nitalic_y ∈ start_ARG ± 1 end_ARG n is a label vector generated by either the true feature x∗subscriptx_*x∗ or a weak feature xsubscriptx_ weakxsansserif_weak and =⁢⊤∈ℝn×nsuperscripttopsuperscriptℝ A= X X ^n× nitalic_A = italic_X italic_X⊤ ∈ blackboard_Rn × n is the Gram matrix. The key to our analysis is studying the survival and contamination of various features when the labels are possibly generated by another feature. Recall that we performed a basis change ↦⁢Umaps-to X XUitalic_X ↦ italic_X U, so that the strong features are drawn iid from N⁢(0,Λ)0ΛN(0, )N ( 0 , Λ ). Writing the transformed data matrix now as where each i∼N⁢(0,In)similar-tosubscript0subscript z_i N(0,I_n)italic_zitalic_i ∼ N ( 0 , Iitalic_n ), we obtain for any unit norm v∈ℝDsuperscriptℝv ^Dv ∈ blackboard_RD that SU⁢(v)SUv SU(v)SU ( v ) =∑i∈[d]λi⁢i⊤⁢−1⁢⟨i,v⟩absentsubscriptdelimited-[]subscriptsuperscriptsubscripttopsuperscript1expectation-valuesubscript = _i∈[d] _i z_i A^-1 y% v_i,v= ∑i ∈ [ d ] λitalic_i italic_zitalic_i⊤ italic_A- 1 italic_y ⟨ start_ARG italic_vitalic_i , v end_ARG ⟩ (MNI) ⁢(v) CN(v)sansserif_CN ( v ) =∑i∈[d](λi⁢i⊤⁢−1⁢)2⁢(1−⟨i,v⟩2).absentsubscriptdelimited-[]superscriptsubscriptsuperscriptsubscripttopsuperscript121superscriptexpectation-valuesubscript2 = _i∈[d]( _i z_i A^-1 % y)^2(1- v_i,v^2).= square-root start_ARG ∑i ∈ [ d ] ( λitalic_i italic_zitalic_i⊤ italic_A- 1 italic_y )2 ( 1 - ⟨ start_ARG italic_vitalic_i , v end_ARG ⟩2 ) end_ARG . (Orthonormality of isubscript v_iitalic_vitalic_i) Since Aitalic_A is close to d⁢IdsubscriptdI_dd Iitalic_d in the isotropic case, and we are working in the regime where PCA fails to extract the bi-level structure (q+r>11q+r>1q + r > 1), one could hope for the best and pretend that −1=1d⁢Idsuperscript11subscript A^-1= 1dI_ditalic_A- 1 = divide start_ARG 1 end_ARG start_ARG d end_ARG Iitalic_d. This step is not rigorous, but we will justify these approximations in Appendix B. Based on the decompositions in Definition 3, we study the case where =sgn⁢(⟨,⟩)sgnexpectation-value y=sgn( g, w)italic_y = sgn ( ⟨ start_ARG italic_g , italic_w end_ARG ⟩ ) for some ∈RDsuperscript w∈ R^Ditalic_w ∈ Ritalic_D but we want to recover the planted direction ∈ℝDsuperscriptℝ v ^Ditalic_v ∈ blackboard_RD. For a subspace V⊆ℝDsuperscriptℝV ^DV ⊆ blackboard_RD and a vector ∈ℝDsuperscriptℝ u ^Ditalic_u ∈ blackboard_RD, let Vsubscript u_Vitalic_uitalic_V denote the projection of uitalic_u onto V. For axis-aligned subspaces V⊆[d]delimited-[]V [d]V ⊆ [ d ], this just corresponds to restricting to the coordinates in V. To simplify the heuristic calculation, we will make the following assumptions. Assumption 5. Let S=[s]delimited-[]S=[s]S = [ s ] denote the spiked subspace after the basis change, and let α,ρ>00α,ρ>0α , ρ > 0 be parameters possibly depending on n. For any vector ∈ℝdsuperscriptℝ u ^ditalic_u ∈ blackboard_Rd, let T≜i∈S:|ui|=ω⁢(1n)≜subscript:subscript1T_ u i∈ S: u_i=ω( 1% n)Tbold_italic_u ≜ start_ARG i ∈ S : | start_ARG uitalic_i end_ARG | = ω ( divide start_ARG 1 end_ARG start_ARG square-root start_ARG n end_ARG end_ARG ) end_ARG denote the spiked coordinates where uitalic_u is large, and R=S∖TsubscriptsubscriptR_ u=S T_ uRbold_italic_u = S ∖ Tbold_italic_u denote the spiked coordinates where uitalic_u is small. We assume the following holds for T=TsubscriptT=T_ wT = Tbold_italic_w and R=RsubscriptR=R_ wR = Rbold_italic_w: (1) We have ⟨T,T⟩=α⁢‖T‖2⁢‖T‖2expectation-valuesubscriptsubscriptsubscriptnormsubscript2subscriptnormsubscript2 v_T, w_T=α v_T_2 % w_T_2⟨ start_ARG italic_vitalic_T , italic_witalic_T end_ARG ⟩ = α ∥ start_ARG italic_vitalic_T end_ARG ∥2 ∥ start_ARG italic_witalic_T end_ARG ∥2. In other words, vitalic_v and witalic_w have correlation α restricted to the heavy coordinates for witalic_w. (2) We have ∑i∈Tvi2⁢wi2=ρ2⁢‖T‖22subscriptsuperscriptsubscript2superscriptsubscript2superscript2superscriptsubscriptnormsubscript22 _i∈ Tv_i^2w_i^2=ρ^2 w_T_2^2∑i ∈ T vitalic_i2 witalic_i2 = ρ2 ∥ start_ARG italic_witalic_T end_ARG ∥22. Note that by LpsuperscriptL^pLitalic_p norm inequalities we always have ρ2⩽1superscript21ρ^2 1ρ2 ⩽ 1. (3) We have |R|=Ω⁢(s)Ω R= (s)| start_ARG R end_ARG | = Ω ( s ), i.e. a constant fraction of witalic_w’s spiked coordinates are small. Remark G.1. When = v= witalic_v = italic_w, Item (2) can be thought of as a relaxed notion of vitalic_v being 1111-sparse; for a given ρ one should roughly think of vitalic_v as being 1ρ1 1ρdivide start_ARG 1 end_ARG start_ARG ρ end_ARG-sparse. Survival bound. Recall that =sgn⁢(⟨,⟩)sgnexpectation-value y=sgn( g, w)italic_y = sgn ( ⟨ start_ARG italic_g , italic_w end_ARG ⟩ ). By applying the noise stability formula again and the fact that zi⁢j∼N⁢(0,1)similar-tosubscript01z_ij N(0,1)zitalic_i j ∼ N ( 0 , 1 ), we deduce that ⁢[i⊤⁢]delimited-[]superscriptsubscripttop E[ z_i y]E [ italic_zitalic_i⊤ italic_y ] =n⁢[zi⁢j⁢yj]absentdelimited-[]subscriptsubscript =n E[z_ijy_j]= n E [ zitalic_i j yitalic_j ] =⁢[sgn⁢(zi⁢j)=yj]⁢[|zi⁢j|]−(1−⁢[sgn⁢(zi⁢j)=j])⁢[|zi⁢j|]absentdelimited-[]sgnsubscriptsubscriptdelimited-[]subscript1delimited-[]sgnsubscriptsubscriptdelimited-[]subscript = Pr[sgn(z_ij)=y_j] E[% z_ij]-(1- Pr[sgn(z_ij)= y_j])% E[ z_ij]= Pr [ sgn ( zitalic_i j ) = yitalic_j ] E [ | start_ARG zitalic_i j end_ARG | ] - ( 1 - Pr [ sgn ( zitalic_i j ) = italic_yitalic_j ] ) E [ | start_ARG zitalic_i j end_ARG | ] =2π⁢(2⁢[sgn⁢(zi⁢j)=yj]−1)absent22delimited-[]sgnsubscriptsubscript1 = 2π(2 Pr[sgn(z_ij)=y_% j]-1)= square-root start_ARG divide start_ARG 2 end_ARG start_ARG π end_ARG end_ARG ( 2 Pr [ sgn ( zitalic_i j ) = yitalic_j ] - 1 ) =(2π)3/2⁢arcsin⁡wi,absentsuperscript232arcsinesubscript =( 2π)^3/2 w_i,= ( divide start_ARG 2 end_ARG start_ARG π end_ARG )3 / 2 arcsin witalic_i , where in the second to last line we have used the fact that the expected magnitude of a standard Gaussian is 2/π2 2/πsquare-root start_ARG 2 / π end_ARG, and in the last line we have used the noise stability formula. By standard concentration inequalities, the deviations will be of order O⁢(n)O( n)O ( square-root start_ARG n end_ARG ). As the behavior of 2π⁢arcsin⁡(x)≈2π⁢x2arcsine2 2π (x)≈ 2πxdivide start_ARG 2 end_ARG start_ARG π end_ARG arcsin ( start_ARG x end_ARG ) ≈ divide start_ARG 2 end_ARG start_ARG π end_ARG x for small x, we deduce that the expectation will dominate whenever |wi|≫1n) w_i 1 n)| start_ARG witalic_i end_ARG | ≫ divide start_ARG 1 end_ARG start_ARG square-root start_ARG n end_ARG end_ARG ). We will now plug in the bi-level scaling. Recall that λi=λF=a⁢dssubscriptsubscript _i= _F= adsλitalic_i = λitalic_F = divide start_ARG a d end_ARG start_ARG s end_ARG for i∈[s]delimited-[]i∈[s]i ∈ [ s ] and λi=λU=(1−a)⁢d−ssubscriptsubscript1 _i= _U= (1-a)dd-sλitalic_i = λitalic_U = divide start_ARG ( 1 - a ) d end_ARG start_ARG d - s end_ARG for i>si>si > s. Thus, ignoring constants, with high probability we should have SU⁢(|)SUconditional SU( v| w)SU ( v | w ) =λF⁢nd⁢∑i∈Tvi⁢arcsin⁡wi±λF⁢nd⁢∑i∈Rvi±λU⁢nd⁢∑i>svi.absentplus-or-minussubscriptsubscriptsubscriptsubscriptarcsinesubscriptsubscriptsubscriptsubscriptsubscriptsubscriptsubscriptsubscript = _Fnd _i∈ T_ wv_i w_i% ± _F nd _i∈ R_ wv_i± _U% nd _i>sv_i\,.= divide start_ARG λitalic_F n end_ARG start_ARG d end_ARG ∑i ∈ T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT vitalic_i arcsin witalic_i ± divide start_ARG λitalic_F square-root start_ARG n end_ARG end_ARG start_ARG d end_ARG ∑i ∈ R start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT vitalic_i ± divide start_ARG λitalic_U square-root start_ARG n end_ARG end_ARG start_ARG d end_ARG ∑i > s vitalic_i . Observe that 2π⁢|x|⩽2π⁢|arcsin⁡x|⩽|x|,22arcsine 2π x 2π x% x,divide start_ARG 2 end_ARG start_ARG π end_ARG | start_ARG x end_ARG | ⩽ divide start_ARG 2 end_ARG start_ARG π end_ARG | start_ARG arcsin x end_ARG | ⩽ | start_ARG x end_ARG | , for all x∈[−1,1]11x∈[-1,1]x ∈ [ - 1 , 1 ]. Consequently, for i∈Tsubscripti∈ T_ wi ∈ Tbold_italic_w, each summand contributes vi⁢arcsin⁡(wi)≈Θ⁢(vi⁢wi)subscriptarcsinesubscriptΘsubscriptsubscriptv_i (w_i)≈ (v_iw_i)vitalic_i arcsin ( start_ARG witalic_i end_ARG ) ≈ Θ ( vitalic_i witalic_i ).777The reason this is not an equality is that there might be some heavy wi′⁢ssuperscriptsubscript′w_i switalic_i′ s which disagree in sign with vi′⁢ssuperscriptsubscript′v_i svitalic_i′ s, but for the settings we consider this estimate will be true. Thus, by Item (1) and the definition of λFsubscript _Fλitalic_F, the first term is Θ⁢(α⋅a⁢ns⋅‖T‖2⁢‖T‖2)Θ⋅subscriptnormsubscriptsubscript2subscriptnormsubscriptsubscript2 (α· ans· v_T_ w_2 w_% T_ w_2)Θ ( α ⋅ divide start_ARG a n end_ARG start_ARG s end_ARG ⋅ ∥ start_ARG italic_vitalic_T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥2 ∥ start_ARG italic_witalic_T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥2 ). For the second term, we will upper bound its magnitude by λF⁢nd⁢|∑i∈Rvi|subscriptsubscriptsubscriptsubscript _F nd _i∈ R_ w% v_idivide start_ARG λitalic_F square-root start_ARG n end_ARG end_ARG start_ARG d end_ARG | start_ARG ∑i ∈ R start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT vitalic_i end_ARG | ⩽a⁢ns⋅‖R‖1nabsent⋅subscriptnormsubscriptsubscript1 ans· v_R_ w_1% n⩽ divide start_ARG a n end_ARG start_ARG s end_ARG ⋅ divide start_ARG ∥ start_ARG italic_vitalic_R start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥1 end_ARG start_ARG square-root start_ARG n end_ARG end_ARG ⩽a⁢ns⋅sn⋅‖R‖2absent⋅subscriptnormsubscriptsubscript2 ans· sn· v_R_% w_2⩽ divide start_ARG a n end_ARG start_ARG s end_ARG ⋅ square-root start_ARG divide start_ARG s end_ARG start_ARG n end_ARG end_ARG ⋅ ∥ start_ARG italic_vitalic_R start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥2 (R⊆SsubscriptR_ w SRbold_italic_w ⊆ S) ⩽a⁢ns⋅sn.absent⋅ ans· sn\,.⩽ divide start_ARG a n end_ARG start_ARG s end_ARG ⋅ square-root start_ARG divide start_ARG s end_ARG start_ARG n end_ARG end_ARG . (‖2=1subscriptnorm21 v_2=1∥ start_ARG italic_v end_ARG ∥2 = 1) Finally, for the third term, since |∑i>svi|⩽‖1⩽dsubscriptsubscriptsubscriptnorm1 _i>sv_i v_1 d| start_ARG ∑i > s vitalic_i end_ARG | ⩽ ∥ start_ARG italic_v end_ARG ∥1 ⩽ square-root start_ARG d end_ARG, after plugging in the definition of λUsubscript _Uλitalic_U, we obtain the asymptotics SU⁢(|)≍ans⋅(α⋅‖T‖2⁢‖T‖2±sn)±nd.asymptotically-equalsSUconditionalplus-or-minus⋅ansplus-or-minus⋅subscriptnormsubscriptsubscriptT2subscriptnormsubscriptsubscriptT2snnd SU( v| w) ans· (% α· v_T_ w_2 w_T_ w_2±% sn)± nd\,.SU ( v | w ) ≍ divide start_ARG an end_ARG start_ARG s end_ARG ⋅ ( start_ARG α ⋅ ∥ start_ARG vroman_T start_POSTSUBSCRIPT w end_POSTSUBSCRIPT end_ARG ∥2 ∥ start_ARG wroman_T start_POSTSUBSCRIPT w end_POSTSUBSCRIPT end_ARG ∥2 ± square-root start_ARG divide start_ARG s end_ARG start_ARG n end_ARG end_ARG end_ARG ) ± square-root start_ARG divide start_ARG n end_ARG start_ARG d end_ARG end_ARG . (35) Note that for the first term to dominate, we must have α=Ω⁢(sn)Ωα= ( sn)α = Ω ( divide start_ARG s end_ARG start_ARG n end_ARG ). Also, if we want to improve our estimate on the second term, we can further split it by TsubscriptT_ vTbold_italic_v, we can gain and get deviations of order |∑i∈R∩Tvi+∑i∈R∩Rvi|subscriptsubscriptsubscriptsubscriptsubscriptsubscriptsubscriptsubscript _i∈ R_ w∩ T_ vv_i+ _i% ∈ R_ w∩ R_ vv_i| start_ARG ∑i ∈ R start_POSTSUBSCRIPT italic_w ∩ Tbold_italic_v end_POSTSUBSCRIPT vitalic_i + ∑i ∈ R start_POSTSUBSCRIPT italic_w ∩ Rbold_italic_v end_POSTSUBSCRIPT vitalic_i end_ARG | ⩽‖R∩T‖1+|R|nabsentsubscriptnormsubscriptsubscriptsubscript1subscript v_R_ w∩ T_ v_1+ % R_ v n⩽ ∥ start_ARG italic_vitalic_R start_POSTSUBSCRIPT italic_w ∩ Tbold_italic_v end_POSTSUBSCRIPT end_ARG ∥1 + divide start_ARG | start_ARG Rbold_italic_v end_ARG | end_ARG start_ARG square-root start_ARG n end_ARG end_ARG (Definition of RsubscriptR_ vRbold_italic_v) ⩽‖R∩T‖1+sn,absentsubscriptnormsubscriptsubscriptsubscript1 v_R_ w∩ T_ v_1+ s% n,⩽ ∥ start_ARG italic_vitalic_R start_POSTSUBSCRIPT italic_w ∩ Tbold_italic_v end_POSTSUBSCRIPT end_ARG ∥1 + divide start_ARG s end_ARG start_ARG square-root start_ARG n end_ARG end_ARG , yielding an ultimate relative deviation of ‖R∩T‖1n+snsubscriptnormsubscriptsubscriptsubscript1 v_R_ w∩ T_ v_1 n+ sndivide start_ARG ∥ start_ARG italic_vitalic_R start_POSTSUBSCRIPT italic_w ∩ Tbold_italic_v end_POSTSUBSCRIPT end_ARG ∥1 end_ARG start_ARG square-root start_ARG n end_ARG end_ARG + divide start_ARG s end_ARG start_ARG n end_ARG. This is significantly better if, say |T|=o⁢(s)subscript T_ v=o(s)| start_ARG Tbold_italic_v end_ARG | = o ( s ), as it allows us to beat the contamination bounds which have relative deviations sn snsquare-root start_ARG divide start_ARG s end_ARG start_ARG n end_ARG end_ARG. Contamination bound. For succinctness, introduce the shorthand hi2=(1−vi2)⁢⟨i,⟩2superscriptsubscriptℎ21superscriptsubscript2superscriptexpectation-valuesubscript2h_i^2=(1-v_i^2) z_i, y^2hitalic_i2 = ( 1 - vitalic_i2 ) ⟨ start_ARG italic_zitalic_i , italic_y end_ARG ⟩2. Then the squared contamination is ⁢(|)2superscriptconditional2 CN( v| w)^2sansserif_CN ( italic_v | italic_w )2 =∑i∈[d]λi2⁢(1−vi2)⋅⊤⁢−1⁢i⁢i⊤⁢−1⁢absentsubscriptdelimited-[]⋅superscriptsubscript21superscriptsubscript2superscripttopsuperscript1subscriptsuperscriptsubscripttopsuperscript1 = _i∈[d] _i^2(1-v_i^2)· y % A^-1 z_i z_i A^-1 y= ∑i ∈ [ d ] λitalic_i2 ( 1 - vitalic_i2 ) ⋅ italic_y⊤ italic_A- 1 italic_zitalic_i italic_zitalic_i⊤ italic_A- 1 italic_y ≈1d2⁢∑i∈[d]λi2⁢hi2absent1superscript2subscriptdelimited-[]superscriptsubscript2superscriptsubscriptℎ2 ≈ 1d^2 _i∈[d] _i^2h_i^2≈ divide start_ARG 1 end_ARG start_ARG d2 end_ARG ∑i ∈ [ d ] λitalic_i2 hitalic_i2 (−1≈1d⁢Idsuperscript11subscript A^-1≈ 1dI_ditalic_A- 1 ≈ divide start_ARG 1 end_ARG start_ARG d end_ARG Iitalic_d) =λF2d2⁢∑i∈Thi2+λF2d2⁢∑i∈Rhi2+λU2d2⁢∑i>shi2absentsuperscriptsubscript2superscript2subscriptsubscriptsuperscriptsubscriptℎ2superscriptsubscript2superscript2subscriptsubscriptsuperscriptsubscriptℎ2superscriptsubscript2superscript2subscriptsuperscriptsubscriptℎ2 = _F^2d^2 _i∈ T_ wh_i^2+% _F^2d^2 _i∈ R_ wh_i^2+ _U% ^2d^2 _i>sh_i^2= divide start_ARG λitalic_F2 end_ARG start_ARG d2 end_ARG ∑i ∈ T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT hitalic_i2 + divide start_ARG λitalic_F2 end_ARG start_ARG d2 end_ARG ∑i ∈ R start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT hitalic_i2 + divide start_ARG λitalic_U2 end_ARG start_ARG d2 end_ARG ∑i > s hitalic_i2 ⩽λF2⁢n2d2⁢∑i∈T(1−vi2)⁢(2π⁢arcsin⁡wi)2+λF2⁢nd2⁢∑i∈R(1−vi2)+λU2⁢nd2⁢(d−s),absentsuperscriptsubscript2superscript2superscript2subscriptsubscript1superscriptsubscript2superscript2arcsinesubscript2superscriptsubscript2superscript2subscriptsubscript1superscriptsubscript2superscriptsubscript2superscript2 _F^2n^2d^2 _i∈ T_ w% (1-v_i^2)( 2π w_i)^2+ _F^2nd^2% _i∈ R_ w(1-v_i^2)+ _U^2nd^2(d-s),⩽ divide start_ARG λitalic_F2 n2 end_ARG start_ARG d2 end_ARG ∑i ∈ T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT ( 1 - vitalic_i2 ) ( divide start_ARG 2 end_ARG start_ARG π end_ARG arcsin witalic_i )2 + divide start_ARG λitalic_F2 n end_ARG start_ARG d2 end_ARG ∑i ∈ R start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT ( 1 - vitalic_i2 ) + divide start_ARG λitalic_U2 n end_ARG start_ARG d2 end_ARG ( d - s ) , where in the last line we have used the observation that the expectation of hisubscriptℎh_ihitalic_i dominates if and only if i∈Tsubscripti∈ T_ wi ∈ Tbold_italic_w. The first term can be bounded (up to constants) as (a⁢ns)2⁢∑i∈T(1−vi2)⁢wi2=(a⁢ns)2⁢‖T‖22⁢(1−ρ2).superscript2subscriptsubscript1superscriptsubscript2superscriptsubscript2superscript2superscriptsubscriptnormsubscriptsubscript221superscript2 ( ans)^2 _i∈ T_ w(1-v_i^2)w_i^2=% ( ans)^2 w_T_ w_2^2(1-ρ^2)\,.( start_ARG divide start_ARG a n end_ARG start_ARG s end_ARG end_ARG )2 ∑i ∈ T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT ( 1 - vitalic_i2 ) witalic_i2 = ( start_ARG divide start_ARG a n end_ARG start_ARG s end_ARG end_ARG )2 ∥ start_ARG italic_witalic_T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥22 ( 1 - ρ2 ) . For the second term, we can bound up to constants as (a⁢ns)2⋅1n⋅∑i∈R(1−vi2)⋅superscript21subscriptsubscript1superscriptsubscript2 ( ans)^2· 1n· _i∈ R_ % w(1-v_i^2)( start_ARG divide start_ARG a n end_ARG start_ARG s end_ARG end_ARG )2 ⋅ divide start_ARG 1 end_ARG start_ARG n end_ARG ⋅ ∑i ∈ R start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT ( 1 - vitalic_i2 ) =(a⁢ns)2⋅|R|−‖R‖22nabsent⋅superscript2subscriptsuperscriptsubscriptnormsubscriptsubscript22 = ( ans)^2· R_ w% - v_R_ w_2^2n= ( start_ARG divide start_ARG a n end_ARG start_ARG s end_ARG end_ARG )2 ⋅ divide start_ARG | start_ARG Rbold_italic_w end_ARG | - ∥ start_ARG italic_vitalic_R start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥22 end_ARG start_ARG n end_ARG ⩽(a⁢ns)2⋅|R|n.absent⋅superscript2subscript ( ans)^2· R_% wn\,.⩽ ( start_ARG divide start_ARG a n end_ARG start_ARG s end_ARG end_ARG )2 ⋅ divide start_ARG | start_ARG Rbold_italic_w end_ARG | end_ARG start_ARG n end_ARG . (‖2=1subscriptnorm21 v_2=1∥ start_ARG italic_v end_ARG ∥2 = 1) Finally, the third term can be bounded by nd nddivide start_ARG n end_ARG start_ARG d end_ARG. Putting these together, we conclude that ⁢(|)≍(a⁢ns)⁢(1−ρ2⁢‖T‖2+|R|n)+nd.asymptotically-equalsconditional1superscript2subscriptnormsubscriptsubscript2subscript CN( v| w) ( ans) (% 1-ρ^2 w_T_ w_2+ R_% wn)+ nd\,.sansserif_CN ( italic_v | italic_w ) ≍ ( start_ARG divide start_ARG a n end_ARG start_ARG s end_ARG end_ARG ) ( start_ARG square-root start_ARG 1 - ρ2 end_ARG ∥ start_ARG italic_witalic_T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥2 + square-root start_ARG divide start_ARG | start_ARG Rbold_italic_w end_ARG | end_ARG start_ARG n end_ARG end_ARG end_ARG ) + square-root start_ARG divide start_ARG n end_ARG start_ARG d end_ARG end_ARG . (36) Let μn=a⁢nssubscript _n= ansμitalic_n = divide start_ARG a n end_ARG start_ARG s end_ARG. In our regime, μn≪1much-less-thansubscript1 _n 1μitalic_n ≪ 1 because q+r>11q+r>1q + r > 1. Combining Equations 35 and 36 yields SU⁢(|)⁢(|)≍μn⋅(α⁢‖T‖2⁢‖T‖2±‖R∩T‖1n±sn)±ndμn⁢(1−ρ2⁢‖T‖2+|R|n)+ndasymptotically-equalsSUconditionalconditionalplus-or-minus⋅subscriptplus-or-minussubscriptnormsubscriptsubscript2subscriptnormsubscriptsubscript2subscriptnormsubscriptsubscriptsubscript1subscript1superscript2subscriptnormsubscriptsubscript2subscript SU( v| w) CN( v| w)% _n· (α v_T_ w_2 % w_T_ w_2± v_R_ w∩ T_ v_1% n± sn)± nd _n ( 1-ρ% ^2 w_T_ w_2+ R_ wn% )+ nddivide start_ARG SU ( v | w ) end_ARG start_ARG sansserif_CN ( italic_v | italic_w ) end_ARG ≍ divide start_ARG μitalic_n ⋅ ( start_ARG α ∥ start_ARG italic_vitalic_T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥2 ∥ start_ARG italic_witalic_T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥2 ± divide start_ARG ∥ start_ARG italic_vitalic_R start_POSTSUBSCRIPT italic_w ∩ Tbold_italic_v end_POSTSUBSCRIPT end_ARG ∥1 end_ARG start_ARG square-root start_ARG n end_ARG end_ARG ± divide start_ARG s end_ARG start_ARG n end_ARG end_ARG ) ± square-root start_ARG divide start_ARG n end_ARG start_ARG d end_ARG end_ARG end_ARG start_ARG μitalic_n ( start_ARG square-root start_ARG 1 - ρ2 end_ARG ∥ start_ARG italic_witalic_T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥2 + square-root start_ARG divide start_ARG | start_ARG Rbold_italic_w end_ARG | end_ARG start_ARG n end_ARG end_ARG end_ARG ) + square-root start_ARG divide start_ARG n end_ARG start_ARG d end_ARG end_ARG end_ARG (37) Hence, for the survival to contamination ratio to grow with n, we need α⁢‖T‖2≫1−ρ2much-greater-thansubscriptnormsubscriptsubscript21superscript2 α v_T_ w_2 1-ρ^2α ∥ start_ARG italic_vitalic_T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥2 ≫ square-root start_ARG 1 - ρ2 end_ARG (Weak supervision) α⁢‖T‖2⁢‖T‖2≫|R|nmuch-greater-thansubscriptnormsubscriptsubscript2subscriptnormsubscriptsubscript2subscript α v_T_ w_2 w_T_ w_2% R_ wnα ∥ start_ARG italic_vitalic_T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥2 ∥ start_ARG italic_witalic_T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥2 ≫ square-root start_ARG divide start_ARG | start_ARG Rbold_italic_w end_ARG | end_ARG start_ARG n end_ARG end_ARG (Favored contamination) μn⁢α⁢‖T‖2⁢‖T‖2≫ndmuch-greater-thansubscriptsubscriptnormsubscriptsubscript2subscriptnormsubscriptsubscript2 _nα v_T_ w_2 w_T_ w% _2 ndμitalic_n α ∥ start_ARG italic_vitalic_T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥2 ∥ start_ARG italic_witalic_T start_POSTSUBSCRIPT italic_w end_POSTSUBSCRIPT end_ARG ∥2 ≫ square-root start_ARG divide start_ARG n end_ARG start_ARG d end_ARG end_ARG Let us put these scalings together to predict the scaling regimes for weak-to-strong generalization. For strong generalization, we have =∗subscript v= v_*italic_v = italic_v∗ and =sgn⁢(⟨,∗⟩)sgnexpectation-valuesubscript y=sgn( g, v_*)italic_y = sgn ( ⟨ start_ARG italic_g , italic_v∗ end_ARG ⟩ ). From the discussion in Section 3.1, we know that the strong learner generalizes if SU⁢(∗|∗)⁢(∗|∗)=ωn⁢(1),SUconditionalsubscriptsubscriptconditionalsubscriptsubscriptsubscript1 SU( v_*| v_*) CN( v_*| v_*)% = _n(1),divide start_ARG SU ( v∗ | v∗ ) end_ARG start_ARG sansserif_CN ( italic_v∗ | italic_v∗ ) end_ARG = ωitalic_n ( 1 ) , and fails to generalize if the ratio is on⁢(1)subscript1o_n(1)oitalic_n ( 1 ). Under Assumption 1, we have ∗=e1subscriptsubscript1 v_*=e_1italic_v∗ = e1, so T=T=1subscriptsubscript1T_ v=T_ w= 1Tbold_italic_v = Tbold_italic_w = start_ARG 1 end_ARG , α=11α=1α = 1, and ρ=11ρ=1ρ = 1, and the expression simplifies to SU⁢(∗|∗)⁢(∗|∗)≍μnμn⋅sn+nd,asymptotically-equalsSUconditionalsubscriptsubscriptconditionalsubscriptsubscriptsubscript⋅subscript SU( v_*| v_*) CN( v_*| v_*)% _n _n· sn+ nd,divide start_ARG SU ( v∗ | v∗ ) end_ARG start_ARG sansserif_CN ( italic_v∗ | italic_v∗ ) end_ARG ≍ divide start_ARG μitalic_n end_ARG start_ARG μitalic_n ⋅ square-root start_ARG divide start_ARG s end_ARG start_ARG n end_ARG end_ARG + square-root start_ARG divide start_ARG n end_ARG start_ARG d end_ARG end_ARG end_ARG , which under the bi-level parameter scaling verifies the conditions for ground truth supervision. This completes the proof sketch; it remains to justify the above estimates rigorously. In the subsequent subsections, we will assume that the above scalings of the survival and contamination are correct and use them to deduce that the 1-sparse assumption is necessary to get a sharp transition in the test error. These calculations can be upgraded to rigorous proofs using the tools are developed in Appendix B. G.1 The necessity of 1-sparse assumption Let’s suppose we get clean labels from sgn⁢(⟨,⟩)sgnexpectation-valuesgn( g, v)sgn ( ⟨ start_ARG italic_g , italic_v end_ARG ⟩ ) and want to learn the unit vector vitalic_v. We will abbreviate T=T=TsubscriptsubscriptT=T_ v=T_ wT = Tbold_italic_v = Tbold_italic_w. In this case, we have α=11α=1α = 1 and ρ2=‖T‖44‖T‖22superscript2superscriptsubscriptnormsubscript44superscriptsubscriptnormsubscript22ρ^2= v_T_4^4 v_T_2^2ρ2 = divide start_ARG ∥ start_ARG italic_vitalic_T end_ARG ∥44 end_ARG start_ARG ∥ start_ARG italic_vitalic_T end_ARG ∥22 end_ARG in Assumption 5. Lemma G.2. Suppose we are given labels according to vitalic_v and want to learn vitalic_v. Then, the survival to contamination ratio is ωn⁢(1)subscript1 _n(1)ωitalic_n ( 1 ) only if ‖T‖44‖T‖22=1−o⁢(1).superscriptsubscriptnormsubscript44superscriptsubscriptnormsubscript2211 v_T_4^4 v_T_2^2=1-o(1).divide start_ARG ∥ start_ARG italic_vitalic_T end_ARG ∥44 end_ARG start_ARG ∥ start_ARG italic_vitalic_T end_ARG ∥22 end_ARG = 1 - o ( 1 ) . In particular, the above condition holds only if the following two conditions hold: (1) ‖T‖22=1−o⁢(1)superscriptsubscriptnormsubscript2211 v_T_2^2=1-o(1)∥ start_ARG italic_vitalic_T end_ARG ∥22 = 1 - o ( 1 ). (2) For each i∈Ti∈ Ti ∈ T, either vi=o⁢(1)subscript1v_i=o(1)vitalic_i = o ( 1 ) or vi=1−o⁢(1)subscript11v_i=1-o(1)vitalic_i = 1 - o ( 1 ). The upshot is that having 1111-sparse labels is necessary for obtaining asymptotically perfect generalization. Proof. Hence, focusing only on the survival terms coming from T, which are the only relevant coordinates for learning, and lower bounding the contamination with just the T terms, we have SU⁢(|)⁢(|)⩽a⁢ns⁢‖T‖22a⁢ns⁢‖T‖22−‖T‖44=‖T‖21−‖T‖44‖T‖22⩽11−‖T‖44‖T‖22,SUconditionalconditionalsuperscriptsubscriptnormsubscript22superscriptsubscriptnormsubscript22superscriptsubscriptnormsubscript44subscriptnormsubscript21superscriptsubscriptnormsubscript44superscriptsubscriptnormsubscript2211superscriptsubscriptnormsubscript44superscriptsubscriptnormsubscript22 SU( v| v) CN( v| v) % ans v_T_2^2 ans v_T_2% ^2- v_T_4^4= v_T_2 1- % v_T_4^4 v_T_2^2 1 % 1- v_T_4^4 v_T_2^2,divide start_ARG SU ( v | v ) end_ARG start_ARG sansserif_CN ( italic_v | italic_v ) end_ARG ⩽ divide start_ARG divide start_ARG a n end_ARG start_ARG s end_ARG ∥ start_ARG italic_vitalic_T end_ARG ∥22 end_ARG start_ARG divide start_ARG a n end_ARG start_ARG s end_ARG square-root start_ARG ∥ start_ARG italic_vitalic_T end_ARG ∥22 - ∥ start_ARG italic_vitalic_T end_ARG ∥44 end_ARG end_ARG = divide start_ARG ∥ start_ARG italic_vitalic_T end_ARG ∥2 end_ARG start_ARG square-root start_ARG 1 - divide start_ARG ∥ start_ARG italic_vitalic_T end_ARG ∥44 end_ARG start_ARG ∥ start_ARG italic_vitalic_T end_ARG ∥22 end_ARG end_ARG end_ARG ⩽ divide start_ARG 1 end_ARG start_ARG square-root start_ARG 1 - divide start_ARG ∥ start_ARG italic_vitalic_T end_ARG ∥44 end_ARG start_ARG ∥ start_ARG italic_vitalic_T end_ARG ∥22 end_ARG end_ARG end_ARG , where the last inequality used the fact that vitalic_v is unit norm. This proves the first necessary condition. We show that if ‖T‖44/‖T‖22=1−o⁢(1)superscriptsubscriptnormsubscript44superscriptsubscriptnormsubscript2211 v_T_4^4/ v_T_2^2=1-o(1)∥ start_ARG italic_vitalic_T end_ARG ∥44 / ∥ start_ARG italic_vitalic_T end_ARG ∥22 = 1 - o ( 1 ), then the second set of necessary conditions hold. Indeed, for the first claim, suppose ‖T‖22⩽1−εsuperscriptsubscriptnormsubscript221 v_T_2^2 1- ∥ start_ARG italic_vitalic_T end_ARG ∥22 ⩽ 1 - ε for some constant ε>00 >0ε > 0. The LpsuperscriptL^pLitalic_p norm inequalities imply that ‖T‖44⩽‖T‖24⩽(1−ε)⁢‖T‖22superscriptsubscriptnormsubscript44superscriptsubscriptnormsubscript241superscriptsubscriptnormsubscript22 v_T_4^4 v_T_2^4 (1-% ) v_T_2^2∥ start_ARG italic_vitalic_T end_ARG ∥44 ⩽ ∥ start_ARG italic_vitalic_T end_ARG ∥24 ⩽ ( 1 - ε ) ∥ start_ARG italic_vitalic_T end_ARG ∥22, so the ratio is at most 1−ε11- 1 - ε, a contradiction. For the second claim, suppose instead there is a coordinate i with vi2=1−εsuperscriptsubscript21v_i^2=1- _i2 = 1 - ε for some constant ε∈(0,1)01 ∈(0,1)ε ∈ ( 0 , 1 ). Then, we have ‖T∖i‖44⩽‖T∖i‖24⩽εsuperscriptsubscriptnormsubscript44superscriptsubscriptnormsubscript24 v_T i_4^4 v_T% i_2^4 ∥ start_ARG italic_vitalic_T ∖ start_ARG i end_ARG end_ARG ∥44 ⩽ ∥ start_ARG italic_vitalic_T ∖ start_ARG i end_ARG end_ARG ∥24 ⩽ ε, but then ‖T‖44⩽(1−ε)2+ε⩽1−Ω⁢(ε)superscriptsubscriptnormsubscript44superscript121Ω v_T_4^4 (1- )^2+ 1-% ( )∥ start_ARG italic_vitalic_T end_ARG ∥44 ⩽ ( 1 - ε )2 + ε ⩽ 1 - Ω ( ε ), a contradiction. ∎

Instant research discovery