Paper deep dive
Scaling Influence Functions in LLMs through Eigenbasis-Corrected One-Bit Gradient Projection
Jaeseung Heo, J Rosser, Dongwoo Kim
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Influence functions estimate how individual training examples affect the behavior of large language models (LLMs). Analyzing how training data influence different behaviors of an LLM involves repeated influence computation. Reusing stored training gradients reduces the computational cost, but storing full gradients is prohibitively expensive at LLM scale. We study how to compress these gradients while preserving influence estimates for future queries that are unknown at storage time. Through a worst-case analysis, we characterize the optimal fixed-dimensional linear representation and propose eigenbasis-corrected one-bit gradient projection (EOGP) to approximate it at scale. Specifically, EOGP uses EK-FAC to reduce gradient dimensionality, then applies PCA within the retained subspace to learn compression directions from the training gradients. We then apply one-bit quantization to the resulting coordinates, allowing more coordinates to be retained within a fixed storage budget. On GPT-2, EOGP predicts retraining outcomes more accurately than the evaluated compression baselines while using one-sixteenth of their per-example storage. On OLMo 2 SFT models from 1B to 32B parameters, EOGP remains competitive with the baselines allocated over 100 times as much storage per example.
Tags
Links
- Source: https://arxiv.org/abs/2609.37842v1
- Canonical: https://arxiv.org/abs/2609.37842v1
Trouble viewing inline? Open PDF directly →
Full Text
77,121 characters extracted from source content.
Expand or collapse full text
Scaling Influence Functions in LLMs through Eigenbasis-Corrected One-Bit Gradient Projection Jaeseung Heo Affiliation: POSTECH Email: jsheo12304@postech.ac.kr J Rosser Affiliation: University of Oxford Email: jrosser@robots.ox.ac.uk Dongwoo Kim Affiliation: POSTECH Email: dongwookim@postech.ac.kr Abstract Influence functions estimate how individual training examples affect the behavior of large language models (LLMs). Analyzing how training data influence different behaviors of an LLM involves repeated influence computation. Reusing stored training gradients reduces the computational cost, but storing full gradients is prohibitively expensive at LLM scale. We study how to compress these gradients while preserving influence estimates for future queries that are unknown at storage time. Through a worst-case analysis, we characterize the optimal fixed-dimensional linear representation and propose eigenbasis-corrected one-bit gradient projection (EOGP) to approximate it at scale. Specifically, EOGP uses EK-FAC to reduce gradient dimensionality, then applies PCA within the retained subspace to learn compression directions from the training gradients. We then apply one-bit quantization to the resulting coordinates, allowing more coordinates to be retained within a fixed storage budget. On GPT-2, EOGP predicts retraining outcomes more accurately than the evaluated compression baselines while using one-sixteenth of their per-example storage. On OLMo 2 SFT models from 1B to 32B parameters, EOGP remains competitive with the baselines allocated over 100 times as much storage per example. 1 Introduction Influence functions are widely used to analyze how training data shape the diverse behaviors of large language models (LLMs). Their applications include studying generalization (Grosse et al., 2023; Ruis et al., 2025; Kou et al., 2025), selecting data for fine-tuning (Zhou et al., 2024; Heo et al., 2026; Chen et al., 2026), and identifying training examples that contribute to undesired behavior (Zhang et al., 2025; Coalson et al., 2025). By estimating how individual training examples affect a model’s predictions (Koh and Liang, 2017), influence functions provide a common framework for investigating these connections between training data and model behavior. Each attribution query asks which training examples contribute to a particular model output or behavior. Analyzing different behaviors of a fixed model therefore involves multiple queries, but the training-example gradients used in influence estimation can be reused across them. Computing and storing these gradients once would avoid repeated backward passes over the training data as new queries arise. At LLM scale, however, storing full gradients is costly: a single training-example gradient for an 8B-parameter model requires 16 GB in half precision. Supporting repeated attribution queries efficiently thus calls for compact gradient representations that can be stored and reused. Existing gradient-compression methods reduce this storage cost through random (Schioppa, 2024; Choe et al., 2025; Hu et al., 2025) or curvature-informed projections (Schioppa et al., 2022; Choe et al., 2025), representing each gradient with a small number of coordinates. The central challenge is deciding which information these coordinates should preserve. Since future attribution queries are not known when the representations are constructed, compression must retain information that remains useful across different subsequent influence calculations. We address this challenge by defining a compression objective based on the discrepancy between influence scores computed from compressed and uncompressed gradients across possible future queries. Under a normalized worst-case influence-error criterion, we show that the top-k PCA coordinates of half-whitened training gradients are optimal among k-dimensional linear representations. This result provides a concrete target for constructing reusable gradient representations. To approximate this target at LLM scale, we propose Eigenbasis-corrected One-bit Gradient Projection (EOGP). Computing PCA directly on full gradients is prohibitively expensive for billion-parameter models. We therefore first reduce their dimensionality using projection directions derived from EK-FAC (George et al., 2018), then apply PCA to the resulting representations to learn the final compression basis. This allows us to refine the representation using the training gradients within a computationally manageable space. We then apply one-bit quantization to the resulting coordinates, allowing more coordinates to be stored within a fixed budget. Together, these steps address which directions to retain and how much precision to allocate to each coordinate. We evaluate the resulting representations through retraining-based validation and comparison with an uncompressed influence reference. On GPT-2 (Radford et al., 2019), EOGP outperforms the tested compression baselines in both linear datamodeling score (LDS) (Park et al., 2023) and counterfactual retraining (Bae et al., 2024). On OLMo 2 SFT models (OLMo et al., 2024) from 1B to 32B parameters, EOGP more accurately reproduces the reference influence rankings at matched storage budgets. Even with less than 16 KB per example, it remains competitive with all tested compression baselines allocated over 100 times as much storage. These results show that compact, reusable gradient representations can retain useful attribution information at substantially lower storage cost. 2 Preliminaries 2.1 Influence Functions Influence functions (Koh and Liang, 2017) estimate how a model’s behavior would change if a training example were removed and the model retrained. Let θ∈ℝdθ ^d denote the model parameters, θ∗∈ℝdθ^* ^d their trained values, and ℓi:ℝd→ℝ _i:R^d the loss function for training example i. The corresponding training gradient is ∇θℓi(θ∗)∈ℝd _θ _i(θ^*) ^d, and H∈ℝd×dH ^d× d denotes the Hessian of the average training loss at θ∗θ^*. For a scalar target measurement f:ℝd→ℝf:R^d , we define the query gradient ∇θf∈ℝd _θf ^d as its gradient with respect to θ, evaluated at θ∗θ^*. For example, a query about the loss on a target document z uses ∇θf=∇θℓz(θ∗) _θf= _θ _z(θ^*). The influence score of training example i for this query is ℐ(i)≔∇θf⊤H−1∇θℓi(θ∗)∈ℝ.I(i) _θf H^-1 _θ _i(θ^*) . (1) For neural networks, the loss Hessian H can be singular, so H−1H^-1 may not exist. Prior work addresses this by using the generalized Gauss–Newton (GGN) matrix G∈ℝd×dG ^d× d, with G⪰0G 0, together with damping (Martens, 2020; Bae et al., 2022; Grosse et al., 2023). For λ>0λ>0, we define Hλ≔G+λId∈ℝd×dH_λ G+λ I_d ^d× d, where IdI_d is the d×d× d identity matrix. Since Hλ≻0H_λ 0, the practical influence score is ℐ^(i)≔∇θf⊤Hλ−1∇θℓi(θ∗)∈ℝ. I(i) _θf H_λ^-1 _θ _i(θ^*) . (2) For a fixed model, the training gradient ∇θℓi(θ∗) _θ _i(θ^*) does not depend on the query and can therefore be reused across queries. In this work, we aim to store these gradients so that subsequent influence calculations do not require repeated backward passes over the training data. 2.2 Kronecker-Factored Curvature Approximation GGN–Fisher equivalence. Storing and factorizing the full GGN is impractical at LLM scale, so prior work has used structured approximations of the Fisher information matrix for scalable influence estimation (Grosse et al., 2023). For softmax cross-entropy on logits, the loss commonly used to train autoregressive LLMs, the GGN coincides with the Fisher F (Martens, 2020). Writing g(x,y)≔∇θℓ(x,y,θ∗)g(x,y) _θ (x,y;θ^*), this matrix is G=F=x∼Xy~∼pθ∗(⋅∣x)[g(x,y~)g(x,y~)⊤],G=F=E_x _XE_ y p_θ^*(· x)\! [g(x, y)g(x, y) ], (3) where XD_X is the empirical distribution of training inputs. Thus, the curvature is a gradient second moment with labels drawn from the model’s predictive distribution. K-FAC approximation. K-FAC (Martens and Grosse, 2015) exploits the layer structure of neural networks to approximate this Fisher matrix efficiently. In its block-diagonal form, it neglects cross-layer blocks and approximates each layer’s block by a Kronecker product. For a linear layer, the two Kronecker factors are the second-moment matrices of its input activations a and the loss gradients δ with respect to its outputs. Glayer=[(aa⊤)⊗(δδ⊤)]≈A⊗S,A≔[aa⊤],S≔[δδ⊤].G_layer=E\! [(a ) (δ ) ]≈ A S, A [a ], S [δ ]. (4) Let QAQ_A and QSQ_S be the orthonormal eigenbases of A and S, respectively. Then QA⊗QSQ_A Q_S is an eigenbasis of the K-FAC approximation A⊗SA S. This structure allows the basis to be obtained from two smaller eigendecompositions and applied through matrix multiplications without materializing the full basis matrix. EK-FAC approximation. EK-FAC (George et al., 2018; Grosse et al., 2023) corrects the eigenvalues of the K-FAC approximation while retaining its block structure and layerwise eigenbases. It obtains the corrected values by expressing gradients in these bases and taking the mean square of each coordinate. With labels sampled as in Equation 3, the resulting approximation is G≈QΛQ⊤,G≈ Q Q , (5) where Q is block diagonal, with each block given by the corresponding layer’s K-FAC eigenbasis QA⊗QSQ_A Q_S, and Λ is the diagonal matrix of corrected eigenvalues. 3 Method This section presents Eigenbasis-corrected One-bit Gradient Projection (EOGP), which stores compact representations of training gradients for reuse in influence estimation. We first formulate gradient compression as minimizing a normalized worst-case influence error and show that storing the top-k PCA coordinates of half-whitened training gradients is optimal among k-dimensional linear representations (§3.1). We then approximate this target using a projection onto the EK-FAC eigenbasis followed by PCA within the retained subspace (§3.2). Finally, we store the resulting training coordinates as one sign bit each, together with a scale per module (§3.3). Figure 1 illustrates the pipeline, and Appendix B gives the full procedure. Figure 1: Overview of EOGP. (a) Training gradients are projected onto an EK-FAC subspace and half-whitened. (b) PCA corrects the compression basis within this subspace. (c) The resulting coordinates are stored as one sign bit per coordinate together with a shared scale. Query gradients undergo the same transformations without one-bit quantization, and influence scores are computed as scaled inner products with the stored bits. 3.1 What to Store: PCA of Half-Whitened Gradients Compression objective. We first ask which k-dimensional linear representation best preserves the influence scores in Equation 2 when queries are unknown at storage time. We consider linear compression methods that store V⊤∇θℓi(θ∗)∈ℝkV _θ _i(θ^*) ^k per training example, where V∈ℝd×kV ^d× k defines the compression. A reconstruction matrix M∈ℝd×kM ^d× k maps these coordinates back to parameter space, giving the influence estimate ∇θf⊤MV⊤∇θℓi(θ∗) _θf MV _θ _i(θ^*). We evaluate compression by the discrepancy between the uncompressed influence score and the estimate obtained from the stored coordinates. Since this discrepancy scales linearly with the query gradient, we constrain query magnitude to obtain a finite worst-case error. The influence score in Equation 2 weights the interaction between the query and training gradients by Hλ−1H_λ^-1, giving greater weight to components along directions of lower curvature. The standard Euclidean norm treats all directions equally and therefore does not capture this directional weighting. To account for how query components enter the influence computation, we use the Hλ−1H_λ^-1-weighted Euclidean norm and define its unit ball ℬB as ‖q‖Hλ−1≔q⊤Hλ−1q,ℬ≔q∈ℝd:‖q‖Hλ−1≤1. \|q\|_H_λ^-1 q H_λ^-1q, \q ^d:\|q\|_H_λ^-1≤ 1 \. (6) Our objective is the worst-case squared influence error over query gradients in ℬB, averaged over training examples: ℰ(V,M)≔i[sup∇θf∈ℬ|∇θf⊤(Hλ−1−MV⊤)∇θℓi(θ∗)|2].E(V,M) _i [ _ _θf | _θf (H_λ^-1-MV ) _θ _i(θ^*) |^2 ]. (7) This criterion allows us to optimize the stored representation without knowing which attribution queries will be evaluated later. Optimal linear representation. To characterize the optimal representation, we define the half-whitened training and query gradients t~i≔Hλ−1/2∇θℓi(θ∗) t_i H_λ^-1/2 _θ _i(θ^*) and q~≔Hλ−1/2∇θf q H_λ^-1/2 _θf. Influence is then the inner product ⟨q~,t~i⟩ q, t_i . Proposition 1 (Optimality of half-whitened PCA). Let 1≤k≤d1≤ k≤ d, and let the columns of U∈ℝd×kU ^d× k be orthonormal eigenvectors corresponding to the k largest eigenvalues of the uncentered second moment Σ≔i[t~it~i⊤] _i[ t_i t_i ]. Then Equation 7 is minimized by V=M=Hλ−1/2UV=M=H_λ^-1/2U, i.e., by storing U⊤t~iU t_i and estimating influence as its inner product with U⊤q~U q. Thus, storing the top-k PCA coordinates of half-whitened training gradients minimizes influence-estimation error under Equation 7. The proof is provided in Section A.1. 3.2 Two-Stage Projection: PCA within an EK-FAC Subspace Performing PCA directly on half-whitened training gradients is impractical at LLM scale, so we approximate it in two stages. The first stage uses the gradient statistics provided by EK-FAC to construct an m-dimensional candidate subspace. The second stage applies PCA to the training gradients projected into this subspace to select the final k directions, where m>km>k. We apply both stages independently to each linear layer (module). Throughout this subsection, we suppress module indices and use d, m, and k for the corresponding per-module dimensions. First stage: subspace selection with EK-FAC. Let C≔i[∇θℓi(θ∗)∇θℓi(θ∗)⊤]∈ℝd×dC _i[ _θ _i(θ^*) _θ _i(θ^*) ] ^d× d denote the uncentered second moment of the training gradients, also known as the empirical Fisher (Martens, 2020). Applying the PCA criterion of Proposition 1 within each module requires the leading eigenvectors of Σ=Hλ−1/2CHλ−1/2. =H_λ^-1/2CH_λ^-1/2. (8) To construct a tractable surrogate for Σ , we approximate both the half-whitening operator and the training-gradient second moment. We begin with the half-whitening operator, which depends on the curvature matrix G. For softmax cross-entropy, G coincides with the true Fisher and can therefore be approximated using EK-FAC, as reviewed in Section 2.2. Writing G≈QΛQ⊤G≈ Q Q , where Λ=diag(λ1,…,λd) =diag( _1,…, _d) contains the corrected eigenvalues, gives Hλ−1/2=(G+λI)−1/2≈Q(Λ+λI)−1/2Q⊤≕H^λ−1/2.H_λ^-1/2=(G+λ I)^-1/2≈ Q( +λ I)^-1/2Q H_λ^-1/2. (9) For candidate subspace selection, we additionally use QΛQ⊤Q Q as a surrogate for C. This is a separate approximation: although the Fisher and the empirical Fisher C are both gradient second moments, the Fisher averages over the model’s predictive label distribution for each input, whereas C uses the corresponding training label. Combining this surrogate with the approximate half-whitening in Equation 9 yields Σproxy _proxy ≔H^λ−1/2QΛQ⊤H^λ−1/2 H_λ^-1/2Q Q H_λ^-1/2 (10) =Qdiag(λ1λ1+λ,…,λdλd+λ)Q⊤. =Q\,diag\! ( _1 _1+λ,…, _d _d+λ )Q . Since λj/(λj+λ) _j/( _j+λ) increases with λj _j for λ>0λ>0, the m EK-FAC eigenvectors with the largest corrected eigenvalues form a top-m PCA basis for Σproxy _proxy. We can therefore construct this candidate subspace directly from the existing EK-FAC statistics, without separately estimating C or performing a full-space eigendecomposition. Let Qm∈ℝd×mQ_m ^d× m contain these eigenvectors and Λm∈ℝm×m _m ^m× m be the diagonal matrix of their corrected eigenvalues. Projecting the EK-FAC half-whitened training gradients onto this basis gives the first-stage coordinates: t~i(1) t_i^(1) ≔Qm⊤H^λ−1/2∇θℓi(θ∗) Q_m H_λ^-1/2 _θ _i(θ^*) (11) =(Λm+λI)−1/2Qm⊤∇θℓi(θ∗)∈ℝm. =( _m+λ I)^-1/2Q_m _θ _i(θ^*) ^m. The Kronecker factorization Q=QA⊗QSQ=Q_A Q_S allows these coordinates to be computed through matrix multiplications with the two smaller factors, without forming a dense d×md× m projection matrix. Second stage: subspace refinement with PCA. The EK-FAC basis is inherited from the K-FAC approximation A⊗SA S, which uses separate second moments of activations and backpropagated gradients. Although EK-FAC corrects the diagonal second moments in this basis, it keeps the basis fixed and leaves off-diagonal terms unmodeled. Consequently, the retained axes need not align with the principal directions of the half-whitened training gradients. To account for the second-moment structure among the retained coordinates, the first stage keeps m>km>k EK-FAC axes. We then find the optimal k-dimensional subspace within their span by applying PCA to the full second moment of the first-stage coordinates, Σ(1)≔i[t~i(1)(t~i(1))⊤], ^(1) _i\! [ t_i^(1)( t_i^(1)) ], including its off-diagonal entries. Let P∈ℝm×kP ^m× k contain the orthonormal eigenvectors of Σ(1) ^(1) associated with its k largest eigenvalues. Combining this PCA projection with the first-stage transformation gives the final coordinates: t~i(2) t_i^(2) ≔P⊤t~i(1) P t_i^(1) (12) =(QmP)⊤H^λ−1/2∇θℓi(θ∗)∈ℝk. =(Q_mP) H_λ^-1/2 _θ _i(θ^*) ^k. The following proposition shows that this correction yields a reconstruction error less than or equal to that obtained using the leading k EK-FAC axes. Proposition 2 (Eigenbasis correction). Let t~iEK≔H^λ−1/2∇θℓi(θ∗) t^EK_i H_λ^-1/2 _θ _i(θ^*) denote the EK-FAC half-whitened training gradients, U⋆≔QmPU_ Q_mP the corrected basis, and QkQ_k the basis formed by the leading k EK-FAC axes retained in QmQ_m. Projecting these gradients onto span(U⋆)span(U_ ) has the following guarantees: (i) The mean squared reconstruction error is less than or equal to that obtained by projecting onto the leading k EK-FAC axes. Formally, i[‖t~iEK−U⋆U⋆⊤t~iEK‖22]≤i[‖t~iEK−QkQk⊤t~iEK‖22].E_i\! [\| t^EK_i-U_ U_ t^EK_i\|_2^2 ] _i\! [\| t^EK_i-Q_kQ_k t^EK_i\|_2^2 ]. In fact, it is minimal among all k-dimensional subspaces of span(Qm)span(Q_m). (i) If span(Qm)span(Q_m) contains a top-k principal subspace of t~iEK t^EK_i, the error equals the minimum attained by full-space k-dimensional PCA within that module. Thus, retaining m>km>k candidate axes allows the final subspace to be optimized within a larger span while storing only k coordinates per training example. The proof is provided in Section A.2. A scalable alternative to PCA. PCA minimizes reconstruction error within the retained span for a fixed output dimension k. However, storing and applying its dense correction matrix P∈ℝm×kP ^m× k becomes costly as k grows. To exploit larger per-example storage budgets without this overhead, we introduce EOGP-R, which replaces the PCA correction with an implicitly applied subsampled randomized Hadamard transform (SRHT) (Tropp, 2011), ∈ℝk×m S ^k× m: t~i(2,R)=t~i(1). t_i^(2,R)= S t_i^(1). The construction and normalization of S are detailed in Section B.2. When the output dimension k is sufficiently large, SRHT preserves norms within a fixed low-dimensional subspace up to bounded distortion with high probability (Tropp, 2011). For query and training representations in this subspace, this guarantee also controls inner-product distortion and the resulting influence-score error introduced by sketching. This variant offers a trade-off between PCA’s fixed-k reconstruction optimality and scalability to larger output dimensions without a dense correction matrix. 3.3 One-Bit Quantization Even after projection, storing every coordinate in floating point limits the number of directions we can retain under a fixed per-example storage budget. We apply one-bit quantization to both EOGP and EOGP-R to accommodate more projected coordinates within this budget. For each module u, let t~i,u(2)∈ℝku t_i,u^(2) ^k_u denote the projected training-gradient coordinates. We store a binary sign vector and one scale: bi,u≔sign(t~i,u(2))∈±1ku,si,u≔1ku‖t~i,u(2)‖1.b_i,u ( t_i,u^(2) )∈\± 1\^k_u, s_i,u 1k_u \| t_i,u^(2) \|_1. (13) The scale captures the average magnitude of the coordinates and is chosen to minimize ‖t~i,u(2)−sbi,u‖22\| t_i,u^(2)-s\,b_i,u\|_2^2 over s (Rastegari et al., 2016). Each module requires kuk_u sign bits and one half-precision scale per training example. The complete representation contains ∑uku _uk_u coordinates. At query time, we apply the same projection to the query gradient and keep its coordinates q~u(2) q_u^(2) unquantized. We estimate influence by ℐ^(i)≔∑usi,u⟨q~u(2),bi,u⟩. I(i) _us_i,u\, q_u^(2),b_i,u . (14) This computes influence from the stored representations without recomputing training gradients. 4 Experiments We evaluate whether compact gradient representations preserve useful training-data attributions under limited storage budgets. We first compare against retraining outcomes on GPT-2 (§4.1), then measure fidelity to EK-FAC influence on billion-parameter LLMs (§4.2). We examine the contributions of the projection and quantization choices (§4.3), and report the storage and computation required to build the representations (§4.4). 4.1 Validation Against Retraining Figure 2: Validation against retraining-based ground truth on GPT-2 across storage budgets: linear datamodeling score (left) and counterfactual retraining (right). Black lines denote uncompressed influence functions with different Hessian approximations and the random-removal baseline. Setup and metrics. We evaluate GPT-2 (Radford et al., 2019) on WikiText-2 (Merity et al., 2016) at matched per-example storage budgets of 12,24,96,384,1536\12,24,96,384,1536\ KB. An uncompressed half-precision gradient requires 162 MB per example. Compression baselines include LoGra (Choe et al., 2025) with PCA and random initialization, GraSS (Hu et al., 2025), and LoRIF (Li et al., 2026). We use EK-FAC influence, K-FAC influence, and TracIn (Pruthi et al., 2020) as uncompressed references, and random removal as an additional baseline for the counterfactual experiment. We use two retraining-based metrics. The linear datamodeling score (LDS) (Park et al., 2023) evaluates how well influence estimates predict retraining outcomes. It measures the rank correlation between the estimated influences of randomly sampled training subsets and the outcomes measured after retraining the model on each subset. Counterfactual retraining (Bae et al., 2024) removes the examples ranked most influential by each method and measures the resulting increase in validation perplexity after retraining. Higher LDS indicates closer agreement with retraining outcomes, while a larger perplexity increase indicates a greater impact of removing the selected examples. Further metric and protocol details are provided in Section D.1. Results. Figure 2 shows that EOGP improves retraining-based attribution quality over the compression baselines, with the largest gains at small storage budgets. In LDS, EOGP at 96 KB exceeds all compression-based baselines at 1,536 KB, despite using 16 times less per-example storage. At 384 KB, it achieves LDS close to the uncompressed K-FAC reference while using over 400 times less per-example storage than the 162 MB uncompressed representation. The counterfactual experiment likewise shows that the examples selected by EOGP produce larger perplexity increases than those selected by the compression baselines. 4.2 Fidelity to EK-FAC at LLM Scale Setup. We evaluate the SFT checkpoints of OLMo 2 (OLMo et al., 2024) with 1B, 7B, 13B, and 32B parameters. Training examples come from the Tülu 3 SFT mixture (Lambert et al., 2024), and queries come from a held-out split. Uncompressed half-precision gradients occupy approximately 2–62 GB per example across these models. We compare compression methods at budgets ranging from a few kilobytes to megabytes per example, corresponding to compression ratios on the order of 10410^4–10610^6 relative to uncompressed half-precision gradients. Further evaluation details are given in Section D.2. Figure 3: LDS versus EK-FAC fidelity on GPT-2, measured by NDCG@20 (left) and Spearman correlation (right). ρ is the Spearman correlation across configurations from Figure 2. Metrics and reference. Since retraining-based evaluation is computationally prohibitive at this scale, we use agreement with EK-FAC influence rankings as a proxy for attribution quality. We choose EK-FAC as the reference because it performs best in our GPT-2 retraining-based evaluation among the evaluated methods that scale to billion-parameter LLMs (Section 4.1), consistent with prior comparisons (Choe et al., 2025). For each query, we measure overall ranking agreement using Spearman correlation and agreement near the top of the ranking using NDCG@20 (Järvelin and Kekäläinen, 2002), then average both metrics over queries. To examine whether these agreement metrics reflect retraining-based attribution quality, Figure 3 shows their relationship with LDS on GPT-2. Each point corresponds to an evaluated configuration from Figure 2. The x-axis shows agreement between the influence rankings produced by each configuration and the EK-FAC reference rankings, measured by NDCG@20 (left) or Spearman correlation (right). The y-axis shows LDS, which measures how well each configuration’s influence estimates predict retraining outcomes. We report the Spearman correlations (ρ) between the x-axis metric and LDS across configurations: 0.95 for NDCG@20 and 0.84 for overall ranking agreement. These positive correlations indicate that configurations whose rankings more closely match EK-FAC generally also better predict the retraining outcomes, providing empirical support for using EK-FAC ranking agreement as a proxy for attribution quality. Figure 4: NDCG@20 (top) and Spearman correlation (bottom) of gradient compression methods against EK-FAC influence functions on OLMo 2 SFT models from 1B to 32B parameters. Results. Figure 4 shows substantial gains over the compression baselines across the evaluated model sizes, particularly at small storage budgets. Fidelity generally improves as more storage is allocated, and continues to increase at the largest tested budgets on the 32B model. The improvements appear in both the overall ranking and the top-ranked examples. These results are obtained at compression ratios of approximately 10410^4–10610^6, supporting the use of compact representations even when storing uncompressed gradients would require tens of gigabytes per example. In Appendix F, we additionally evaluate LoGra and GraSS with the same EK-FAC half-whitening and show that EOGP consistently outperforms both variants. 4.3 Ablation Studies Figure 5: Component ablation of EOGP (left) and comparison of 16-bit and 1-bit storage at matched coordinate counts (right). We examine the effects of PCA basis correction (§3.2) and one-bit quantization (§3.3) on OLMo 2 7B SFT. We compare methods at matched per-example storage budgets (Figure 5, left) and matched numbers of stored coordinates (Figure 5, right). These comparisons assess performance under a fixed storage constraint and representation quality at a fixed dimension, respectively. Comparison at matched storage budgets. At matched storage budgets, EOGP consistently achieves higher NDCG@20 than direct selection of EK-FAC basis directions (Figure 5, left). Both variants use one-bit quantization, demonstrating the benefit of refining the compression basis through PCA of the first-stage coordinates. For EOGP, one-bit representations also outperform FP16 representations by allowing more coordinates to be retained within the same storage budget. We provide further evidence of EOGP’s robustness to one-bit quantization and discuss a possible explanation in Appendix G. Comparison at matched coordinate counts. At matched coordinate counts, EOGP achieves higher NDCG@20 than every baseline stored at the same precision, under both FP16 and one-bit storage (Figure 5, right). This advantage at fixed dimension supports the quality of the two-stage projection. Within EOGP, one-bit and FP16 representations achieve similar NDCG@20, indicating that one-bit quantization substantially reduces storage with little loss in ranking agreement. 4.4 Storage and Computational Costs Per-example storage. With kuk_u coordinates in module u, storing the compressed representation of one training example across all modules requires ∑u(⌈ku/8⌉+2) _u( k_u/8 +2) bytes, including packed sign bits and a two-byte scale per module. At ku=2,048k_u=2,048, the 32B model requires approximately 115.6 KB per training example, compared with 62.4 GB for an uncompressed half-precision gradient. Shared storage overhead. The Kronecker eigenbases, half-whitening weights, and PCA correction matrices are shared across all training examples and queries and are accounted for separately from the per-example payloads. For the 32B model, each module’s PCA correction matrix requires approximately 1 GB when its first-stage and final output dimensions are mu=262,144m_u=262,144 and ku=2,048k_u=2,048, respectively. These shared storage costs do not grow with the number of stored examples. Section E.1 provides formulas for per-example and shared storage. Store-construction time. On a single NVIDIA B200, our implementation of EOGP achieves store-construction times comparable to LoGra. For the 32B model at approximately 2,048 output coordinates per module, the per-example time is 0.247 seconds for EOGP and 0.278 seconds for LoGra. These costs include gradient computation, projection, and disk I/O, including correction-matrix loading and transfer for EOGP. Fitting and saving the PCA correction matrices from cached first-stage coordinates takes an additional 18.4 minutes as a one-time preprocessing cost. Implementation optimizations and detailed timing protocols are provided in Section E.3. 5 Related Work Scalable influence estimation. Influence estimation in large neural networks relies on efficient approximations to inverse-curvature products (Koh and Liang, 2017; Guo et al., 2021; Kwon et al., 2024). Arnoldi iteration approximates a dominant Hessian eigenspace to compute influence in a reduced space (Schioppa et al., 2022). K-FAC and EK-FAC use structured curvature approximations (Martens and Grosse, 2015; George et al., 2018), with EK-FAC enabling influence analysis at LLM scale (Grosse et al., 2023). Gradient compression for data attribution. TRAK scales attribution through random projections (Park et al., 2023), while LESS and its quantized extension QLESS select instruction data using projected gradient similarities rather than the inverse-curvature-based influence scores studied here (Xia et al., 2024; Ananta et al., 2025). For influence estimation, TrackStar (Chang et al., 2025) uses curvature-corrected gradients for influence retrieval at LLM scale. LoGra uses Kronecker-factored projections with random or PCA-based initialization (Choe et al., 2025). GraSS combines gradient sparsification with sparse random projection (Hu et al., 2025). LoRIF stores projected gradients as low-rank factors and approximates inverse curvature via truncated SVD (Li et al., 2026). 6 Conclusion We present EOGP for building compact gradient stores for influence estimation across future queries. Under a normalized worst-case influence-error criterion, we show that PCA of half-whitened training gradients is optimal among fixed-dimensional linear representations. EOGP approximates this target through PCA correction within a retained EK-FAC subspace and uses scaled one-bit quantization to retain more coordinates within a fixed storage budget. Our results show that EOGP preserves useful attribution information with substantially less storage than existing compression methods, making reusable gradient stores more practical at LLM scale. References Ananta et al. (2025) M. Ananta, M. F. Adilazuarda, Z. M. K. Zuhri, A. Purwarianti, and A. F. Aji QLESS: a quantized approach for data valuation and selection in large language model fine-tuning. arXiv preprint arXiv:2502.01703. Cited by: §5. Bae et al. (2024) J. Bae, W. Lin, J. Lorraine, and R. Grosse Training data attribution via approximate unrolling. Advances in Neural Information Processing Systems 37, p. 66647–66686. Cited by: §1, §4.1. Bae et al. (2022) J. Bae, N. Ng, A. Lo, M. Ghassemi, and R. B. Grosse If influence functions are the answer, then what is the question?. Advances in Neural Information Processing Systems 35, p. 17953–17967. Cited by: §2.1. Chang et al. (2025) T. Chang, D. Rajagopal, T. Bolukbasi, L. Dixon, and I. Tenney Scalable influence and fact tracing for large language model pretraining. In International Conference on Learning Representations, Vol. 2025, p. 40976–40997. Cited by: §5. Chen et al. (2026) S. Chen, Y. Qi, M. Ai, Y. Sun, R. Qiu, J. Zou, and J. He Influence-preserving proxies for gradient-based data selection in LLM fine-tuning. In International Conference on Learning Representations, Cited by: §1. Choe et al. (2025) S. Choe, H. Ahn, J. Bae, K. Zhao, Y. Chung, A. Pratapa, W. Neiswanger, E. Strubell, T. Mitamura, J. Schneider, et al. What is your data worth to GPT? LLM-scale data valuation with influence functions. Advances in Neural Information Processing Systems 38, p. 145944–145985. Cited by: §1, §4.1, §4.2, §5. Coalson et al. (2025) Z. Coalson, J. Bae, N. Carlini, and S. Hong IF-Guide: influence function-guided detoxification of LLMs. Advances in Neural Information Processing Systems 38, p. 78824–78856. Cited by: §1. Eckart and Young (1936) C. Eckart and G. Young The approximation of one matrix by another of lower rank. Psychometrika 1 (3), p. 211–218. Cited by: §A.1. George et al. (2018) T. George, C. Laurent, X. Bouthillier, N. Ballas, and P. Vincent Fast approximate natural gradient descent in a Kronecker factored eigenbasis. Advances in Neural Information Processing Systems 31. Cited by: §1, §2.2, §5. Grosse et al. (2023) R. Grosse, J. Bae, C. Anil, N. Elhage, A. Tamkin, A. Tajdini, B. Steiner, D. Li, E. Durmus, E. Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296. Cited by: §1, §2.1, §2.2, §2.2, §5. Guo et al. (2021) H. Guo, N. Rajani, P. Hase, M. Bansal, and C. Xiong FastIF: scalable influence functions for efficient model interpretation and debugging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 10333–10350. Cited by: §5. Heo et al. (2026) J. Heo, K. Yun, Y. Choi, S. Hwang, J. Ok, and D. Kim Interaction-aware influence functions for group attribution. arXiv preprint arXiv:2605.15675. Cited by: §1. Hu et al. (2025) P. Hu, J. Melkonian, W. Tang, H. Zhao, and J. Ma GraSS: scalable data attribution with gradient sparsification and sparse projection. Advances in Neural Information Processing Systems 38, p. 43880–43904. Cited by: §1, §4.1, §5. Järvelin and Kekäläinen (2002) K. Järvelin and J. Kekäläinen Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS) 20 (4), p. 422–446. Cited by: §4.2. Koh and Liang (2017) P. W. Koh and P. Liang Understanding black-box predictions via influence functions. In International Conference on Machine Learning, p. 1885–1894. Cited by: §1, §2.1, §5. Kou et al. (2025) S. Kou, Q. Tian, H. Xu, Z. Zeng, and Z. Deng Which data attributes stimulate math and code reasoning? An investigation via influence functions. Advances in Neural Information Processing Systems 38, p. 12302–12324. Cited by: §1. Kwon et al. (2024) Y. Kwon, E. Wu, K. Wu, and J. Y. Zou DataInf: efficiently estimating data influence in LoRA-tuned LLMs and diffusion models. In International Conference on Learning Representations, Vol. 2024, p. 21921–21942. Cited by: §5. Lambert et al. (2024) N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. Tulu 3: pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124. Cited by: §4.2. Li et al. (2026) S. Li, H. Le, J. Xu, and M. Salzmann LoRIF: low-rank influence functions for scalable training data attribution. arXiv preprint arXiv:2601.21929. Cited by: §4.1, §5. Martens and Grosse (2015) J. Martens and R. Grosse Optimizing neural networks with Kronecker-factored approximate curvature. In International Conference on Machine Learning, p. 2408–2417. Cited by: §2.2, §5. Martens (2020) J. Martens New insights and perspectives on the natural gradient method. Journal of Machine Learning Research 21 (146), p. 1–76. Cited by: §2.1, §2.2, §3.2. Merity et al. (2016) S. Merity, C. Xiong, J. Bradbury, and R. Socher Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843. Cited by: §4.1. OLMo et al. (2024) T. OLMo, P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, S. Huang, M. Jordan, et al. 2 OLMo 2 furious. arXiv preprint arXiv:2501.00656. Cited by: §1, §4.2. Park et al. (2023) S. M. Park, K. Georgiev, A. Ilyas, G. Leclerc, and A. Madry TRAK: attributing model behavior at scale. In International Conference on Machine Learning, Cited by: §1, §4.1, §5. Pruthi et al. (2020) G. Pruthi, F. Liu, S. Kale, and M. Sundararajan Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems 33, p. 19920–19930. Cited by: §4.1. Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever Language models are unsupervised multitask learners. Technical report OpenAI. Cited by: §1, §4.1. Rastegari et al. (2016) M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi XNOR-Net: ImageNet classification using binary convolutional neural networks. In European Conference on Computer Vision, p. 525–542. Cited by: §3.3. Ruis et al. (2025) L. Ruis, M. Mozes, J. Bae, S. R. Kamalakara, D. Talupuru, A. Locatelli, R. Kirk, T. Rocktäschel, E. Grefenstette, and M. Bartolo Procedural knowledge in pretraining drives reasoning in large language models. In International Conference on Learning Representations, Vol. 2025, p. 29367–29429. Cited by: §1. Schioppa et al. (2022) A. Schioppa, P. Zablotskaia, D. Vilar, and A. Sokolov Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, p. 8179–8186. Cited by: §1, §5. Schioppa (2024) A. Schioppa Efficient sketches for training data attribution and studying the loss landscape. Advances in Neural Information Processing Systems 37, p. 37692–37735. Cited by: §1. Tropp (2011) J. A. Tropp Improved analysis of the subsampled randomized Hadamard transform. Advances in Adaptive Data Analysis 3 (01n02), p. 115–126. Cited by: §B.2, §3.2, §3.2. Xia et al. (2024) M. Xia, S. Malladi, S. Gururangan, S. Arora, and D. Chen LESS: selecting influential data for targeted instruction tuning. In International Conference on Machine Learning, Cited by: §5. Zhang et al. (2025) H. Zhang, Z. Zhang, Y. Zhang, Y. Zhai, H. Peng, Y. Lei, Y. Yu, H. Wang, B. Liang, L. Gui, et al. Correcting large language model behavior via influence function. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 14477–14485. Cited by: §1. Zhou et al. (2024) X. Zhou, S. Fan, and M. Jaggi HyperINF: unleashing the HyperPower of the Schulz’s method for data influence estimation. arXiv preprint arXiv:2410.05090. Cited by: §1. Appendix A Theoretical Details A.1 Optimality of Half-Whitened PCA We prove Proposition 1 for a fixed symmetric positive-definite matrix HλH_λ and half-whitened gradients t~i=Hλ−1/2∇θℓi(θ∗) t_i=H_λ^-1/2 _θ _i(θ^*) satisfying i‖t~i‖22<∞E_i\| t_i\|_2^2<∞. The supremum is over the full query-gradient ball in Equation 6. Proof. Let Σ=i[t~it~i⊤] =E_i[ t_i t_i ], with eigenvalues μ1≥⋯≥μd≥0 _1≥·s≥ _d≥ 0, and let 1≤k≤d1≤ k≤ d. We show that the minimum of Equation 7 is ∑j=k+1dμj _j=k+1^d _j and that the proposed encoder–decoder pair attains it. Reduction to low-rank reconstruction. For arbitrary V,M∈ℝd×kV,M ^d× k, define R≔Hλ1/2MV⊤Hλ1/2,rank(R)≤k.R H_λ^1/2MV H_λ^1/2, (R)≤ k. (15) Write q~=Hλ−1/2∇θf q=H_λ^-1/2 _θf. Since ∇θℓi(θ∗)=Hλ1/2t~i _θ _i(θ^*)=H_λ^1/2 t_i and ∇θf=Hλ1/2q~ _θf=H_λ^1/2 q, the influence error satisfies ∇θf⊤(Hλ−1−MV⊤)∇θℓi(θ∗)=q~⊤(I−R)t~i. _θf (H_λ^-1-MV ) _θ _i(θ^*)= q (I-R) t_i. (16) The change of variables between ∇θf _θf and q~ q is bijective and maps the normalized query-gradient ball to the Euclidean unit ball. Therefore, ℰ(V,M)=i[sup‖q~‖2≤1|q~⊤(I−R)t~i|2]=i‖(I−R)t~i‖22.E(V,M)=E_i [ _\| q\|_2≤ 1 | q (I-R) t_i |^2 ]=E_i \|(I-R) t_i \|_2^2. (17) The last equality follows from Cauchy–Schwarz, with equality for a unit query in the direction of (I−R)t~i(I-R) t_i whenever this residual is nonzero. If it is zero, both sides are zero. Conversely, every matrix R of rank at most k admits a factorization R=XY⊤R=XY with X,Y∈ℝd×kX,Y ^d× k, padding with zero columns when needed. Choosing M=Hλ−1/2XM=H_λ^-1/2X and V=Hλ−1/2YV=H_λ^-1/2Y realizes this R in Equation 15. Thus optimizing over V,MV,M is equivalent to optimizing Equation 17 over all rank-at-most-k matrices R. Orthogonal projection suffices. Let =range(R)S= (R), and let Π _S be the orthogonal projector onto S. For each example, (I−Π)t~i(I- _S) t_i is orthogonal to S, whereas (Π−R)t~i( _S-R) t_i belongs to S. Consequently, ‖(I−R)t~i‖22 \|(I-R) t_i \|_2^2 =‖(I−Π)t~i‖22+‖(Π−R)t~i‖22 = \|(I- _S) t_i \|_2^2+ \|( _S-R) t_i \|_2^2 (18) ≥‖(I−Π)t~i‖22. ≥ \|(I- _S) t_i \|_2^2. Taking expectations and using Π⊤=Π=Π2 _S = _S= _S^2 gives ℰ(V,M)≥tr((I−Π)Σ)=tr(Σ)−tr(ΠΣ).E(V,M)≥ ((I- _S) )= ( )- ( _S ). (19) The principal subspace attains the minimum. This PCA optimality step is a consequence of the classical Eckart–Young theorem (Eckart and Young, 1936). Write Σ=∑j=1dμjujuj⊤ = _j=1^d _ju_ju_j in an orthonormal eigenbasis. Set αj=uj⊤Πuj=‖Πuj‖22 _j=u_j _Su_j=\| _Su_j\|_2^2. Because Π _S is an orthogonal projector of rank at most k, these weights satisfy 0≤αj≤10≤ _j≤ 1 and ∑j=1dαj=tr(Π)≤k _j=1^d _j= ( _S)≤ k. It follows that tr(ΠΣ)=∑j=1dμjαj≤∑j=1kμjαj+μk(k−∑j=1kαj)≤∑j=1kμj. ( _S )= _j=1^d _j _j≤ _j=1^k _j _j+ _k (k- _j=1^k _j )≤ _j=1^k _j. (20) The first inequality uses μj≤μk _j≤ _k for j>kj>k and the bound on the sum of the weights. The second uses μj≥μk _j≥ _k for j≤kj≤ k and αj≤1 _j≤ 1. Combining Equations 19 and 20, every encoder–decoder pair satisfies ℰ(V,M)≥tr(Σ)−∑j=1kμj=∑j=k+1dμj.E(V,M)≥ ( )- _j=1^k _j= _j=k+1^d _j. (21) Now take U=[u1,…,uk]U=[u_1,…,u_k] and V=M=Hλ−1/2UV=M=H_λ^-1/2U. Then R=UU⊤R=U , so ℰ(Hλ−1/2U,Hλ−1/2U)=tr((I−U⊤)Σ)=∑j=k+1dμj.E (H_λ^-1/2U,H_λ^-1/2U )= ((I-U ) )= _j=k+1^d _j. (22) This pair therefore attains the global minimum. Its stored representation and influence estimate are precisely V⊤∇θℓi(θ∗) V _θ _i(θ^*) =U⊤t~i, =U t_i, (23) ∇θf⊤MV⊤∇θℓi(θ∗) _θf MV _θ _i(θ^*) =q~⊤UU⊤t~i=⟨U⊤q~,U⊤t~i⟩. = q U t_i= U q,U t_i . (24) This proves Proposition 1. ∎ The result uses the uncentered second moment and remains valid when eigenvalues are repeated. A.2 Proof of Eigenbasis Correction We prove Proposition 2 within one module. Let xi=H^λ−1/2∇θℓi(θ∗)x_i= H_λ^-1/2 _θ _i(θ^*) have finite second moment, let Qm∈ℝd×mQ_m ^d× m have orthonormal columns, and set ai=Qm⊤xia_i=Q_m x_i. Let P∈ℝm×kP ^m× k contain orthonormal leading eigenvectors of i[aiai⊤]E_i[a_ia_i ], where 1≤k≤m1≤ k≤ m. Proof. Every k-dimensional subspace of range(Qm) (Q_m) has an orthonormal basis U=QmWU=Q_mW for some W∈ℝm×kW ^m× k satisfying W⊤W=IkW W=I_k. Decomposing xi−UU⊤xix_i-U x_i into components orthogonal to and within range(Qm) (Q_m) yields i‖xi−UU⊤xi‖22 _i\|x_i-U x_i\|_2^2 =i‖(I−QmQm⊤)xi‖22 =E_i\|(I-Q_mQ_m )x_i\|_2^2 (25) +i‖ai−WW⊤ai‖22. +E_i\|a_i-W a_i\|_2^2. The first term is independent of W. By the PCA argument in Section A.1, the second is minimized by W=PW=P. Thus QmPQ_mP minimizes reconstruction error over all k-dimensional subspaces of range(Qm) (Q_m). The leading k retained EK-FAC axes are among these feasible choices, proving part (i). For part (i), suppose range(Qm) (Q_m) contains a globally optimal k-dimensional principal subspace of xix_i within the module. This subspace is feasible for the restricted problem, so the restricted minimum is at most the full-space minimum. It is also at least that minimum because the feasible set is restricted. The two errors therefore agree. ∎ A.3 Scaled One-Bit Quantization Optimal scale for a fixed sign vector. For projected coordinates a∈ℝka ^k with k≥1k≥ 1, let b=sign(a)∈±1kb=sign(a)∈\± 1\^k, allowing either sign for zero coordinates. For a scale s∈ℝs , the identities b⊤a=‖a‖1b a=\|a\|_1 and b⊤b=kb b=k give ‖a−sb‖22=‖a‖22−2s‖a‖1+ks2=k(s−‖a‖1k)2+‖a‖22−‖a‖12k.\|a-sb\|_2^2=\|a\|_2^2-2s\|a\|_1+ks^2=k (s- \|a\|_1k )^2+\|a\|_2^2- \|a\|_1^2k. (26) Thus, s=‖a‖1/ks=\|a\|_1/k minimizes the squared reconstruction error for the sign vector b, including s=0s=0 when a=0a=0. Appendix B Algorithms and Projection Implementation B.1 Projection paths and unified procedure Both stages operate independently on the linear-layer modules defined in Section 3.2. Across all evaluated models, EOGP applies PCA to the first-stage coordinates, while EOGP-R replaces PCA with an SRHT. The model-specific module lists, dimensions, and storage-budget allocations are recorded with the experimental settings. For module u, the first-stage map and the subsequent projection are Lu=(Λu,m+λuI)−1/2Qu,m⊤,Tu(v)=Pu⊤v,for EOGP,uv,for EOGP-R.L_u=( _u,m+ _uI)^-1/2Q_u,m , T_u(v)= casesP_u v,&for EOGP,\\ S_uv,&for EOGP-R. cases (27) Here, Pu∈ℝmu×kuP_u ^m_u× k_u contains the PCA directions, and u∈ℝku×mu S_u ^k_u× m_u is an implicitly applied SRHT. The Kronecker factors and retained-coordinate indices implement LuL_u without materializing Qu,mQ_u,m or LuL_u. We write gi,ug_i,u and quq_u for the training and query gradient blocks of module u. Algorithm 1 EOGP and EOGP-R fitting, storage, and querying 1: Factorized EK-FAC bases QuQ_u, corrected eigenvalues Λu _u, damping values λu>0 _u>0, dimensions 1≤ku≤mu≤du1≤ k_u≤ m_u≤ d_u, method variant, and a PCA fitting set C for EOGP 2: Fitting shared projections 3: for each module u do 4: Select Qu,m,Λu,mQ_u,m, _u,m using the mum_u largest corrected eigenvalues 5: For EOGP-R, initialize the implicit SRHT u S_u 6: end for 7: if using EOGP then 8: for each i∈i do 9: Compute the training gradient gig_i once 10: for each module u do 11: Compute and cache zi,u←Lugi,uz_i,u← L_ug_i,u 12: end for 13: end for 14: For each u, fit PuP_u by uncentered PCA of zi,u:i∈\z_i,u:i \ 15: end if 16: Storing one training example i 17: Compute gig_i once 18: for each module u do 19: xi,u←Tu(Lugi,u)x_i,u← T_u(L_ug_i,u) 20: Store packed binary signs bi,u←sign±(xi,u)b_i,u _±(x_i,u) and si,u←FP16(‖xi,u‖1/ku)s_i,u 16(\|x_i,u\|_1/k_u) 21: end for 22: Scoring a query f 23: Compute q=∇θf(θ∗)q= _θf(θ^*) once 24: For each u, compute the unquantized coordinates yu←Tu(Luqu)y_u← T_u(L_uq_u) 25: return ℐ^(i)=∑usi,u⟨yu,bi,u⟩ I(i)= _us_i,u y_u,b_i,u for each stored example i The batched implementation is described in Section E.3. B.2 PCA fitting and random projection PCA fitting. For EOGP, we form the first-stage coordinates zi,u=Lugi,uz_i,u=L_ug_i,u using training-data labels and apply PCA without subtracting the sample mean. Let Zu∈ℝn×muZ_u ^n× m_u contain these coordinates as rows, where n=||n=|C|. The columns of PuP_u are the top-kuk_u right singular vectors of ZuZ_u, equivalently the eigenvectors associated with the largest eigenvalues of Zu⊤Zu/nZ_u Z_u/n. The nonzero principal directions can be recovered from the eigendecomposition of the n×n× n Gram matrix ZuZu⊤Z_uZ_u , avoiding construction of an mu×mum_u× m_u second-moment matrix. This is the PCA correction analyzed in Proposition 2. Random projection. For EOGP-R, a subsampled randomized Hadamard transform (SRHT) (Tropp, 2011) maps the first-stage coordinates directly to kuk_u dimensions, u=pukuRuHpuDuJu,pu=2⌈log2mu⌉. S_u= p_uk_u\,R_uH_p_uD_uJ_u, p_u=2 _2m_u . (28) Here, JuJ_u zero-pads the mum_u coordinates to width pup_u, DuD_u is a diagonal matrix of independent random signs, HpuH_p_u is the orthonormal Walsh–Hadamard matrix, and RuR_u selects kuk_u rows uniformly without replacement. The same realized transform is used for gradient storage and query projection. This variant does not fit a PCA correction. B.3 Computation and complexity Query scoring. Using the unquantized query coordinates yu=Tu(Luqu)y_u=T_u(L_uq_u) from Algorithm 1 and the stored signs bi,ub_i,u and scales si,us_i,u, the influence score is ℐ^(i)=∑usi,u⟨yu,bi,u⟩. I(i)= _us_i,u y_u,b_i,u . (29) The same query coordinates are reused to score all stored examples. Projection cost. Applying the PCA correction in EOGP costs O(muku)O(m_uk_u) operations per module. The implicit SRHT in EOGP-R costs O(pulogpu)O(p_u p_u) operations per module, where pup_u is the padded dimension in Equation 28. Storage and scanning. Computing signs and scales from an already projected training representation costs O(∑uku)O( _uk_u) operations. After projecting a query, scoring all NstoreN_store examples costs O(Nstore∑uku)O\! (N_store _uk_u ) (30) arithmetic operations. Storage in bytes and measured runtimes are reported in Sections E.1 and E.3. Appendix C Experimental Configuration This section describes the configurations used in Section 4. C.1 Models, data, and gradient computation We evaluate GPT-2 on WikiText-2 and released OLMo 2 SFT checkpoints on Tülu 3 data. The GPT-2 attribution checkpoint is fine-tuned using the protocol in Section D.1. Table 1 lists the source checkpoints and datasets. Table 1: Source model checkpoints and datasets. Model Source checkpoint Dataset GPT-2 gpt2 WikiText-2 OLMo 2 1B allenai/OLMo-2-0425-1B-SFT Tülu 3 mixture OLMo 2 7B allenai/OLMo-2-1124-7B-SFT Tülu 3 mixture OLMo 2 13B allenai/OLMo-2-1124-13B-SFT Tülu 3 mixture OLMo 2 32B allenai/OLMo-2-0325-32B-SFT Tülu 3 mixture Data preparation. We tokenize wikitext-2-raw-v1 from Salesforce/wikitext with the GPT-2 tokenizer and divide concatenated text into 512-token blocks. Training and validation blocks provide attribution candidates and queries, respectively. OLMo 2 1B and 32B use allenai/tulu-3-sft-olmo-2-mixture-0225, while 7B and 13B use allenai/tulu-3-sft-olmo-2-mixture. Each conversation is one example, with queries selected separately from attribution candidates. Evaluation protocols are given in Appendix D. Loss and query measurement. Both training and query gradients use token-summed next-token cross entropy. GPT-2 includes all prediction positions, while OLMo 2 includes only assistant-response positions and their turn-ending tokens. Attributed parameters. We attribute to the attention and MLP linear layers, including biases in GPT-2. Embeddings, the language-model output head, and normalization parameters are excluded. Attribution gradients are computed in evaluation mode. C.2 Curvature fitting and projection dimensions Fitting configuration. We estimate EK-FAC statistics using Kronfluence. Fisher statistics use labels sampled from the model’s next-token distribution with input contexts fixed to the data. Factor statistics are averaged over non-padding tokens, and corrected eigenvalues over examples using sequence-level gradients. For each module u with dud_u attributed parameters, we use relative damping λu=0.1tr(Λu)du, _u=0.1\, tr( _u)d_u, (31) where Λu _u is the diagonal matrix containing the module’s full set of corrected eigenvalues. The resulting half-whitening transformation is applied to both training and query gradients. PCA fitting data. For OLMo 2, the PCA fitting set C contains 10,000 conversations sampled from the same SFT mixture with a fixed random seed, after excluding all candidate, query, and curvature-fitting examples. Each conversation is truncated to at most 2,048 tokens. Projection dimensions. EOGP applies PCA to the first-stage coordinates on all models. For OLMo 2, mu=262,144m_u=262,144, and the final dimension is uniform across modules, ranging in powers of two from 256 to 8,192. EOGP-R uses the same first-stage width and between 256 and 32,768 SRHT output coordinates. Storage budgets. Budgets count per-example coordinates and scales, with projection dimensions chosen to match storage across methods. In the main comparisons, EOGP and EOGP-R use one-bit coordinates with FP16 scales, while compressed baselines use FP16. Figure 5 compares both precisions; in its right panel, which fixes coordinate counts on OLMo 2 7B, the one-bit curves also form a matched-storage comparison, since equal coordinate counts imply equal one-bit storage, and EOGP outperforms all one-bit baselines. GPT-2 follows the nominal budget labels in Figure 2, and OLMo 2 reports payloads in decimal units. Shared artifacts are accounted for separately in Section E.1. C.3 Baseline configurations All methods share attributed modules, evaluation examples, queries, and loss definitions. For OLMo 2, EOGP, LoGra, GraSS, and LoRIF fit their curvature estimates on the same 10,000 conversations. Storage formats and budget matching follow Section C.2. Compression baselines. Our LoGra implementation is based on LogIX11 1 https://github.com/logix-project/logix, with random and PCA initialization on both GPT-2 and OLMo 2. LoGra uses the same relative damping coefficient (0.10.1) as in Section C.2. PCA initialization uses factor eigenvectors fitted with Kronfluence22 2 https://github.com/pomonam/kronfluence, and the factor rank a determines a2a^2 stored coordinates per module. Our GraSS implementation follows the official code, using factor-wise random masking and CountSketch with output dimensions matched to LoGra. Our LoRIF reimplementation uses randomly projected gradient matrices and a rank-256 inverse-curvature approximation. Uncompressed references. EK-FAC and K-FAC use Kronfluence with model-sampled labels and the damping in Section C.2. The TracIn reference is the unweighted gradient inner product at the final checkpoint. Gaussian random projection. This control uses independent standard Gaussian factors to produce a2a^2 coordinates per module. Training gradients are projected directly, and query gradients are preconditioned by the full-space damped EK-FAC inverse before projection. Inner products are scaled by a−2a^-2. Randomization. LoGra and LoRIF use one projection realization per configuration. GraSS uses one on GPT-2 and averages over three seeds on OLMo 2. The Gaussian control averages over three independent draws, and EOGP-R over up to three disjoint blocks of a shared SRHT output. Retraining and uncertainty estimation follow Section D.1. Appendix D Evaluation Protocols D.1 GPT-2 retraining evaluation We evaluate attribution on WikiText-2 using the linear datamodeling score (LDS) and counterfactual retraining. Linear datamodeling score. We reuse the publicly released subset masks and retraining losses from Kronfluence for R=100R=100 subsets, each retaining half of the training blocks. All methods and storage budgets use the same subsets and losses on the validation blocks. The query measurement is the token-summed next-token cross-entropy loss of a validation block. Let SrS_r denote the retained training subset and YrqY_rq its released retraining loss for query q. Under our removal-based influence convention, the predicted subset score is Arq=−∑i∈Srℐ^q(i).A_rq=- _i∈ S_r I_q(i). (32) For each query, we compute the Spearman correlation across subsets, LDSq=ρS((Arq)r=1R,(Yrq)r=1R),LDS_q= _S ((A_rq)_r=1^R,(Y_rq)_r=1^R ), (33) and report its mean over queries, excluding undefined correlations. We use average ranks for ties. We obtain 95% percentile confidence intervals from 1,000 bootstrap resamples of the subsets, recomputing the query-averaged LDS for each resample. Counterfactual retraining. For each method and storage budget, we rank the training blocks by the sum of their signed influence scores over the first 50 validation blocks. We remove the examples with the largest scores, reusing the same ranking for removal counts from 50 to 300 in increments of 50. Each retraining run starts from the pretrained GPT-2 weights and uses AdamW with a learning rate of 3×10−53× 10^-5, weight decay of 0.010.01, three epochs, and a training batch size of eight. We use a set S of five seeds, matched across methods. Perplexity is evaluated on the same 50 validation blocks used to construct the ranking by exponentiating their mean token-level cross-entropy loss. Let PPLs(b)PPL_s(b) denote the perplexity for seed s after removing b examples. We report ΔPPL(b)=15∑s∈[PPLs(b)−PPLs(0)], (b)= 15 _s [PPL_s(b)-PPL_s(0) ], (34) where the zero-removal runs are shared across methods. Error bars show the population standard deviation of the post-removal perplexities across seeds. For the random-removal control, each seed defines a random permutation of the training set, and the first b examples are removed. D.2 Fidelity to EK-FAC influence We compare compressed influence scores with uncompressed EK-FAC scores on the same candidate training examples. OLMo 2 uses 1,000 candidates and 100 queries, while GPT-2 uses the training and validation blocks as candidates and queries, respectively. Module contributions are summed before evaluation, and both score vectors use signed scores under the same removal-based convention, without taking absolute values or clipping. Spearman correlation. For query q, let qe_q and qa_q denote the reference and compressed score vectors over the candidate examples. We compute ρq=Corr(rank(q),rank(q)),ρ¯S=1|valid|∑q∈validρq. _q=Corr (rank(a_q),rank(e_q) ), ρ_S= 1|Q_valid| _q _valid _q. (35) Ties receive average ranks. A correlation is undefined when either score vector is constant, and validQ_valid contains the queries with defined correlations. NDCG@20. We use binary relevance based on membership in the reference top 20. Let TqT_q contain the 20 examples with the largest signed EK-FAC scores, and let πq(t) _q(t) denote the example at position t when compressed scores are sorted in descending order. Then NDCG@20(q)=∑t=120πq(t)∈Tqlog2(t+1)∑t=1201log2(t+1).NDCG@20(q)= _t=1^20 1\ _q(t)∈ T_q\ _2(t+1) _t=1^20 1 _2(t+1). (36) The denominator is the ideal discounted gain when all 20 relevant examples occupy the first 20 positions. We report the arithmetic mean of NDCG@20 over queries. D.3 Relationship between fidelity and retraining Figure 3 compares EK-FAC fidelity with LDS across the 25 compressed configurations and three reference methods in Figure 2. For each configuration, all metrics use the same attribution score matrix. We average each metric over queries before computing correlations across configurations. Let J denote these 28 configurations, and let LjL_j, NjN_j, and SjS_j denote their query-averaged LDS, NDCG@20, and Spearman fidelity. The resulting correlations are ρS((Lj)j∈,(Nj)j∈) _S\! ((L_j)_j ,(N_j)_j ) ≈0.95, ≈ 0.95, (37) ρS((Lj)j∈,(Sj)j∈) _S\! ((L_j)_j ,(S_j)_j ) ≈0.84. ≈ 0.84. Appendix E Storage and Runtime Accounting E.1 Per-example payload and shared artifacts We account separately for per-example representations and artifacts shared across the gradient store. Let U denote the attributed modules, mum_u and kuk_u the first-stage and final output dimensions of module u, respectively, and NstoreN_store the number of stored examples. The attributed parameter set is specified in Section C.1. Per-example payload. EOGP and EOGP-R store one sign bit per coordinate and one FP16 scale per module. With byte packing performed separately for each module, the payload is B1bit=∑u∈(⌈ku8⌉+2)bytes.B_1bit= _u ( k_u8 +2 ) . (38) For a uniform dimension k, this becomes ||(⌈k/8⌉+2)|U|( k/8 +2). Storing the coordinates directly in FP16 requires BFP16=2∑ukuB_FP16=2 _uk_u bytes, while an uncompressed FP16 gradient requires Braw=2dattrB_raw=2d_attr bytes, where dattrd_attr is the number of attributed parameters. Shared artifacts. The Kronecker eigenbases and half-whitening weights are shared across examples and queries. In EOGP, each module’s PCA correction matrix Pu∈ℝmu×kuP_u ^m_u× k_u is also shared and stored in FP16, requiring BP,u=2mukubytes.B_P,u=2m_uk_u . (39) For the 32B model with mu=262,144m_u=262,144 and ku=2,048k_u=2,048, each FP16 correction matrix requires approximately 1.07 GB, totaling 481 GB across all 448 modules. This cost is independent of the number of stored examples, so its relative contribution decreases as the store grows. For stores containing 100 million or one billion examples, the correction matrices would occupy approximately 4.2% or 0.42% of the aggregate per-example payload, respectively. These estimates illustrate how the shared correction matrices can be amortized when constructing attribution stores for large pretraining or mid-training corpora. When shared storage is constrained, reducing kuk_u decreases the correction-matrix storage linearly. For example, keeping mum_u unchanged and reducing kuk_u to 256 lowers the total correction-matrix storage to approximately 60.1 GB for the 32B model. As shown in Figure 4, EOGP maintains an advantage over the evaluated compression baselines at small per-example storage budgets. Total storage. Including shared artifacts and file overhead, the total storage is Btotal=NstoreB1bit+Bshared+Boverhead,B_total=N_storeB_1bit+B_shared+B_overhead, (40) where BsharedB_shared counts shared curvature and projection artifacts, including ∑u∈BP,u _u B_P,u for the PCA correction matrices in EOGP, and BoverheadB_overhead counts file metadata. Per-example budgets count representation payloads, with shared artifacts accounted for separately. E.2 Storage estimates by model Table 2 compares per-example storage for uncompressed FP16 gradients and one-bit representations at ku=2,048k_u=2,048. Both EOGP and EOGP-R use 2048/8+2=2582048/8+2=258 bytes per module per example. Table 2: Per-example representation payloads at ku=2,048k_u=2,048 per module. Uncompressed gradients cover the attributed parameters, and one-bit representations include packed signs and FP16 scales. Shared artifacts are accounted for in Section E.1. Model Uncompressed FP16 (GB/example) One-bit representation (KB/example) OLMo 2 1B 2.147 28.896 OLMo 2 7B 12.952 57.792 OLMo 2 13B 25.376 72.240 OLMo 2 32B 62.411 115.584 E.3 Implementation and runtime Setup. Table 3 compares our implementation with LoGra’s official LogIX implementation on a single NVIDIA B200. EOGP uses mu=262,144m_u=262,144, ku=2,048k_u=2,048, and FP16 correction matrices, while LoGra stores 2,0252,025 FP16 coordinates per module. Curvature statistics are fitted beforehand, and timings exclude curvature fitting and model loading. Table 3: PCA fitting and store-construction times on one NVIDIA B200. Fitting starts from cached coordinates and includes saving the correction matrices. Store-construction totals combine separately timed stages, including disk I/O. Model EOGP PCA fitting (min) Store construction (s/example) EOGP LoGra OLMo 2 1B 4.5 0.0563 0.0577 OLMo 2 7B 8.9 0.0999 0.1315 OLMo 2 13B 11.1 0.1474 0.1661 OLMo 2 32B 18.4 0.2470 0.2780 Implementation. First-stage coordinates are cached in FP16 using asynchronous disk writes. We then process modules sequentially, loading each correction matrix onto the GPU and applying it to a batch of coordinates before one-bit encoding and recording. Each loaded matrix is reused across the batch. Timing protocol. Store construction includes gradient computation, projection, and disk I/O, including correction-matrix loading and transfer for EOGP. Coordinate generation is timed on 40 documents per model averaging approximately 500 tokens. Subsequent PCA projection and storage use batches of approximately 11,000 documents, over which matrix-loading costs are amortized. Figure 6: Comparison of EOGP with LoGra and GraSS on OLMo 2 7B SFT, using either their original curvature treatment or EK-FAC half-whitening before projection. NDCG@20 (left) and Spearman correlation (right) measure agreement with EK-FAC influence at matched per-example storage budgets. One-time PCA fitting. Fitting starts from cached first-stage coordinates of approximately 10,000 examples and follows Section B.2. Timings include reading the coordinates, computing the correction matrices, and saving them in FP16. This one-time cost is 18.4 minutes for the 32B model and is shared across examples and queries. Batched query processing. For the 32B model with mu=262,144m_u=262,144 and ku=2,048k_u=2,048, processing 100 queries against 1,000 stored examples takes 60.36 seconds on a single NVIDIA B200, starting from cached first-stage coordinates. This includes loading and transferring the FP16 correction matrices, applying the PCA projection, and scoring the stored examples. Each correction matrix is loaded once per batch, giving an amortized processing cost of 0.604 seconds per query. Appendix F Comparison under Shared EK-FAC Curvature Table 4: Pearson correlation between FP16 and one-bit influence scores on GPT-2. We report the median of the query-wise correlations over 481 queries. Storage budgets refer to one-bit storage per training example. The highest correlation at each budget is shown in bold. Method 12 KB 24 KB 96 KB EOGP 0.9020.902 0.8900.890 0.9190.919 LoGra (Random) 0.6040.604 0.4470.447 0.0860.086 LoGra (PCA) 0.7500.750 0.5700.570 −0.355-0.355 GraSS 0.3300.330 0.0790.079 −0.205-0.205 LoRIF 0.8200.820 0.8230.823 0.8220.822 Figure 7: LDS of compression methods under one-bit and FP16 storage on GPT-2 at matched per-example storage budgets. Horizontal lines denote uncompressed references. Curves end where the required projection dimension exceeds the method’s supported dimension. To examine whether EOGP benefits primarily from using the same curvature approximation as the evaluation reference, we construct variants of LoGra and GraSS that apply our EK-FAC half-whitening before their respective projections. On OLMo 2 7B SFT, EOGP substantially outperforms both variants at matched per-example storage budgets, showing that matching the reference curvature alone does not explain its advantage. The original LoGra and GraSS configurations also generally outperform their half-whitened variants, supporting the effectiveness of their post-projection curvature approximations in this setting. One possible explanation is that estimating curvature after projection accounts for the variances and correlations of the retained coordinates, whereas applying half-whitening before projection produces a different transformation. These results suggest that the effectiveness of a curvature approximation depends on how it is combined with gradient compression. Appendix G Effect of One-Bit Quantization on Attribution Quality We further examine how one-bit storage affects different gradient representations using the GPT-2 setting in Section 4.1. We compare attribution quality at matched per-example storage budgets and separately measure how quantization changes influence scores while holding the projection dimension fixed. Attribution quality under one-bit storage. Figure 7 compares one-bit and FP16 storage for the evaluated compression methods. For baselines that estimate curvature after projection, larger output dimensions increase the memory required for curvature estimation, so we evaluate their one-bit variants only up to the largest feasible storage budget. Under one-bit storage, EOGP achieves the highest LDS among the evaluated compression methods at each displayed budget, with performance improving as more storage is allocated. The one-bit variants of LoGra and GraSS exhibit a different trend, with LDS declining as the budget increases. LoRIF improves more gradually but remains below EOGP. At larger storage budgets, EOGP also outperforms the other compression baselines when all methods use FP16 storage, indicating that its advantage is not solely attributable to one-bit quantization. These results show that the effectiveness of one-bit storage depends on the representation being quantized, and that EOGP’s advantage persists in a retraining-based evaluation. Preservation of influence scores. To examine the effect of quantization directly, we compare influence scores obtained with FP16 and one-bit storage at the same projection dimension within each method. For each query, we compute the Pearson correlation between the two sets of scores across candidate training examples, then report the median correlation over queries in Table 4. For EOGP, these correlations range from 0.8900.890 to 0.9190.919 across the evaluated configurations, indicating that the scores remain strongly correlated after quantization. LoGra and GraSS show substantially weaker agreement in several configurations. These correlations measure the preservation of score patterns rather than absolute score magnitudes, since Pearson correlation is unchanged by a positive rescaling or an additive shift. Together with the LDS results, they provide complementary evidence that EOGP preserves useful attribution information under one-bit storage. A possible explanation. One possible explanation is that half-whitening makes the representation more compatible with scaled sign quantization. This quantization preserves coordinate signs but replaces their magnitudes with a common scale given by the mean absolute coordinate value. It therefore approximates vectors more accurately when their coordinate magnitudes are relatively uniform. By rescaling gradients using curvature information, half-whitening may reduce magnitude imbalances in the resulting coordinates and thereby lessen the distortion introduced by quantization. The relevant property is the relative variation in coordinate magnitudes, since uniformly rescaling a vector also rescales its quantization scale and leaves the relative reconstruction error unchanged.