Paper deep dive
TICDA: Tabular In-Context Data Attribution
Yacine Benihaddadene, Milan Bhan, Eliot Dugelay, Mohammed Jawhar, Benjamin Wong, Nicolas Chesneau, Duong Nguyen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Tabular foundation models (TFMs) achieve strong predictive performance by conditioning on labeled demonstrations provided in context, without any parameter update. Yet how individual demonstrations shape a given prediction remains poorly understood. This gap matters in practice: the context is often assembled from whatever labeled data is available, potentially leading to the inclusion of mislabeled, redundant, or low-quality examples that degrade performance. Standard data attribution methods do not transfer to the TFM setting: resampling-based approaches such as DemoShapley require a combinatorial number of forward passes, and gradient-based estimators such as influence functions require computing training point's effect on the model parameters, which in-context learning never updates. We introduce TICDA, a method that measures the influence of every demonstration in the context directly from linear surrogates trained on TFM latent embeddings, in a single forward pass and at negligible cost. We show that TICDA offers the best compromise against competitors across four tasks: detecting labeling errors, curating context to preserve predictive accuracy while lowering inference cost, producing attribution scores that transfer across TFMs, and supporting an acquisition strategy for efficient active learning.
Tags
Links
- Source: https://arxiv.org/abs/2610.07996v1
- Canonical: https://arxiv.org/abs/2610.07996v1
Trouble viewing inline? Open PDF directly →
Full Text
69,055 characters extracted from source content.
Expand or collapse full text
TICDA: Tabular In-Context Data Attribution Yacine Benihaddadene Affiliation: Ekimetrics Milan Bhan Affiliation: Ekimetrics Eliot Dugelay Affiliation: ETH Zurich*Equal contribution Mohammed Jawhar Affiliation: Ekimetrics Benjamin Wong Affiliation: Ekimetrics Nicolas Chesneau Duong Nguyen Abstract Tabular foundation models (TFMs) achieve strong predictive performance by conditioning on labeled demonstrations provided in context, without any parameter update. Yet how individual demonstrations shape a given prediction remains poorly understood. This gap matters in practice: the context is often assembled from whatever labeled data is available, potentially leading to the inclusion of mislabeled, redundant, or low-quality examples that degrade performance. Standard data attribution methods do not transfer to the TFM setting: resampling-based approaches such as DemoShapley require a combinatorial number of forward passes, and gradient-based estimators such as influence functions require computing training point’s effect on the model parameters, which in-context learning never updates. We introduce TICDA, a method that measures the influence of every demonstration in the context directly from linear surrogates trained on TFM latent embeddings, in a single forward pass and at negligible cost. We show that TICDA offers the best compromise against competitors across four tasks: detecting labeling errors, curating context to preserve predictive accuracy while lowering inference cost, producing attribution scores that transfer across TFMs, and supporting an acquisition strategy for efficient active learning. 1 Introduction Tabular foundation models (TFMs) have demonstrated that transformer-based architectures using in-context learning (ICL) (Dong et al., 2024) can achieve state-of-the-art performance on small- to medium-sized predictive tabular tasks (Erickson et al., 2025; Purucker et al., 2026). A TFM produces a prediction by conditioning on a set of labeled demonstrations in its context, without any parameter update. A succession of models has extended this paradigm to larger and more heterogeneous datasets (Hollmann et al., 2023; Qu et al., 2025; Hollmann et al., 2025; Qu et al., 2026; Ma et al., 2025), raising the interest in TFMs (van Breugel and van der Schaar, 2024). Despite this success, comparatively little is understood about how TFMs use their context, and in particular about the influence that each individual demonstration exerts on a given prediction (Rundel et al., 2024). This influence is not merely of interpretive interest: since the context is assembled from whatever labeled data happens to be available, mislabeled, redundant, or otherwise low-quality demonstrations can be introduced silently and either degrade predictive performance without any obvious signal or induce useless computational overhead. Those effects have been observed in TFMs: TabPFN loses accuracy when its demonstrations come from a distribution that drifts over time (Helli et al., 2024), unsupervised extensions of TabPFN to anomaly detection become unstable when their context is noisy or contaminated (Marszałek et al., 2026), and Rundel et al. (2024) show that a TabPFN context chosen by data valuation predicts better than a random context of the same size, which matters because the cost of TabPFN grows quadratically with the number of demonstrations. The combination of the widespread deployment of TFMs and the opacity of their in-context inference makes attributing predictions to individual demonstrations increasingly critical. Training data attribution (TDA) methods aim to quantify the influence that individual training points exert on a model’s behavior and performance (Hammoudeh and Lowd, 2024). Transposing this idea to TFMs is not straightforward: the relevant unit is a demonstration conditioning a frozen pretrained model through its context, rather than a training point that shaped a set of learned parameters. Gradient-based estimators such as influence functions (Koh and Liang, 2017) are defined through the model parameters and thus have no natural counterpart for a demonstration whose effect is exerted at inference time. Gradient-free resampling-based approaches such as Data Shapley (Ghorbani and Zou, 2019) require evaluating the model over a combinatorial number of data subsets, which is prohibitive when each evaluation amounts to a full forward pass over the context. To overcome these limitations, we introduce TICDA, a gradient-based approach for estimating the influence of every demonstration in the context at negligible cost. Our main contributions are as follows: 1. We propose TICDA, which measures the influence of every in-context demonstration directly from linear surrogates trained on TFM latent embeddings, in a single forward pass. 2. We show that TICDA detects labeling errors among demonstrations competitively with existing methods, while incurring lower cost. 3. We show that TICDA enables context curation that either improves or maintains predictive performance while reducing inference cost. 4. We establish that the attribution scores produced by TICDA transfer across heavier TFMs. 5. We derive an acquisition strategy from TICDA influence scores for efficient active learning. This paper is organized as follows. Section 2 introduces TICDA, and Section 3 shows how TICDA enables accurate labeling error detection, context curation, TFM transfer and active learning. Section 4 gives a review of how existing approaches compute data attribution, both for deep models in general and in the ICL setting. 2 TICDA: A Method to Compute fixed-representation influences In this section, we introduce TICDA, a method for measuring the influence that each demonstration of a TFM context exerts on a given prediction. We measure this influence as a leave-one-out (LOO) effect: the change of the loss on a labeled query zqz_q when demonstration i is removed and the prediction is made again without it, where zq=(xq,yq)z_q=(x_q,y_q) is an input xqx_q with its label yqy_q. Let =(xj,yj)j=1nD=\(x_j,y_j)\_j=1^n be a classification context with K classes and observed one-hot labels yj∈ℝKy_j ^K, let f be a frozen TFM, and write zj=(xj,yj)z_j=(x_j,y_j) for the j-th demonstration. The LOO influence of demonstration ziz_i on the query zqz_q is the change in the loss of the TFM when the demonstration is removed from the context: ΔiqTFM=ℒ(f(xq,∖i),yq)−ℒ(f(xq,),yq), ^TFM_iq=L (f(x_q;D_ i),\,y_q )-L (f(x_q;D),\,y_q ), (1) where f(xq,)f(x_q;D) is the prediction of the TFM for xqx_q given the context D, ∖iD_ i is the context without demonstration i, and ℒL is the predictive loss. A positive value of ΔiqTFM ^TFM_iq means that the demonstration helps the query, since its removal raises the loss. Evaluating Equation 1 for every demonstration requires n forward passes over contexts of n−1n-1 demonstrations, which is prohibitive for the contexts of thousands of demonstrations that TFMs are designed for. To avoid doing one forward pass per demonstration, we follow the influence function approach (Koh and Liang, 2017). Consider a model with parameters θ (pre)trained on the n demonstrations of D: its fitted parameters θ θ minimize the empirical risk 1n∑j=1nℒ(zj,θ) 1n _j=1^nL(z_j,θ), with ℒ(zj,θ)L(z_j,θ) the loss of its prediction on xjx_j against yjy_j. Upweighting a training point ziz_i by an infinitesimal amount changes the loss at zqz_q by ℐup,loss(zi,zq)=−∇θℒ(zq,θ^)⊤Hθ^−1∇θℒ(zi,θ^),I_up,loss(z_i,z_q)=- _θL(z_q, θ) H_ θ^-1\, _θL(z_i, θ), (2) where ∇θ _θ denotes the gradient with respect to the parameters and Hθ^=1n∑j=1n∇θ2ℒ(zj,θ^)H_ θ= 1n _j=1^n∇^2_θL(z_j, θ) is the Hessian. A positive value of ℐup,loss(zi,zq)I_up,loss(z_i,z_q) means that giving the training point ziz_i more weight raises the loss at zqz_q, and vice versa. ℐup,loss(zi,zq)I_up,loss(z_i,z_q) is the first-order estimate (up to a factor) of the change of ΔiqTFM ^TFM_iq (Koh and Liang, 2017). The influence function approach relies on differentiating the loss with respect to the model’s parameters θ, through which a training point exerts its effect. However, a TFM learns in context, and its parameters remain fixed during inference. A demonstration influences a prediction only through the context in which the model reads it and there are no parameter updates. Hence, the gradients required in Equation 2 cannot be computed. In order to overcome this limitation, TICDA operates in two steps: (1) TICDA fits a ridge surrogate on the representations that the TFM constructs for the demonstrations (Section 2.1), and (2) TICDA holds these representations fixed (Section 2.2). Step (1) provides the parameters that influence functions require, while step (2) allows a single forward pass to serve all demonstrations. Under these two choices, Equation 2 admits a closed-form solution yielding two scores, self-influence and query-influence (Section 2.3). Figure 1 gives an overview of the TICDA methodology and we detail its components in the following sections. 2.1 Attribution surrogate To obtain parameters to which every demonstration contributes, we propose to apply Equation 2 to a ridge linear probe fitted on the representations of the TFM rather than to its weights. Let ϕ(⋅∣)∈ℝdφ(· ) ^d denote the row representation read at a given hidden layer of f when the context is D. We write mj=ϕ(xj∣)m_j=φ(x_j ) for demonstration j, mq=ϕ(xq∣)m_q=φ(x_q ) for a query, and M∈ℝn×dM ^n× d and Y∈ℝn×KY ^n× K for the demonstration and label matrices. This notation makes the dependence on D explicit: in a TFM, the representation of a row is computed from the whole context, including its labels for the backbones that we study, and not from the row alone. On these representations, we fit a ridge surrogate to the observed labels: B^=argminBJ(B),J(B)=12∥MB−Y∥F2+λ2∥B∥F2, B= _BJ(B), J(B)= 12 MB-Y _F^2+ λ2 B _F^2, (3) where ∥⋅∥F\|·\|_F is the Frobenius norm and λ>0λ>0 is the regularization term. This objective admits the closed-form solution B^=H−1M⊤Y,H=M⊤M+λId. B=H^-1M Y, H=M M+λ I_d. (4) For a row with representation mjm_j, the surrogate predicts the class-score vector B^⊤mj B m_j, its residual rj=B^⊤mj−yjr_j= B m_j-y_j measures the discrepancy from the observed one-hot label, and ℓj(B)=ℓ(B⊤mj,yj) _j(B)= (B m_j,y_j) denotes its loss, with ℓ(s,y)=12∥s−y∥22 (s,y)= 12 s-y _2^2. The surrogate provides parameters, B B, to which every demonstration contributes and through which Equation 2 becomes applicable. Figure 1: TICDA overview. TICDA (1) extracts TFM embeddings, (2) trains a linear surrogate on the classification task of interest, (3) estimates influence score based on the surrogate and (4) can be applied for label-error detection, context curation, cross-TFM transfer and active learning. 2.2 Fixed representations simplification The surrogate alone does not remove the cost of Equation 1. Removing demonstration i changes the surrogate loss on the query by ΔiqR=ℓ(B^(∖i)⊤ϕ(xq∣∖i),yq)−ℓ(B^()⊤ϕ(xq∣),yq), ^R_iq= ( B(D_ i) φ(x_q _ i),\,y_q )- ( B(D) φ(x_q ),\,y_q ), (5) where B^() B(D) denotes the coefficients fitted on the context D, and this change has two distinct origins: (1) demonstration i leaves the regression, so that the coefficients are refitted, which amounts to a rank-one update of a d×d× d system, and (2) the context itself changes, so that the TFM re-encodes every remaining demonstration and the query, M(∖i)M(D_ i) being different from M()M(D) with one row removed. The second effect requires a forward pass of the TFM for each candidate deletion. Equation 5 is thus as expensive as Equation 1, and an approach willing to pay this cost would have no use for a surrogate, since it could measure the loss of the TFM directly. Motivated by the fact that one demonstration represents a small fraction of a context of thousands of demonstrations, we make the following simplification: for every demonstration i, ϕ(⋅∣∖i)≈ϕ(⋅∣).φ(· _ i)\;≈\;φ(· ). (6) We provide evidence in Appendix C.1 that Equation 6 is reasonable. Specifically, differences are approximately 0.5%0.5\% of the representation norm with 4,096 demonstrations and decrease as the number of demonstrations increases. All representations are extracted once, from the full context, and the removal of a demonstration only affects the fit of the surrogate. This choice has two consequences. First, a single forward pass serves all n demonstrations instead of one pass each. Second, the surrogate becomes an ordinary ridge regression on a fixed design, i.e., a strictly convex and twice-differentiable objective. That is the setting in which the influence function of Equation 2 holds. 2.3 Demonstration upweighting influence Under Equation 6, we follow Koh and Liang (2017) and measure a demonstration’s influence by slightly modifying its weight in the ridge objective and evaluating the resulting change in query loss. Changing the weight of demonstration i from 11 to 1+ϵ1+ε gives B^ϵ=argminB[J(B)+ϵℓi(B)], B_ε= _B [J(B)+ε\, _i(B) ], (7) where the representations and the global regularization penalty remain fixed. Differentiating the normal equations at ϵ=0ε=0 gives dB^ϵ/dϵ=−H−1miri⊤d B_ε/dε=-H^-1m_ir_i . No Hessian over the parameters of the TFM is involved. We define the influence of demonstration i on query q as siq:=−dℓq(B^ϵ)dϵ|ϵ=0=(mq⊤H−1mi)(rq⊤ri).s_iq:=- . d\, _q( B_ε)dε |_ε=0= (m_q H^-1m_i ) (r_q r_i ). (8) The sign indicates whether the demonstration locally helps or hurts the query under the surrogate: siq>0s_iq>0 means that increasing its weight reduces query loss, while siq<0s_iq<0 means that it increases query loss. The pairwise score siqs_iq describes the influence of demonstration i on the surrogate loss at a query q. Evaluating it on the demonstration itself or aggregating it over several queries yields two complementary summaries, which can be used for different applications. Self-influence. Using demonstration i’s own representation and label as the query in Equation 8 gives its self-influence: Ti=sii=hi∥ri∥22T_i=s_i=h_i\, r_i _2^2. Defined this way, TiT_i is the first-order change of the demonstration’s own loss when it is removed. This score is the squared residual multiplied by the leverage hih_i (Appendix B.1). Demonstrations with a large TiT_i are those whose observed label is not supported by the rest of the context. Hence, TiT_i can be used to detect labeling errors, as we show in Section 3. Query-influence. Given a set V of queries, the mean influence ui=1||∑q∈siqu_i= 1|V| _q s_iq shows how demonstration i supports the predictions on V. Demonstrations with a negative uiu_i are those whose removal lowers the mean loss on V. Hence, uiu_i can be used to curate the context, by removing the demonstrations with the lowest uiu_i computed on a labeled validation set (Section 3). 3 Experimental Settings This section presents the experimental study conducted across 38 datasets from TabArena (Erickson et al., 2025) and several TFMs. We first run TICDA on two TFMs to assess how well it detects labeling errors, including a sensitivity analysis over the hidden space size and the influence computation method (self or query). We then show how TICDA enables to reduce context size, and therefore the inference cost of TFMs, without sacrificing performance, even improving it in some cases. Finally, we show how TICDA can be used to build an acquisition strategy for active learning under low data regime. 3.1 Experimental Setup Datasets. TICDA and competitors are tested on 38 classification datasets from TabArena. We follow the preprocessing and evaluation protocol of TabArena and limit each context to at most 4,0964,096 demonstrations. All methods use the same context and test splits on each dataset. Models. We consider two frozen TFMs as backbone for TICDA: TabICLv2 (Qu et al., 2026) and TabDPT 1.2 (Ma et al., 2025). TICDA reads the row representations of TabICLv2 before its in-context learning layers and those of TabDPT after the 16th of its 32 transformer layers. No backbone parameters are updated during attribution. We first evaluate each attribution method independently on every backbone. Then, to test whether the scores transfer to models that TICDA never reads, we rank the demonstrations with one of these two backbones and apply this ranking unchanged to curate the context of TabPFN-3 (Grinsztajn et al., 2026) and TabPFN-3.5 (Jäger et al., 2026). Evaluation Protocols and Metrics. We evaluate TICDA through LOO estimation, labeling error detection, computational efficiency, context curation, transfer to other TFMs, and acquisition strategy for active learning. For LOO estimation, we compute the Spearman correlation between TICDA and exhaustive leave-one-out (LOO), as defined in Equation 1, and refer to this agreement as faithfulness. For labeling error detection, we follow Ghorbani and Zou (2019): we corrupt the labels of 20%20\% of the context demonstrations, treat them as positives, rank demonstrations by self-influence (TiT_i), and report the AUC of the recovered ranking. We report attribution runtime to assess computational efficiency. For context curation, under a fixed budget we rank demonstrations by their mean influence (uiu_i) on a validation set, remove the lowest-scored, and re-evaluate the backbone on the unchanged test set, comparing balanced accuracy against the uncurated context and against random removal. For active learning, we test whether TICDA can guide data acquisition in a pool-based setting by acquiring the candidates whose addition most reduces the surrogate loss on a labeled validation set, and measure label efficiency through the area under the learning curve. TICDA and Competitors. We use the base configuration of TICDA introduced in Section 2. Given the frozen row representations, TICDA fits a ridge-regularized linear probe and derives the self-influence scores (TiT_i) used for label-error detection and the mean query-influence scores (uiu_i) used for context curation without further calls to the backbone. We fix the ridge regularization coefficient to λ=10λ=10 for all datasets and backbones, without dataset-specific tuning. We also implement TICDA-IT for context curation experiments, consisting in performing iterative curation, where 5%5\% of the demonstrations are removed at each step, followed by refitting and re-ranking with the TICDA data attribution computation protocol, until the target removal threshold is reached. We compare TICDA to four competitors. We also run Integrated gradients (IG) (Sundararajan et al., 2017) to estimate each demonstration’s contribution by integrating prediction gradients along a path from a reference input, constituted by replacing the input demonstration of interest with padding. We implement other competitors that we present in more details in Section 4: DETAIL (Zhou et al., 2024) (following basic setting from the paper) and an affordable version of DemoShapley (Xie et al., 2025) for TFMs to approximate Shapley values from eight Monte Carlo context permutations. Beyond being used as a reference to compute faithfulness, we also compare our results to Leave-one-out (LOO) for labeling error detection and context curation. Table 1: Data attribution for in-context demonstrations on tabular foundation models. Results are aggregated over 38 datasets (mean ± std). Best results are bold, second best underlined; stars compare the winner with the runner-up (two-sided paired t-test, ∗p<.10^*p<.10, p∗∗<.05^**p<.05, ∗p<.01^***p<.01). LOO DemoShapley IG DETAIL TICDA (ours) TabDPT Faith. ↑ — 0.91∗0.91^*** ±0.07± 0.07 0.200.20 ±0.35± 0.35 −0.06-0.06 ±0.18± 0.18 0.79¯ 0.79 ±0.09± 0.09 AUC ↑ 0.90∗0.90^*** ±0.08± 0.08 0.88¯ 0.88 ±0.10± 0.10 0.600.60 ±0.20± 0.20 0.470.47 ±0.13± 0.13 0.88¯ 0.88 ±0.08± 0.08 TabICL Faith. ↑ — 0.65¯ 0.65 ±0.15± 0.15 0.300.30 ±0.22± 0.22 −0.11-0.11 ±0.22± 0.22 0.670.67 ±0.21± 0.21 AUC ↑ 0.80¯ 0.80 ±0.16± 0.16 0.760.76 ±0.11± 0.11 0.640.64 ±0.18± 0.18 0.370.37 ±0.17± 0.17 0.89∗0.89^*** ±0.08± 0.08 3.2 Experimental Results LOO Faithfulness and Labeling Error Detection. Table 1 reports results aggregated over the 38 datasets from TabArena. Across both backbones, TICDA ranks first or second on every metric. On TabDPT, TICDA is the second-best method for both faithfulness (0.790.79) and AUC (0.880.88), trailing DemoShapley on faithfulness and tying it on labeling error detection. On TabICL, TICDA reaches the best faithfulness (0.670.67) and the best AUC (0.890.89), the latter significantly above LOO. No competitor is this consistent: DemoShapley is strong on TabDPT but degrades on TabICL, while IG and DETAIL lead to poor results both in terms of faithfulness and labeling error detection. Figure 2: Average computation time per method. Figure 2 additionally reports computation time per method (log scale). TICDA and DETAIL are the only methods that run in under three seconds (1.881.88–2.742.74 s), two to three orders of magnitude faster than LOO, DemoShapley, and IG. DETAIL is cheap but, as Table 1 shows, performs poorly on faithfulness and labeling error detection. TICDA is thus the only method that pairs top-tier faithfulness and labeling-error detection with negligible cost, making it the best overall compromise. We ablate two design choices of TICDA: the influence score used to rank demonstrations and the dimensionality of the representations on which attribution is performed. We compare self-influence TiT_i with query-influence, as defined in Section 2.3. Following Zhou et al. (2024), we also compare the full 512512-dimensional TabICL representations with a random projection to 20%20\% of their original dimension, corresponding to 102102 dimensions. Results for the reduced representations are averaged over three projection seeds. We report the same quality metrics as above: faithfulness with LOO, label-error detection performance (AUC), and computation runtime. Table 2: Ablation study on TabICL (38 datasets, 20%20\% label corruption, mean ± std). Configuration Metrics Influence Dimension Faith. ↑ AUC ↑ Time (s) ↓ query 20%20\% 0.32±0.160.32_ ± 0.16 0.85±0.100.85_ ± 0.10 4.90±1.304.90_ ± 1.30 query full 0.33±0.160.33_ ± 0.16 0.86±0.100.86_ ± 0.10 5.09±1.315.09_ ± 1.31 self 20%20\% 0.67±0.210.67_ ± 0.21 0.89±0.080.89_ ± 0.08 3.51±0.683.51_ ± 0.68 self full 0.67±0.210.67_ ± 0.21 0.89±0.080.89_ ± 0.08 3.71±0.693.71_ ± 0.69 Table 2 shows that the influence scoring methodology has a substantially larger effect than the representation dimensionality. With full representations, self-influence significantly increases faithfulness (from 0.33 to 0.67) and AUC (from 0.86 to 0.89) relative to query-influence. Reducing the representation dimension leaves AUC and faithfulness unchanged, while only slightly improving computation time. These results especially support the use of TICDA self-influence for label-error detection. We additionally show in Appendix E the sensitivity of our approach to the regularization coefficient. Figure 3: Curation: aggregated balanced accuracy evolution on TabDPT and TabICL under different levels of demonstration corruption. Each method ranks in-context demonstrations by its attribution scores, and the lowest-scored demonstrations are removed. Context Curation. Figure 3 reports aggregated curation results on the 38 selected TabArena datasets, at two initial corruption levels (20%20\% and 40%40\%) and across increasing fractions of removed context. TICDA-IT is the most robust curation strategy across both settings: it keeps balanced accuracy at or above the full-context level as demonstrations are removed, whereas IG and DETAIL degrade sharply, DemoShapley tracks the random baseline, and LOO curates well only at 20%20\% corruption. The gain is clearest under heavy corruption: at 40%40\%, the balanced accuracy of both TICDA and TICDA-IT increases with removal and peaks around 4040–45%45\%, so that discarding up to half of the demonstrations outperforms the full context. Read together with the compute results of Figure 2, this shows that TICDA-based curation preserves or improves accuracy while using up to 50%50\% fewer demonstrations, at a small fraction of the cost of LOO, the only baseline of comparable curation quality. Finally, TICDA-IT consistently outperforms TICDA, most visibly at 20%20\% corruption where TICDA alone drifts below the full-context level. Recomputing data attribution during curation is therefore worthwhile. Table 3: TICDA TDA Transfer for context curation between source and target models. Balanced accuracy (mean ± std, aggregated over all datasets) is reported at several levels of removal. Demonstrations removed Target Source 0%0\% 10%10\% 20%20\% 50%50\% TabPFN-3 Random 59.059.0 ±14.9± 14.9 58.758.7 ±14.6± 14.6 58.358.3 ±14.4± 14.4 56.656.6 ±14.1± 14.1 TabICLv2 59.059.0 ±14.9± 14.9 63.563.5 ±15.8± 15.8 65.065.0 ±15.8± 15.8 64.464.4 ±15.4± 15.4 TabDPT 59.059.0 ±14.9± 14.9 62.662.6 ±15.7± 15.7 64.064.0 ±15.7± 15.7 63.463.4 ±15.2± 15.2 TabPFN-3.5 Random 60.460.4 ±15.5± 15.5 60.160.1 ±15.2± 15.2 59.859.8 ±15.1± 15.1 57.957.9 ±14.3± 14.3 TabICLv2 60.460.4 ±15.5± 15.5 64.164.1 ±15.9± 15.9 64.564.5 ±15.9± 15.9 64.664.6 ±15.5± 15.5 TabDPT 60.460.4 ±15.5± 15.5 63.063.0 ±16.0± 16.0 63.863.8 ±16.0± 16.0 63.163.1 ±15.5± 15.5 Table 3 demonstrates that attributions computed on one backbone transfer effectively to curate demonstrations for unseen targets (TabPFN-3 and TabPFN-3.5). At 50% removal using transferred scores from either TabICLv2 or TabDPT, balanced accuracy reaches approximately 64%, exceeding full-context baselines by over four points and substantially outperforming random removal. Transfer is robust across source models, with TabDPT performing comparably to the stronger TabICLv2. 3.3 TDA for TFM Active Learning TICDA-based Acquisition Strategy. We finally investigate whether TICDA can guide data acquisition in pool-based active learning with TabICLv2, on the same training/validation/test splits as the curation experiment. For each dataset, the pool P is the training split, drawn anew in each repetition, for larger training sets. The labeled context 0D_0 starts with n0=32n_0=32 demonstrations of P, one per class and the others drawn uniformly, and the rest of P forms the unlabeled pool 0U_0. At each round t, a strategy selects a batch ℬtB_t of btb_t candidates from tU_t, whose labels are revealed and which join the context: t+1=t∪ℬtD_t+1=D_t _t. The batch grows with the context, bt=max(4,2⌊log2(|t|/8)⌋)b_t= (4,2 _2(|D_t|/8) ), and acquisition stops once half of the pool is labeled. TICDA acquires the candidates whose addition would most reduce the surrogate loss on a labeled validation set V. At each round, the representations of the context, the candidates and V are read in a single forward pass with tD_t as context and held fixed (Equation 6), and the surrogate of Equation 4 is fitted on the context with λ=10λ=10. Adding a candidate x with label c to the context changes this surrogate in closed form (Equation 11 with t=−1t=-1): B^(+x,c)=B^−11+hxH−1mx(B^⊤mx−ec)⊤ B(+x,c)= B- 11+h_x\,H^-1m_x ( B m_x-e_c ) with hx=mx⊤H−1mxh_x=m_x H^-1m_x where ece_c is the one-hot vector of class c. TICDA scores x under its most favorable label: score(x)=maxc[L(B^)−L(B^(+x,c))]score(x)= _c [L_V ( B )-L_V ( B(+x,c) ) ], with L(B)=1K∑k=1K1|k|∑q∈kℓ(B⊤mq,yq)L_V(B)= 1K _k=1^K 1|V_k| _q _k (B m_q,y_q ), where kV_k holds the validation examples of class k, so that every class weighs equally, as in balanced accuracy. Each batch is built greedily: after each selection, H is updated with the representation of the selected candidate, a rank-one update that requires no label, and the remaining candidates are rescored; B B is refitted once the labels of the batch are revealed. Because early contexts contain far fewer demonstrations than the d=512d=512 dimensions of the representations, the representations are first projected onto dt=min(512,max(4,⌊|t|/5⌋))d_t= (512, (4, |D_t|/5 ) ) Gaussian random directions, redrawn at every round, so that the surrogate is fitted with about five demonstrations per dimension; as TabICL averages eight ensemble views, one surrogate is fitted per view and their predictions are averaged. V is a class-stratified sample of the validation split, drawn once per repetition, that holds 20%20\% of the final budget (at least 32 examples); it is never added to the context, is not counted in the labeling budget, and is not used by the baselines. Experimental Protocol. We compare TICDA with random sampling, which draws the btb_t candidates uniformly, and with uncertainty sampling (Lewis and Gale, 1994), a strong baseline that selects the btb_t candidates on which the TFM is least certain. We name this baseline entropy, since it measures this uncertainty by the predictive entropy −∑k=1Kpk(x)logpk(x)- _k=1^Kp_k(x) p_k(x), where p(x)p(x) is the class distribution predicted by the TFM given tD_t. All strategies share the initial contexts, pools, test examples and batch sizes. After each round, we measure the balanced accuracy on the test split, and summarize each learning curve by its normalized area under the curve (AULC): the average balanced accuracy over log2|t| _2|D_t|, from 32 labels to the end of the budget or to 512 labels. Results are averaged over R=5R=5 repetitions, which differ in the initial context, the pool subsample and V, on 27 datasets: 24 from TabArena, jannis (Grinsztajn et al., 2022), higgs (Baldi et al., 2014) and otto (Purucker et al., 2026). Experimental Results. Table 4 reports the aggregated results and Figure 4 shows an example of a learning curve on the APSFailure Dataset from TabArena. TICDA achieves the best average AULC across both the full budget (68.35%) and the data-constrained regime with ≤512≤ 512 labels (69.02%), outperforming entropy sampling and random baselines. Full budget ≤512≤ 512 labels (27 datasets) (23 datasets) Random 66.23 ± 14.17 66.38 ± 11.72 entropy 67.56 ± 14.95 67.51 ± 12.66 TICDA 68.35∗ ± 14.74 69.02∗ ± 12.09 Table 4: Active learning: normalized area under the balanced-accuracy curve (×100× 100, mean ± std over datasets). Bold: best; underlined: second; ∗: paired t-test against the runner-up, p<.01p<.01. Figure 4: Balanced accuracy evolution on APSFailure against the number of labeled demonstrations. 4 Related Work Training data attribution (TDA) methods attribute model performance to individual training points, grouped into three categories (Hammoudeh and Lowd, 2024): (1) resampling-based methods, which retrain models on perturbed subsets; (2) gradient-based methods, which approximate influence from gradients; and (3) training-dynamics methods, which track influence across checkpoints. Their forms depend on the training regime: classical training (where point effects are absorbed into weights) versus in-context learning (where examples influence inference directly). We present in the following on resampling-based and gradient-based approaches, the only families developed for both regimes (Deng et al., 2025). Resampling-based Approaches. Resampling-based approaches measure how a prediction or the model’s performance changes under finite perturbations of the training set, removing or reweighting individual points, mirroring perturbation-based feature attribution such as SHAP (Lundberg and Lee, 2017). The elementary case is leave-one-out (LOO) retraining, formalized for linear regression by Cook’s distance (Cook, 1977): a single point is removed and the model refit. Data Shapley (Ghorbani and Zou, 2019) values each example by its marginal contribution to performance, at the cost of repeatedly retraining the model. In-Run Data Shapley (Wang et al., 2025b) makes this affordable by defining a variant of these values for one specific training run, which it computes from the gradients of that run. Data Shapley extends naturally to the ICL setting, where the demonstrations play the role of the training set. DemoShapley (Xie et al., 2025) values each demonstration in the prompt used for Large Language Models (LLMs) by its marginal contribution across resampled prompt permutations. While this approach enables to detect high value data to improve LLM performance, this is only affordable for the few-shot LLM setting it targets, but not for the long contexts of TFMs, where the subsets grow combinatorially and each needs a full forward pass. Gradient-based Approaches. Gradient-based approaches approximate a training point’s effect through the model’s gradients, without retraining. Influence functions (Koh and Liang, 2017) provide a first-order Taylor approximation to LOO retraining, approximating the effect of infinitesimally upweighting a training point on the loss at a test point, which is the formulation of Equation 2 from which TICDA is derived. The Hessian being computationally expensive to compute, several follow-up works have been proposed to make it tractable for deep models by either estimating the Hessian more efficiently (Schioppa et al., 2022; Kwon et al., 2024; Wang et al., 2025a) or by tracing gradients along the training run (Pruthi et al., 2020). However, these approaches do not transfer naturally to ICL demonstrations, since a demonstration updates no model parameter. DETAIL (Zhou et al., 2024) addresses the aforementioned limitations for Large Language Models (LLMs) and is the method conceptually closest to TICDA. Building on the view that transformers implement an internal optimizer during ICL (von Oswald et al., 2023), it models this learner as a linear ridge probe on the hidden states of the demonstrations and applies Equation 2 to this probe rather than to the model’s weights, a surrogate view we share. Its motivation, design, and interpretation, however, do not transfer to TFMs, because DETAIL is built for LLM prompts. The two methods differ in three respects: the regime in which the probe is fitted, since LLM prompts hold few demonstrations in a high-dimensional space whereas TFM contexts hold many rows in a low-dimensional one; the perturbation each score differentiates; and the treatment of contextual representations under a single forward pass. We defer the detailed analysis to Appendix C. 5 Discussion and Conclusion TICDA rests on the fixed-representation approximation of Equation 6: removing a demonstration is assumed to leave the representations of the remaining rows and the query unchanged, so that only the surrogate fit is refitted. Appendix C.1 shows that this is well justified when a single demonstration is a small fraction of a large context. Our faithfulness evaluation compares TICDA with exhaustive LOO on all 38 TabArena datasets, but only with contexts of at most 4,0964,096 demonstrations. Longer contexts, which TICDA is meant to serve, remain untested against this reference, since exhaustive deletion requires one forward pass of the TFM per demonstration. We see TICDA as a promising methodology to optimize context under ICL settings that could be integrated into other existing framework (Bhan et al., 2024; Yang et al., 2023; Xu et al., 2023) carefully targeting ICL examples. Finally, extending TICDA beyond the fixed-representation approximation and to regression and multimodal settings are natural directions for future work. In this paper we introduced TICDA, a gradient-based data attribution method for tabular foundation models that estimates the influence of every in-context demonstration in a single forward pass. TICDA overcomes the central obstacle to applying influence functions under in-context learning, the absence of a parameter update, by fitting a ridge surrogate on the frozen row representations and holding them fixed under demonstration removal, yielding closed-form influence scores at negligible cost. Across 38 TabArena datasets and two backbones, TICDA is the only method that pairs top-tier faithfulness and labeling-error detection with a runtime well below LOO, DemoShapley, and IG. Its curation preserves or improves model performance while discarding up to half of the demonstrations, and the scores transfer across TFMs, allowing attribution to be computed once and amortized. AI use statement Generative AI tools have been used to polish the paper and help in generating tables and figures. These tools also helped in reviewing the manuscript to detect typos and inconsistencies. We also used AI tools to accelerate code production. We take full responsibility for the final content. References Baldi et al. (2014) P. Baldi, P. Sadowski, and D. Whiteson Searching for exotic particles in high-energy physics with deep learning. Nature Communications 5, p. 4308. External Links: Document Cited by: §3.3. Bhan et al. (2024) M. Bhan, J. Vittaut, N. Chesneau, and M. Lesot Self-AMPLIFY: improving small language models with self post hoc explanations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 10974–10991. External Links: Document Cited by: §5. Cook (1977) R. D. Cook Detection of influential observation in linear regression. Technometrics 19 (1), p. 15–18. External Links: Document Cited by: §4. Deng et al. (2025) J. Deng, Y. Hu, P. Hu, T. Li, S. Liu, J. T. Wang, D. Ley, Q. Dai, B. Huang, J. Huang, et al. A survey of data attribution: methods, applications, and evaluation in the era of generative AI. Note: SSRN preprint External Links: Link, Document Cited by: §4. Dong et al. (2024) Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, B. Chang, et al. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 1107–1128. External Links: Document Cited by: §1. Erickson et al. (2025) N. Erickson, L. Purucker, A. Tschalzev, D. Holzmüller, P. M. Desai, D. Salinas, and F. Hutter TabArena: a living benchmark for machine learning on tabular data. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, Vol. 38. Cited by: §1, §3. Ghorbani and Zou (2019) A. Ghorbani and J. Zou Data Shapley: equitable valuation of data for machine learning. In International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 97, p. 2242–2251. Cited by: §1, §3.1, §4. Grinsztajn et al. (2026) L. Grinsztajn, K. Flöge, O. Key, F. Birkel, et al. TabPFN-3: technical report. arXiv preprint arXiv:2605.13986. Cited by: §3.1. Grinsztajn et al. (2022) L. Grinsztajn, E. Oyallon, and G. Varoquaux Why do tree-based models still outperform deep learning on typical tabular data?. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, Vol. 35, p. 507–520. Cited by: §3.3. Hammoudeh and Lowd (2024) Z. Hammoudeh and D. Lowd Training data influence analysis and estimation: a survey. Machine Learning 113 (5), p. 2351–2403. External Links: Document Cited by: §1, §4. Helli et al. (2024) K. Helli, D. Schnurr, N. Hollmann, S. Müller, and F. Hutter Drift-resilient TabPFN: in-context learning temporal distribution shifts on tabular data. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 37, p. 98742–98781. Cited by: §1. Hollmann et al. (2023) N. Hollmann, S. Müller, K. Eggensperger, and F. Hutter TabPFN: a transformer that solves small tabular classification problems in a second. In International Conference on Learning Representations (ICLR), Cited by: §1. Hollmann et al. (2025) N. Hollmann, S. Müller, L. Purucker, A. Krishnakumar, M. Körfer, S. B. Hoo, R. T. Schirrmeister, and F. Hutter Accurate predictions on small data with a tabular foundation model. Nature 637 (8045), p. 319–326. External Links: Document Cited by: §1. Jäger et al. (2026) B. Jäger, N. Erickson, L. Grinsztajn, F. Birkel, et al. TabPFN-3.5: technical report. arXiv preprint arXiv:2609.17895. Cited by: §3.1. Koh and Liang (2017) P. W. Koh and P. Liang Understanding black-box predictions via influence functions. In International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 70, p. 1885–1894. Cited by: §B.1, §1, §2.3, §2, §2, §4. Kwon et al. (2024) Y. Kwon, E. Wu, K. Wu, and J. Y. Zou DataInf: efficiently estimating data influence in LoRA-tuned LLMs and diffusion models. In International Conference on Learning Representations (ICLR), p. 21921–21942. Cited by: §4. Lewis and Gale (1994) D. D. Lewis and W. A. Gale A sequential algorithm for training text classifiers. In SIGIR ’94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, p. 3–12. External Links: Document Cited by: §3.3. Lundberg and Lee (2017) S. M. Lundberg and S. Lee A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 30. Cited by: §4. Ma et al. (2025) J. Ma, V. Thomas, R. Hosseinzadeh, A. Labach, H. Kamkari, J. C. Cresswell, K. Golestan, G. Yu, A. L. Caterini, and M. Volkovs TabDPT: scaling tabular foundation models on real data. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 38, p. 172692–172722. Cited by: §1, §3.1. Marszałek et al. (2026) P. Marszałek, T. Kuśmierczyk, and M. Śmieja TACTIC for navigating the unknown: tabular anomaly detection via in-context inference. arXiv preprint arXiv:2603.14171. Cited by: §1. Paszke et al. (2017) A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer Automatic differentiation in PyTorch. In NIPS 2017 Workshop on Autodiff, Cited by: Appendix A. Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al. Scikit-learn: machine learning in Python. Journal of Machine Learning Research 12, p. 2825–2830. Cited by: Appendix A. Pruthi et al. (2020) G. Pruthi, F. Liu, S. Kale, and M. Sundararajan Estimating training data influence by tracing gradient descent. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33, p. 19920–19930. Cited by: §4. Purucker et al. (2026) L. Purucker, A. Tschalzev, N. Erickson, G. Blayer, D. Holzmüller, A. Arazi, A. Pfefferle, M. Tajjar, G. Varoquaux, and F. Hutter Beyond IID: how general are tabular foundation models, really?. arXiv preprint arXiv:2606.30410. Cited by: §1, §3.3. Qu et al. (2025) J. Qu, D. Holzmüller, G. Varoquaux, and M. Le Morvan TabICL: a tabular foundation model for in-context learning on large data. In International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 267, p. 50817–50847. Cited by: §1. Qu et al. (2026) J. Qu, D. Holzmüller, G. Varoquaux, and M. Le Morvan TabICLv2: a better, faster, scalable, and open tabular foundation model. In International Conference on Machine Learning (ICML), Note: arXiv:2602.11139 Cited by: §1, §3.1. Rundel et al. (2024) D. Rundel, J. Kobialka, C. von Crailsheim, M. Feurer, T. Nagler, and D. Rügamer Interpretable machine learning for TabPFN. In Explainable Artificial Intelligence (xAI 2024), Communications in Computer and Information Science, p. 465–476. External Links: Document Cited by: §1. Schioppa et al. (2022) A. Schioppa, P. Zablotskaia, D. Vilar, and A. Sokolov Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, p. 8179–8186. External Links: Document Cited by: §4. Sherman and Morrison (1950) J. Sherman and W. J. Morrison Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. The Annals of Mathematical Statistics 21 (1), p. 124–127. External Links: Document Cited by: §B.2. Sundararajan et al. (2017) M. Sundararajan, A. Taly, and Q. Yan Axiomatic attribution for deep networks. In International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 70, p. 3319–3328. Cited by: Appendix D, §3.1. van Breugel and van der Schaar (2024) B. van Breugel and M. van der Schaar Position: why tabular foundation models should be a research priority. In International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 235, p. 48976–48993. Cited by: §1. von Oswald et al. (2023) J. von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov Transformers learn in-context by gradient descent. In International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 202, p. 35151–35174. Cited by: §4. Wang et al. (2025a) A. Wang, E. Nguyen, R. Yang, J. Bae, S. A. McIlraith, and R. Grosse Better training data attribution via better inverse Hessian-vector products. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 38, p. 101825–101860. Cited by: §4. Wang et al. (2025b) J. T. Wang, P. Mittal, D. Song, and R. Jia Data Shapley in one training run. In International Conference on Learning Representations (ICLR), p. 12358–12395. Cited by: §4. Wolf et al. (2019) T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. HuggingFace’s transformers: state-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. Cited by: Appendix A. Xie et al. (2025) S. Xie, M. Luo, C. D. Stern, M. Du, and L. Cheng DemoShapley: valuation of demonstrations for in-context learning. In 2025 IEEE International Conference on Big Data (BigData), p. 4081–4090. External Links: Document Cited by: §3.1, §4. Xu et al. (2023) B. Xu, Q. Wang, Z. Mao, Y. Lyu, Q. She, and Y. Zhang kkNN prompting: beyond-context learning with calibration-free nearest neighbor inference. In International Conference on Learning Representations (ICLR), Cited by: §5. Yang et al. (2023) Z. Yang, Y. Zhang, D. Sui, C. Liu, J. Zhao, and K. Liu Representative demonstration selection for in-context learning with two-stage determinantal point process. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 5443–5456. External Links: Document Cited by: §5. Zhou et al. (2024) Z. Zhou, X. Lin, X. Xu, A. Prakash, D. Rus, and B. K. H. Low DETAIL: task DEmonsTration attribution for interpretable in-context learning. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 37, p. 25284–25310. Cited by: Appendix C, Appendix D, §3.1, §3.2, §4. Appendix A Scientific Libraries We used several open-source libraries in this work: pytorch (Paszke et al., 2017), HuggingFace transformers (Wolf et al., 2019) and sklearn (Pedregosa et al., 2011). We will make our code public upon acceptance. Appendix B Derivation of the influence scores This appendix derives the scores of Section 2 and relates them to the exact quantities that they approximate. We first prove the closed form of the influence score (Equation 8), then give the exact effect of deleting a demonstration when the representations are held fixed, which shows what a first-order score leaves out, state what the fixed-representation simplification of Equation 6 itself neglects, and finally measure how much the representations change when a demonstration is removed. Let aj=mjrj⊤∈ℝd×Ka_j=m_jr_j ^d× K denote the gradient of the loss ℓj _j of row j at B B, and ⟨U,V⟩F=tr(U⊤V) U,V _F=tr(U V) denote the Frobenius inner product of two matrices. B.1 First-order influence To obtain Equation 8, we need the derivative of the fitted coefficients with respect to the weight of a demonstration, since the query loss depends on this weight only through the coefficients. We follow the derivation of influence functions (Koh and Liang, 2017), every step of which is available in closed form for a ridge surrogate. The perturbed objective of Equation 7 is a ridge regression in which demonstration i has weight 1+ϵ1+ε, so that its minimizer B^ϵ B_ε satisfies the normal equations: (H+ϵmimi⊤)B^ϵ=M⊤Y+ϵmiyi⊤, (H+ε\,m_im_i ) B_ε=M Y+ε\,m_iy_i , where H=M⊤M+λIdH=M M+λ I_d is positive definite for λ>0λ>0. Differentiating these equations at ϵ=0ε=0 and using ri=B^⊤mi−yir_i= B m_i-y_i gives dB^ϵdϵ|ϵ=0=−H−1miri⊤=−H−1ai. . d B_εdε |_ε=0=-H^-1m_ir_i =-H^-1a_i. (9) The query loss depends on ϵε only through the coefficients, and its gradient with respect to B at B B is aqa_q, so that the chain rule gives dℓq(B^ϵ)dϵ|ϵ=0=⟨aq,dB^ϵdϵ|ϵ=0⟩F=−⟨aq,H−1ai⟩F. . d\, _q( B_ε)dε |_ε=0= a_q,\, . d B_εdε |_ε=0 _F=- a_q,\,H^-1a_i _F. Equation 8 defines siqs_iq as the opposite of this derivative, and the identity ⟨mqrq⊤,H−1miri⊤⟩F=(mq⊤H−1mi)(rq⊤ri) m_qr_q ,H^-1m_ir_i _F=(m_q H^-1m_i)(r_q r_i) gives its closed form. The matrix H is the Hessian of the surrogate objective with respect to each of the K columns of B, so that the inverse Hessian of Equation 2 reduces to K copies of the inverse of a d×d× d matrix. No Hessian over the parameters of the TFM is therefore involved. Equation 2 is written for an objective that averages the losses, whereas Equation 3 sums them. The two conventions define the same surrogate when the averaged objective uses the coefficient λ/nλ/n, and they give scores that differ by the constant factor n, which leaves every ranking unchanged. Taking the demonstration itself as the query gives the self-influence equation introduced in Section 2.3, Ti=hi∥ri∥22T_i=h_i r_i _2^2, where hi=mi⊤H−1mih_i=m_i H^-1m_i (10) is the ridge leverage of demonstration i, that is, the sensitivity of the prediction of the surrogate for this demonstration to its own label. Since H−mimi⊤H-m_im_i remains positive definite, the leverages satisfy 0≤hi<10≤ h_i<1, and their sum is ∑ihi=d−λtr(H−1)<d _ih_i=d- (H^-1)<d, so that the mean leverage of a context is below d/nd/n. In a tabular context, where n exceeds d by an order of magnitude, the leverage therefore multiplies the squared residual by a small factor on average, and this factor is largest for the demonstrations whose representation lies in a direction that few other demonstrations occupy. The self influence equation introduced in Section 2.3 uses, on both sides of the product, the representation that the TFM computes for row i in its role of demonstration. Encoding the same row as a query may give a different vector, since a TFM receives a demonstration with its label and a query without it; the corresponding score would require an additional forward pass, and we do not use it. B.2 Exact deletion at fixed representations The score of Equation 8 is a derivative, whereas deleting a demonstration changes its weight from one to zero. That raises the question of how far a first-order score is from the effect of an actual deletion. When the representations are held fixed and the surrogate is a ridge regression on a fixed design, this question has a closed-form answer. We derive this exact effect below, first on a query and then on the demonstration itself, and show that it differs from the first-order score by corrections that vanish with the leverage, so that the two nearly coincide in a tabular context. This exact effect also serves as a reference for TICDA in Section 3. Let B^(t) B(t) minimize J(B)−tℓi(B)J(B)-t\, _i(B) for 0≤t≤10≤ t≤ 1, so that t=1t=1 corresponds to the context without demonstration i. Since this objective differs from J by a term of rank one, the Sherman–Morrison identity (Sherman and Morrison, 1950) gives B^(t)−B^=t1−thiH−1miri⊤, B(t)- B= t1-t\,h_i\,H^-1m_ir_i , (11) whose derivative at t=0t=0 is the opposite of Equation 9. Writing aqi=mq⊤H−1mia_qi=m_q H^-1m_i, the deletion therefore changes the loss of the surrogate on query q by ΔiqF=aqi1−hirq⊤ri+aqi22(1−hi)2∥ri∥22, ^F_iq= a_qi1-h_i\,r_q r_i+ a_qi^22(1-h_i)^2\, r_i _2^2, (12) where the first term is the score siqs_iq divided by 1−hi1-h_i, and the second term is of second order in the leverages, since |aqi|≤hihq a_qi ≤ h_ih_q with hq=mq⊤H−1mqh_q=m_q H^-1m_q. For the demonstration itself, aii=hia_i=h_i, the residual after deletion is ri/(1−hi)r_i/(1-h_i), and the change of its loss is ΔiiF=12∥ri∥22[(1−hi)−2−1]=Ti1−hi/2(1−hi)2, ^F_i= 12 r_i _2^2 [(1-h_i)^-2-1 ]=T_i\, 1-h_i/2(1-h_i)^2, (13) so that the exact effect and the self-influence differ by a factor that depends on the leverage alone and that tends to one as the leverage tends to zero. Since the mean leverage is below d/nd/n, this factor is close to one for most demonstrations of a tabular context, which is the regime in which the first-order score is accurate. It is not accurate in a context that has fewer demonstrations than dimensions, where the leverages approach one. Equation 12 is the exact fixed-representation deletion of Section 3. It requires no refit, since it reuses the inverse of H, the residuals and the leverages of the single fit. B.3 What the fixed-representation simplification neglects The two previous results hold for representations that do not respond to the perturbation, which is the simplification of Equation 6. To state what this simplification leaves out, suppose that the context varies along a differentiable path indexed by ϵε, so that the coefficients and the representations both respond. The chain rule then gives dℓqdϵ=rq⊤B˙⊤mq+rq⊤B^⊤m˙q, d _qdε=r_q B m_q+r_q B m_q, (14) where B˙ B and m˙q m_q denote derivatives with respect to ϵε. The second term is the response of the representation of the query, and the derivative B˙ B of the first term contains, in addition to Equation 9, the response of every row of M. Equation 6 sets these two responses to zero and keeps Equation 9 alone. The finite counterpart of this statement is the recomputed deletion ΔiqR ^R_iq of Equation 5, in which we remove demonstration i from the context, let the TFM encode the remaining demonstrations and the query again, and refit the surrogate with the same regularization. The difference between ΔiqR ^R_iq and ΔiqF ^F_iq measures the effect of Equation 6 alone, provided that the two quantities share the queries, their role and their labels. The difference between ΔiqR ^R_iq and the leave-one-out effect ΔiqTFM ^TFM_iq of Equation 1 measures in turn what the surrogate does not capture, since both quantities encode the context again and differ only in the loss that they read. Both require one forward pass of the TFM per deleted demonstration, which is the cost that TICDA avoids. Appendix C Relation to DETAIL: the role of the ridge penalty This appendix explains how the score of TICDA differs from that of DETAIL (Zhou et al., 2024). Although the two methods are designed for different domains of applications, hence their motivations and the justifications of the approximation/simplification, as explained in Section 4, they both rely on the same surrogate. Both methods fit the ridge regression of Equation 3, so they share the coefficients B B and the matrix H, and both apply Equation 2 to it. The main difference in the design is in the construction of each demonstration’s loss: for TICDA, it is the squared error ℓi(B) _i(B) of the demonstration, whereas DETAIL adds the ridge penalty to it, in full and not as a share divided among the n demonstrations: pi(B)=ℓi(B)+λ2∥B∥F2,p_i(B)= _i(B)+ λ2 B _F^2, where ∥⋅∥F · _F is the Frobenius norm, so that the second term is the penalty of J itself. We show below what this choice changes, first for the question that each score answers, then for the gradient of a demonstration, and finally for the score itself. An influence function estimates, to first order, what happens to a model when the loss of one example is removed from its training objective. What is removed therefore depends on what a method calls the loss of a demonstration. To compare the two cases, we write the objective of Equation 3 as a sum over the demonstrations J(B)=∑jℓj(B)+λ2∥B∥F2J(B)= _j _j(B)+ λ2 B _F^2 For TICDA, the loss of demonstration i is ℓi _i, and removing it from J gives J(B)−ℓi(B)=∑j≠iℓj(B)+λ2∥B∥F2,J(B)- _i(B)= _j≠ i _j(B)+ λ2 B _F^2, i.e., the other demonstrations with the same penalty. TICDA therefore estimates the effect of fitting the surrogate without demonstration i and with the same regularization, which is leave-one-out (Equation 1) applied to the surrogate instead of the TFM. For DETAIL, the loss of demonstration i is pip_i, and removing it from J gives J(B)−pi(B)=∑j≠iℓj(B),J(B)-p_i(B)= _j≠ i _j(B), i.e., the other demonstrations with no penalty at all, since J contains the penalty only once. DETAIL therefore estimates the joint effect of removing the demonstration and of removing the regularizer. The same difference appears in the gradients from which the two scores are computed. With the notation of Appendix B, the gradient of ℓi _i at B B is ai=miri⊤a_i=m_ir_i , for DETAIL the gradient at demonstration i is gi=ai+λB^,g_i=a_i+λ B, DETAIL adds the same matrix λB^λ B to the gradient of every demonstration, and to that of the query. To see what this added matrix represents, we use the fact that B B minimizes J. Since J is the sum of the losses ℓj _j and of the penalty, its gradient at B B is ∑jaj+λB _ja_j+λ B, and this gradient is zero because B B minimizes J, the matrix that DETAIL adds is therefore minus the sum of the gradients of all the demonstrations: λB^=−∑jaj.λ B=- _ja_j. Substituting this expression into gi=ai+λB^g_i=a_i+λ B cancels the term aia_i and gives gi=−∑j≠iaj.g_i=- _j≠ ia_j. (15) The gradient that DETAIL attributes to demonstration i is thus the sum of the gradients of all the other demonstrations, with the opposite sign. For example, in a context of 1,0001,000 demonstrations, the gradients that it attributes to two demonstrations i and j are two sums of 999999 terms k, k≠ik≠ i or k≠jk≠ j, respectively. 998998 components of these two sums are the same. Hence, what DETAIL adds is a term common to all demonstrations. Finally, consider the score that each method gives to a demonstration for its own loss, which we call its self-score. Up to a constant factor, the self-score of demonstration i is the quadratic form ⟨gi,H−1gi⟩F g_i,H^-1g_i _F of the gradient gig_i that the method attributes to i, where ⟨⋅,⋅⟩F ·,· _F is the Frobenius inner product of Appendix B. For TICDA, the gradient is aia_i and this quadratic form is the self-influence Ti=⟨ai,H−1ai⟩FT_i= a_i,H^-1a_i _F (Appendix B.1). Replacing aia_i by ai+λB^a_i+λ B, like in DETAIL, expands this quadratic form as the square of a sum, which produces two additional terms: ⟨gi,H−1gi⟩F=Ti+2λ⟨ai,H−1B^⟩F⏟cross term Xi+λ2⟨B^,H−1B^⟩F⏟constant C. g_i,\,H^-1g_i _F=T_i+ 2λ\, a_i,\,H^-1 B _F_cross term X_i+ λ^2 B,\,H^-1 B _F_constant C. (16) We denote the middle term by XiX_i and last term by C. C is the same for all demonstrations, so that it has no effect on their ranking. The cross term XiX_i depends on the demonstration, and it has a direct reading. As in Equation 7, let ϵε be the additional weight given to demonstration i and B^ϵ B_ε the corresponding minimizer. By Equation 9 and the symmetry of H−1H^-1, the penalty term of this fit responds to the weight as dϵλ2∥B^ϵ∥F2|ϵ=0=λ⟨B^,dB^ϵdϵ|ϵ=0⟩F=−λ⟨ai,H−1B^⟩F, . ddε\, λ2 B_ε _F^2 |_ε=0=λ B,\, . d B_εdε |_ε=0 _F=-λ\, a_i,\,H^-1 B _F, where the first equality is the derivative of a squared norm and the second substitutes Equation 9. The cross term 2λ⟨ai,H−1B^⟩F2λ a_i,H^-1 B _F is thus equal to minus twice this derivative. It is positive when the derivative is negative, which happens for a demonstration whose weight reduces the norm of the coefficients, and it is negative otherwise. Since it can take either sign, it can change the ranking given by TiT_i. The two scores thus answer different questions. C.1 Empirical check of the fixed-representation simplification The simplification of Equation 6 assumes that removing a demonstration leaves the representations of the remaining demonstrations and of the query unchanged, whereas a TFM computes each of these representations from the whole context. To see how far this assumption is from what the TFM actually does, and how this gap depends on the size of the context, we remove a demonstration, let the TFM encode the reduced context again, and compare every representation with the one computed from the full context. For a removed demonstration i, we measure the relative change in the representation of row j, either a remaining demonstration or a query: ρij=∥ϕ(xj∣∖i)−ϕ(xj∣)∥2∥ϕ(xj∣)∥2, _ij= φ(x_j _ i)-φ(x_j ) _2 φ(x_j ) _2, (17) where ∥⋅∥2 · _2 is the Euclidean norm. We measure this ratio for TabICL, with the same pre-ICL representations as in Section 3.1. We use four TabArena datasets that are large enough for contexts of 8,192 demonstrations: GiveMeSomeCredit, SDSS17, bank-marketing and HR_Analytics. For each dataset, we draw 50 queries from the test split and random contexts of 512, 1,024, 2,048, 4,096 and 8,192 demonstrations from the training split. As in Section 3.1, we corrupt 20%20\% of the labels in each context. We then pick 16 demonstrations, half of them mislabeled, and delete each of them separately: we remove that single demonstration, encode the remaining context, and put it back before the next deletion. The 16 demonstrations that we delete are in every context. We repeat this for two random draws at every size per dataset. Table 5: Relative change ρij _ij of the TabICL representations when one demonstration is removed (Equation 17), in percent of the norm of the representation, for contexts of increasing size. For each dataset, we take the median of ρij _ij over all pairs of a removed demonstration i and a row j, and we report the mean ± standard deviation of this median over the four datasets. A value of zero would mean that the representations do not change. Demonstrations in the context Remaining demonstrations Queries 512 2.90 ± 0.60 2.93 ± 0.62 1,024 1.73 ± 0.61 1.71 ± 0.61 2,048 0.89 ± 0.24 0.88 ± 0.25 4,096 0.48 ± 0.14 0.48 ± 0.15 8,192 0.24 ± 0.07 0.24 ± 0.08 Figure 5: Relative change ρij _ij of the TabICL representations when one demonstration is removed, against the number n of demonstrations in the context, on logarithmic axes. The markers show the mean over the four datasets of the per-dataset median, as in Table 5, and the band one standard deviation for the remaining demonstrations. The change of the queries coincides with that of the remaining demonstrations, and both follow the dotted line, which decreases as 1/n1/n. Table 5 and Figure 5 show that the representations do change after a deletion, but by a small amount that shrinks as the context grows: the median change roughly halves each time the number of demonstrations doubles, from about 3%3\% of the norm with 512 demonstrations to about 0.5%0.5\% with 4,096, the largest context of Section 3.1. This decrease is close to the rate 1/n1/n at which the share of a single demonstration in the context falls, and it holds in each of the four datasets, for the remaining demonstrations as well as for the queries. The change is larger when the removed demonstration is itself mislabeled, by a factor of 1.61.6 to 1.81.8 at every context size. Equation 6 is therefore an approximation rather than an exact property of the TFM, and its error on each representation becomes small in the large contexts that TICDA is designed for. Appendix D Implementation Details In the following we give additional information about implementation of methods analyzed in the body of the paper. Integrated gradients (IG) (Sundararajan et al., 2017) are computed with 32 steps to approximate gradients with Riemann sum on the straight line between instance to explain and baselines. DETAIL is implemented based on hyperparameters provided by the paper (Zhou et al., 2024), with a regularization term at 10. Appendix E Additional Experiments Below we show how the regularization term λ impacts TICDA performance both in terms of labeling error detection and context curation. (a) Context curation sensitivity to λ. (b) Labeling error detection sensitivity to λ. Figure 6: TICDA sensitivity to λ regularization.