Paper deep dive

The Standardization Trap: Certifying Joint Label Processing in Tabular Foundation Models

Duong Nguyen, Nicolas Chesneau, Milan Bhan

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 130

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Linear regression and kernel smoothing offer tractable explanations of in-context learning: in both, the features determine the weight assigned to each context label. However, whether this fixed-weight account describes pretrained tabular foundation models (TFMs) remains unclear. Testing this account using derivatives runs into a standardization trap: public TFM packages standardize the labels before the model sees them, yet ordinary derivatives also reflect behavior outside the set of standardized labels, making a model appear nonlinear even when every prediction it makes agrees with a fixed-weight map. We propose two certificates that depend only on predictions at standardized labels and can reject two distinct explanations: fixed-weight prediction and sums of independent nonlinear label transformations. Across the five public TFMs that we evaluate, our certificates show that changing one context label alters how other labels influence the prediction, a behavior we call joint processing. We further find that joint processing emerges with training and that attention scores carry most of the measured interaction. Together, these findings motivate TFM explanations that account for how context labels change the influence of individual examples.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

126,587 characters extracted from source content.

Expand or collapse full text

The Standardization Trap: Certifying Joint Label Processing in Tabular Foundation Models Duong Nguyen Nicolas Chesneau Milan Bhan Email: first.last@ekimetrics.com Abstract Linear regression and kernel smoothing offer tractable explanations of in-context learning: in both, the features determine the weight assigned to each context label. However, whether this fixed-weight account describes pretrained tabular foundation models (TFMs) remains unclear. Testing this account using derivatives runs into a standardization trap: public TFM packages standardize the labels before the model sees them, yet ordinary derivatives also reflect behavior outside the set of standardized labels, making a model appear nonlinear even when every prediction it makes agrees with a fixed-weight map. We propose two certificates that depend only on predictions at standardized labels and can reject two distinct explanations: fixed-weight prediction and sums of independent nonlinear label transformations. Across the five public TFMs that we evaluate, our certificates show that changing one context label alters how other labels influence the prediction, a behavior we call joint processing. We further find that joint processing emerges with training and that attention scores carry most of the measured interaction. Together, these findings motivate TFM explanations that account for how context labels change the influence of individual examples. 1 Introduction Influential studies have connected in-context learning to linear regression and kernel smoothing (Akyürek et al., 2023; von Oswald et al., 2023; Zhang et al., 2024; Collins et al., 2024). In fixed-weight versions of these accounts, the features determine how the examples are weighted, while the context labels supply the values being combined. Predictions change when labels change, but the coefficients do not. This gives a precise starting point for understanding adaptation without parameter updates. We ask how far this account extends to pretrained tabular foundation models (TFMs). At inference, TFMs predict the label of a query using labeled context examples while keeping their parameters fixed. How do TFMs use those context labels? Do the labels only supply the values being combined, or do they also change how the examples are weighted? Consider predicting a house price from nearby sales. One strategy assigns a fixed weight to each house using location and building attributes, then averages the observed prices. Another also examines the prices themselves: an unusually high price may be discounted when isolated but reinforced when nearby houses show the same pattern. In the second strategy, one price changes the weight of other prices, even though the houses, their attributes, and the query remain fixed. Do the labels of a pretrained TFM only supply values that fixed weights combine, or does one label also change the weight given to another? To answer this question, we measure label sensitivity: the first derivative of the prediction with respect to one context label. For a model that assigns a fixed weight to a context observation, this sensitivity is the weight itself, so it does not move when the labels change. We therefore look at second derivatives and ask whether the sensitivity to one label moves when another label changes. Furthermore, a model could transform each label separately, for example by limiting extreme values, and then add the contributions. This way, the prediction might respond nonlinearly to label changes, while no label would change the weight of another. We therefore distinguish fixed affine prediction, independent nonlinear label transformations, and interaction across examples. That distinction allows us to precisely ask what kind of adaptation a model has learned, rather than treating every nonlinear response as the same phenomenon. Measuring such label interactions with derivatives runs into a surprising trap: tests based on derivative analysis can suggest that labels interact even when every prediction that the model makes combines the labels with fixed weights. Label standardization restricts the labels entering the network to vectors with a fixed mean and scale. The raw labels never reach the network directly, yet ordinary derivatives also describe behavior away from the set of standardized labels. For example, adding to a fixed-weight map a component that is zero on this set leaves every prediction unchanged but changes the derivatives (Section 3.2). We could therefore mistake behavior that never affects the prediction for evidence of adaptation. Our contributions are as follows: 1. Certificates that distinguish three forms of label processing: We derive two certificates that account for standardization and distinguish three forms of label processing: (1) fixed-weight prediction, in which the weights do not depend on the labels, (2) sums of independent nonlinear label transformations, in which a label changes only its own influence, and (3) joint processing, in which one label changes the influence of the others. A nonzero value of the first certificate excludes (1), and a nonzero value of the second excludes both (1) and (2), which establishes (3). 2. Evidence of joint processing and its association with training: Using the proposed certificates, we show evidence of joint processing in the internal readouts of all tested models. Randomly initialized copies of the same architectures also respond nonlinearly to labels, but they show far less interaction across examples, which indicates that joint processing appears with training rather than coming from the architecture alone. 3. Identification of the derivative paths carrying the measured interaction: We show that holding the attention scores fixed during differentiation removes most of the measured local curvature while leaving the predictions unchanged, so most of the measured interaction passes through the attention scores. Our paper is organized as follows. Section 2 reviews related work on explanations of how in-context learning works. Section 3 introduces the standardization trap and derives our two certificates. Section 4 applies these certificates to five public TFMs and examines the roles of training and attention mechanisms in joint label processing. Finally, Section 5 discusses the implications and limitations of our findings. 2 Related work Fixed-weight accounts of in-context learning. Several prior works have shown that a transformer can carry out linear regression, or steps of gradient descent, on the context examples (von Oswald et al., 2023; Akyürek et al., 2023; Ahn et al., 2023; Zhang et al., 2024), and attention can be interpreted as a kernel operation (Tsai et al., 2019; Collins et al., 2024). In the simple versions of these explanations, the prediction is a weighted sum of the context labels. The features then characterize the weight of each example. The labels only supply the values to be combined (i.e., changing a label never changes a weight). We test whether pretrained TFMs process context labels the same way. With the features fixed, can one set of weights explain the predictions for every set of labels? Writing a model in kernel form does not settle the question, because the kernel itself may already depend on the labels. For example, Miftachov et al. (2026) fine-tune TabICL (Qu et al., 2025) so that its final prediction is a kernel average, but the inputs of this kernel are embeddings that the model has computed from the features and the labels together. Any adaptation to the labels has then happened before the kernel is applied. The whole computation therefore has to be examined, including how representations are built and how parameters such as a bandwidth are chosen. Learning a task from its labels. In other accounts, the labels do more than supply values. They also help decide which map is applied. Prior-data fitted networks are trained to approximate a Bayesian predictor. Such a predictor updates its belief about the task from the observed labels (Müller et al., 2022). Bayesian accounts of in-context learning make the same point: the labeled examples reveal which task is being solved, and the prediction follows from that task (Xie et al., 2022; Panwar et al., 2024). A transformer can also select among several algorithms according to how well each one predicts examples held out from its fit (Bai et al., 2023). In all three accounts, changing the labels can change the inferred task or the selected algorithm, and with it the weight given to every label. Our measurements agree with this picture. Nagler (2023) addresses a different question, why the bias of a PFN need not vanish as the context grows unless the model focuses on the examples near the query, whereas we study how a model responds when the labels change at a fixed context size. From internal computation to prediction. Another line of work studies the internal computation of TFMs. Gupta et al. (2026b) show that the hidden layers of TabPFN encode intermediate quantities of this computation, such as partial products of an arithmetic target, and Ye et al. (2025) find that TabPFN v2 (Hollmann et al., 2025) can serve as a feature extractor whose features separate the classes well. Predictions are refined gradually across layers, with so much overlap that one layer applied six times matches six distinct layers in a small model (Rezaei Balef et al., 2026). Ablation experiments find that one attention head dominates the others at its peak layer (Gupta et al., 2026a). Model families also read the context labels in different ways, from an attention-weighted vote over the labels to a class-mean readout (Biloš et al., 2026). Under noisy labels and irrelevant features, TabPFN’s attention stays on the informative features (Hu and Ghelichi, 2026). Benchmarks measure how well the models predict (Grinsztajn et al., 2022; Erickson et al., 2025; Purucker et al., 2026). For language models, some studies ask how much a prediction depends on the labels of the demonstrations (Min et al., 2022; Kossen et al., 2024). Other studies connect in-context learning to gradient descent, induction heads, and task vectors (Dai et al., 2023; Olsson et al., 2022; Hendel et al., 2023; Todd et al., 2024), or trace a prediction back to individual demonstrations (Zhou et al., 2024). These studies leave one research question open: does one label change how another label is used? We address this question in the following section. 3 Measuring how models use labels This section presents how we address the research question introduced above. The objective is to measure how a TFM uses its context labels, and in particular whether changing one label changes how other labels influence the prediction. Derivatives are the natural tool, but using them directly runs into a trap: label standardization confines the labels reaching the network to a proper subset of the label space, while derivatives in ambient coordinates also describe behavior off that subset. A model that coincides with a fixed affine map on every admissible label vector can therefore exhibit nonzero second derivatives. We develop tests that remove this ambiguity, distinguishing joint processing from fixed affine prediction and from independent nonlinear transformations of individual labels. 3.1 From a label’s value to its influence Let i∈1,…,ni∈\1,…,n\ index the context examples. Each example has a feature vector xi∈ℝdx_i ^d and a label yiy_i. The context feature matrix and corresponding label vector are X=[x1⋯xn]⊤∈ℝn×dX=[x_1·s\,x_n] ^n× d and =(y1,…,yn)⊤∈ℝny=(y_1,…,y_n) ^n, respectively. For a query with feature vector x∗x_*, write the prediction as y^=f⁡(x∗,X,) y=f(x_*,X,y). Note that the network itself does not receive the raw labels y. A TFM package first standardizes y to ~ y in a wrapper, with components y~i=(yi−y¯)/sy y_i=(y_i- y)/s_y, where y¯ y is the context-label mean and sys_y is the standardization scale. The network computes the internal prediction y^int=g⁡(~) y_int=g( y), and the output conversion returns y y in the original label units. This distinction determines what we measure: the response of the internal map to standardized labels, with one estimator and a fixed output conversion. We describe this response through the label sensitivity ai(~)=∂g⁡(~)/∂y~ia_i( y)=∂ g( y)/∂ y_i, which measures how strongly label i affects the prediction. Several existing interpretations of in-context learning describe the prediction as affine in the labels: y^int=b+∑i=1nai(x∗,X)y~i, y_int=b+ _i=1^na_i(x_*,X)\, y_i, (1) where the offset b and the weights ai(x∗,X)a_i(x_*,X) depend on the features and the query but not on the labels. For example, the kernel-smoothing interpretation of Collins et al. (2024) takes a weighted average of the labels, with nonnegative weights that sum to one. The converged single-layer linear-attention predictor analyzed by Zhang et al. (2024) allows weights of either sign. In these accounts, the sensitivities are the weights themselves, and stay constant when the context labels change. We call such a map a fixed-weight map. Other interpretations let the labels change the weights themselves, so that ai=ai(x∗,X,~)a_i=a_i(x_*,X, y). A map whose sensitivities change with the labels can still be of two kinds. It can transform one label at a time and add the results, so that a label changes only its own sensitivity, or the labels can interact, which we call joint processing. To test whether a model uses a fixed-weight map, one can perturb the labels and compare the sensitivity vectors before and after the perturbation. A fixed-weight map returns the same vector twice, so a large change would seem to show that the weights are not fixed. In the next section, we show that this test falls into a trap, and so does the second-derivative test that extends it. We then present protocols that avoid this trap and certify how a model uses its context labels. Figure 1: Labels are processed jointly, and attention carries most of the measured interaction. a: three ways a model could use context labels while the features and the query stay fixed. The blue arrow is the influence aia_i of row i on the internal prediction y^int y_int. With fixed weights, this influence never changes. If labels are handled one at a time, a label can change only its own influence. In joint processing, another label y~j y_j changes how much row i matters, which is what all five models show (§4.2). b: inside an attention step, labels can act through the attention scores and through the representations that those scores combine. A stop-gradient, applied at every attention step, lets the scores pass unchanged but stops labels from influencing them when derivatives are taken. The prediction stays the same, while most of the measured interaction disappears (§4.3). The stop-gradient shows which derivative paths carry the interaction in the computation as implemented, not that these paths are necessary. 3.2 The standardization trap Figure 2: The standardization trap and the two certificates (§3.2). a: the network only ever sees labels on the standardization manifold ℳM, but derivatives also react to label changes that would leave it. For a label perturbation v∈ℝnv ^n, the projector PTP_T keeps the part that slides along ℳM, so that a small step keeps the label mean and changes the label norm only negligibly. b: each shape shows how strongly the prediction curves along every direction u within the tangent space, drawn as the distance from the center. This curvature splits into a part that is equal in every direction, λPTλ P_T, and the remainder T. A function such as gag_a that is affine on ℳM can produce only the first part (Lemma 1), so AT=∥T∥FA_T= T _F measures what such a map cannot explain. The equal part is enlarged for visibility. c: S contains every curvature pattern that arises when each label is transformed separately, as in ga,ϕg_a,φ. PTP_ST is the closest such pattern to T, and the length of what is left over is XsepX_sep (Lemma 2). d: the same steps on a constructed Hessian. It sums terms that act only off ℳM, a separate effect for each label, and couplings between pairs of labels. Only the pair couplings reach T−PT-P_ST. A natural way to test for fixed weights is to differentiate the sensitivities once more. The Hessian measures how the sensitivity to label i changes when label j changes: Hij(~)=∂ai∂y~j=∂2g∂y~i∂y~j.H_ij( y)= ∂ a_i∂ y_j= ∂^2g∂ y_i\,∂ y_j. A fixed-weight map has H=0H=0, and it is tempting to conclude in turn that a nonzero H means that the weights are not fixed. This conclusion falls into the standardization trap. The rest of this section explains the trap on an example and how to avoid it. The trap comes from the standardization step (Section 3.1), which does more than change the label units. Because it fixes the mean and the squared norm of the labels, it confines the inputs of the network to the standardization manifold ℳ=~∈ℝn:⊤~=0,∥~∥2=r2,M=\ y ^n:1 y=0,\ y ^2=r^2\, where 1 is the all-ones vector and r is the norm that the scale sys_y imposes, with r2=nr^2=n for the population standard deviation. This manifold is a sphere inside the subspace of mean-zero vectors. Derivatives, however, also describe how g behaves when the labels leave this sphere, which never happens in use. As a demonstration, consider a fixed-weight map as in Eq. (1), with offset b and weights a∈ℝna ^n, and add a term that vanishes on ℳM: ga(~)=b+a⊤~+q⁡(∥~∥2−r2),q≠0.g_a( y)=b+a y+q\, ( y ^2-r^2 ), q≠ 0. (2) On every valid input, the added term is zero. Hence, gag_a and the fixed-weight map b+a⊤~b+a y make exactly the same predictions, and no experiment that feeds labels to the model can tell them apart. Their derivatives nevertheless differ. The sensitivities of gag_a are a+2q~a+2q y, so they change whenever the labels are permuted, and its Hessian is 2qI2qI, which can be arbitrarily large. Neither effect changes a single valid prediction. Naive tests would therefore report that gag_a adapts its weights to the labels, although it applies fixed weights wherever it is ever evaluated. Nothing prevents a trained network from containing such terms, because training never evaluates the network away from ℳM and so never constrains how it extends there. For any map g, a valid test must therefore depend only on the values that g takes on ℳM. The first step in building such a test is to keep only the label changes that stay on ℳM. A change v∈ℝnv ^n of the labels keeps their mean when its entries sum to zero, and for small steps it keeps their norm when it is perpendicular to the current label vector ~ y. The changes that satisfy both conditions are the tangent directions at ~ y: they slide along the sphere instead of leaving it (Figure 2a). The matrix PT=I−⊤n−~~⊤r2P_T=I- 11 n- y y r^2 turns any change into a tangent one. The Hessian H~=∇~2g(~)H_ y= _ y^2g( y) involves two label changes at once, so we apply PTP_T on both sides. This gives the tangent curvature PTH~PTP_TH_ yP_T, which records how the sensitivities change when the labels move along tangent directions only. This correction is not enough. For example, going back to our example with gag_a, the Hessian 2qI2qI gives the tangent curvature 2qPT2qP_T (since PTP_T is idempotent), which is not zero, so the test still fires. The reason is that ℳM is curved, whereas a tangent direction is a straight line that touches ℳM at one specific point only. A step of length δ along a unit tangent direction v therefore leaves the sphere slightly, since ∥~+δv∥2=r2+δ2 y+δ v ^2=r^2+δ^2. The gap δ2δ^2 is too small to affect the slope of a function, but second derivatives measure exactly the terms in δ2δ^2, and the added term of gag_a turns this gap into the value qδ2qδ^2. The gap δ2δ^2 does not depend on the direction v of the step. The tangent curvature 2qPT2qP_T that it produces for gag_a therefore has the same value 2q2q in every tangent direction. We call a curvature that is equal in all tangent directions isotropic. Lemma 1 shows that this form is not special to gag_a: every function that agrees with a fixed-weight map on ℳM has an isotropic tangent curvature, whatever it does away from ℳM. Lemma 1 (Affine predictions on the manifold). Let g be any smooth function on a neighborhood of ℳM that agrees on ℳM with an affine function of ~ y. Then at every ~∈ℳ y , PTH~PT=λ⁡(~)PTP_TH_ yP_T=λ( y)\,P_T for some scalar λ⁡(~)λ( y). We provide the proof of Lemma 1 in Appendix A. Every function covered by the lemma is built like gag_a: a fixed-weight map plus terms that vanish on ℳM. By Lemma 1, such a map can leave only isotropic curvature. We therefore define the operator T as the tangent curvature minus its isotropic part, and we measure its size ATA_T: T=PTH~PT−tr⁡(PTH~PT)n−2PT,AT=∥T∥F,T=P_TH_ yP_T- tr(P_TH_ yP_T)n-2P_T, A_T= T _F, (3) where ∥⋅∥F · _F is the Frobenius norm. The trace of the tangent curvature is the sum of the curvatures along n−2n-2 perpendicular tangent directions, so dividing it by n−2n-2 gives the average curvature over the tangent directions (Figure 2b), and T keeps only the departure from this average. For gag_a, whose Hessian is H~=2qIH_ y=2qI, both quantities in Eq. (3) can be computed by hand: PTH~PT P_TH_ yP_T =PT(2qI)PT=2qPTPT=2qPT, =P_T(2qI)P_T=2q\,P_TP_T=2q\,P_T, tr⁡(PTH~PT) (P_TH_ yP_T) =2qtr⁡(PT)=2q(n−2). =2qtr(P_T)=2q\,(n-2). The first line uses PTPT=PTP_TP_T=P_T, since projecting twice changes nothing, and the second line uses tr⁡(PT)=n−2tr(P_T)=n-2, since PTP_T removes two directions from the n-dimensional identity. Substituting both lines into Eq. (3) gives T=2qPT−2q(n−2)n−2PT=0T=2q\,P_T- 2q\,(n-2)n-2\,P_T=0, so the corrected test stays silent on gag_a, as it should. The value of q plays no role in this cancellation, so the same argument gives T=0T=0 for every isotropic curvature λPTλ P_T, regardless of the value of λ. More generally, T depends only on the predictions on ℳM, regardless of how g extends away from the manifold. A nonzero ATA_T therefore excludes fixed-weight prediction on valid labels. A nonzero ATA_T shows that g is nonlinear on ℳM, but nonlinearity alone is not enough to establish joint processing. A model can transform one label at a time and add the results, as in ga,ϕ(~)=ga(~)+∑k=1nϕk(y~k),g_a,φ( y)=g_a( y)+ _k=1^n _k( y_k), (4) where each smooth function ϕk _k acts on a single label, for example by damping extreme values. For most choices of ϕk _k, this map is not affine on ℳM, and it has AT>0A_T>0. Yet no label changes the influence of another one, since the added term contributes only the diagonal matrix D=diag⁡(ϕk′(y~k))D=diag ( _k ( y_k) ) to the Hessian. We cannot detect this case by looking for zero off-diagonal entries, because the projection mixes the coordinates: on ℳM no label can move alone, since the other labels must compensate to keep the mean and the norm fixed, and PTDPTP_TDP_T is no longer diagonal. We test instead whether the corrected curvature T could have come from some diagonal matrix. Let =PTDPT−tr⁡(PTDPT)n−2PT:DdiagonalS=\P_TDP_T- tr(P_TDP_T)n-2P_T:D\ diagonal\ be the linear space of corrected curvatures that one-label-at-a-time maps, such as ga,ϕg_a,φ, can produce, and let P_S denote the orthogonal projection onto it. We measure what remains outside this space: Xsep=∥T−PT∥F.X_sep= T-P_ST _F. (5) For ga,ϕg_a,φ we have T∈T and hence Xsep=0X_sep=0, whatever the functions ϕk _k and the coefficient q. Figure 2 illustrates both projections, and Lemma 2 states the general result. Lemma 2 (Certificate of joint label processing). Let g be smooth on a neighborhood of ℳM and agree on ℳM with an affine-plus-separable map b+a⊤~+∑kϕk(y~k)b+a y+ _k _k( y_k). Then T∈T and Xsep=0X_sep=0 at every point of ℳM. The statistic XsepX_sep is extension-invariant: any two smooth maps that agree on ℳM give the same value there. A nonzero XsepX_sep therefore certifies that one label changes the influence of another. Both tests are one-sided: a zero result alone does not prove membership in the corresponding class. Table 1 follows the two examples and a joint-processing map through each correction. We provide in Appendix A the proofs, the computation of P_S, and numerical checks of this pattern. Table 1: From the Hessian to the two certificates. Each row applies one more correction to the Hessian H of the map. The last three columns give the result for gag_a, which is a fixed-weight map on ℳM (Eq. 2), for the one-label-at-a-time map ga,ϕg_a,φ (Eq. 4, with D=diag⁡(ϕk′(y~k))D=diag( _k ( y_k))), and for a map in which labels interact. Only ATA_T vanishes for every fixed-weight map, and only XsepX_sep also vanishes for every one-label-at-a-time map. Quantity What it keeps gag_a ga,ϕg_a,φ Joint map Hessian H every label change 2qI2qI 2qI+D2qI+D ≠0≠ 0 Tangent curvature PTHPTP_THP_T changes that stay on ℳM 2qPT2qP_T 2qPT+PTDPT2qP_T+P_TDP_T ≠0≠ 0 T, of size ATA_T minus the isotropic part 00 in S, ≠0≠ 0 ≠0≠ 0 T−PT-P_ST, of size XsepX_sep minus the one-label-at-a-time part 00 00 ≠0≠ 0 The absolute values of ATA_T and XsepX_sep themselves are hard to interpret, because they depend on the scale of the prediction and of its sensitivities, which differs between models and datasets. We therefore report two shares: A¯T=AT2∥PTH~PT∥F2,X¯sep=Xsep2AT2. A_T= A_T^2 P_TH_ yP_T _F^2, X_sep= X_sep^2A_T^2. (6) Both lie between 00 and 11, because each correction removes a part that is orthogonal to the part it keeps, and we set a share to 00 when its denominator vanishes. X¯sep X_sep is the share of the corrected curvature that no one-label-at-a-time map can produce, and like XsepX_sep it depends only on the predictions on ℳM. A¯T A_T is different: its denominator still contains the isotropic part of the tangent curvature, which depends on how g behaves away from ℳM. Apart from vanishing exactly when ATA_T does, A¯T A_T serves another purpose: 1−A¯T1- A_T is the fraction of the computed tangent curvature that the isotropic correction subtracts, so it shows how much this correction changes the measurement in the tested networks. A fixed-weight map has A¯T=0 A_T=0, and a one-label-at-a-time map has X¯sep=0 X_sep=0. 4 Experiments and results Our experiments answer two questions about pretrained TFMs: (1) do the models process their context labels jointly, which is the question that the two certificates were built to settle (Section 4.2), and (2) where does the interaction come from, both whether it is already present before training and which paths inside the network carry it (Section 4.3). Section 4.1 first presents the setup that all experiments share. 4.1 Experimental setup Models. We evaluate five public TFMs: TabICL-v2 (Qu et al., 2026), TabPFN-v3 (Grinsztajn et al., 2026), TabPFN-v2.6 (Grinsztajn et al., 2025), TabFM 1.0 (Kong and Das, 2026), and TabDPT v1.2 (Ma et al., 2025; Hosseinzadeh et al., 2026). They differ in architecture, attention schedule, readout, and pretraining data (Appendix B.1). The first four are pretrained on synthetic tasks and have related designs: TabICL-v2, TabPFN-v3 and TabFM build a representation of each row with attention across features and rows and then predict in a final in-context stage, whereas TabPFN-v2.6 alternates attention across features and rows in every layer. TabDPT is pretrained on real tables, which makes it the most independent of the five. Tasks and Protocol. We compute the two certificates on four regression datasets with 256256 context rows: one synthetic dataset and the OpenML datasets kin8nm, pol, and superconduct. The synthetic dataset is a smooth regression task on 1616 continuous features that we generate ourselves (Appendix B.2), so that it serves as a control that real tables cannot provide: we can redraw its label noise from the true model, no model can have seen it during pretraining, and it has no outliers or discrete features that could explain what we measure. We call one model on one dataset a setting, which gives 2020 settings. In each setting, we repeat the measurement on three randomly sampled contexts, with four queries each. Appendix Table 5 lists the datasets, context sizes, and repetitions of every experiment in the paper. Implementation Details. Since we run our analysis based on TFM responses, we take derivatives with respect to the standardized labels and keep the conversion back to the original label units fixed (Section 3.1). The differentiated prediction comes from a single estimator inside each package, whereas the packages’ default predictions combine several estimators, which we test separately (Section 4.2). Second derivatives need an attention implementation that can be differentiated twice, which the fused kernels used in deployment are not, so we switch to standard (eager) attention for these computations (Appendix B.3). 4.2 To what extent do TFMs process labels jointly? We first determine which of the three descriptions of Section 3.1 fits each TFM: fixed weights, one label at a time, or joint processing (Figure 1a). To this end, we compute the exact Hessian of the internal prediction in each of the 2020 settings and apply the two certificates, so that each description is tested by a statistic that is zero whenever the description holds. Table 2 reports the results. Table 2: All five models process labels jointly, and most of the interaction passes through attention. The shares (Eq. (6)) and the relative fall of ATA_T and XsepX_sep when derivatives cannot pass through the attention scores (Section 4.3), in percent, each averaged over the four datasets. A fixed-weight map would give A¯T=0 A_T=0, and a one-label-at-a-time map would give X¯sep=0 X_sep=0. Appendix Table 8 gives every setting. Share (%) Attention drop (%) Model A¯T A_T X¯sep X_sep ATA_T XsepX_sep TabICL-v2 98.798.7 63.163.1 81.681.6 82.082.0 TabPFN-v3 98.598.5 70.870.8 85.985.9 90.090.0 TabPFN-v2.6 97.797.7 73.473.4 90.990.9 95.395.3 TabFM 99.299.2 67.567.5 78.278.2 82.182.1 TabDPT 98.398.3 40.440.4 68.068.0 75.375.3 All models 98.598.5 63.063.0 80.980.9 85.085.0 Fixed weights do not explain the predictions of any model, since A¯T A_T is nonzero in every setting (Appendix Table 8). Its average of 98.5%98.5\% in Table 2 also shows that the isotropic correction subtracts only 1.5%1.5\% of the computed tangent curvature, so for these models the correction changes the measurement little, although the test would not be valid without it. Transforming one label at a time does not explain the predictions either. Such a map would give X¯sep=0 X_sep=0, whereas X¯sep X_sep averages 63%63\% over the 2020 settings and lies between 23%23\% and 92%92\% in every one of them (Appendix Table 8). The package default inference procedures of the tested models combine several estimators. We tested those settings and found that the combined predictions are not affine in the labels either (Appendix C.2). 4.3 Where do the interactions come from? The certificates analyzed above establish that labels interact inside the models, but they do not say where this interaction comes from. We first test whether the architecture alone produces the interaction or whether it appears with training, and we then identify which paths inside the network carry it. Training changes the interaction among labels. We want to know whether the interaction comes with the architecture or appears with training. We therefore compare each trained model with 1212 randomly initialized copies of the same architecture on two of the four datasets. A randomly initialized network is already a nonlinear function of its labels, so it departs from fixed weights before training: ATA_T is nonzero for both the trained models and the random copies and therefore cannot tell them apart. The normalized certificate X¯sep X_sep does separate trained from untrained networks. On the two datasets of this comparison, every query of a trained model gives at least 29%29\%, whereas the random copies stay at most 3.4%3.4\% (Appendix C.3). Attention carries most of the measured interaction. Attention combines representations with weights that it computes from attention scores, and labels can change both the representations being combined and the scores that weight them. To separate these contributions, we block derivatives through the attention scores while keeping their values unchanged in the forward computation (Figure 1b), so that the model makes exactly the same prediction while its derivatives retain only the paths that do not pass through these scores. Blocking these derivative paths removes most of the measured interaction (columns “Attention drop” in Table 2). The cross-example residual XsepX_sep falls by 85%85\% on average, from 75%75\% for TabDPT to 95%95\% for TabPFN-v2.6, and the corrected curvature norm ATA_T falls by 81%81\% on average. These percentages describe reductions in norms, whereas the comparison with random copies uses shares of squared curvature, and they neither measure accuracy loss nor establish that these paths are necessary for good predictions. Appendix C.4 compares the curvature operators directly and describes a forward intervention. These results add to the ongoing debate on whether attention weights explain the predictions of a model (Wiegreffe and Pinter, 2019; Jain and Wallace, 2019; Bibal et al., 2022; Bhan et al., 2023). 5 Discussion and conclusion In the evaluated models, context labels can change how other labels affect the prediction. The packages standardize the labels before the networks see them, and ordinary derivatives also describe how the network would behave on labels that are not standardized, so they can report nonlinearity even for models that apply fixed weights to every valid input. Our two certificates measure behaviors on standardized labels only. The first certificate ATA_T is zero for every fixed-weight map, and the second certificate XsepX_sep is zero for every map that transforms one label at a time. Both are nonzero in all five models, so no model is explained by either fixed weights or transforming its labels one at a time. Of the curvature that departs from fixed weights, 23%23\% to 92%92\% comes from interaction that no one-label-at-a-time map can produce. This interaction is much weaker in randomly initialized copies of the same architectures, which indicates that it comes with training rather than from the architecture alone. Blocking derivatives through the attention scores removes most of it while leaving the predictions unchanged, so in the networks as implemented most of the interaction passes through these scores. Our findings guide explanations of how TFMs use their context labels. They exclude the fixed-weight account, in which the features alone set the weight of each label, because in every evaluated model a change in one label also changes the weights of the others. Yet they do not identify the computation behind this interaction, since several computations make labels interact: (1) the model may use all labels to infer which task generated the data, so that changing any label changes the inferred task and with it the weight of every label, (2) a local regression may reweight the neighbors of the query by their labels, and (3) the representations of the examples may themselves depend on the labels. Our certificates detect the interaction but cannot tell these computations apart. The fixed-weight account also remains a useful approximation even though it fails as an exact description: a fixed-weight map fitted to a model’s predictions explains a median of 72% of the variation in these predictions when the labels are redrawn from the same task (Appendix C.6). An explanation should therefore state for which labels it holds and whether it reproduces only the predictions or also how one label changes the influence of the others. The remaining question is whether these label interactions improve prediction, since our derivative tests establish their presence but not their necessity. Blocking the attention scores does not answer it either, because the predictions stay the same. Resolving it requires an intervention that removes the interaction from the forward computation and a measurement of the resulting change in predictive accuracy. References Ahn et al. (2023) K. Ahn, X. Cheng, H. Daneshmand, and S. Sra Transformers learn to implement preconditioned gradient descent for in-context learning. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2306.00297 Cited by: §2. Akyürek et al. (2023) E. Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou What learning algorithm is in-context learning? Investigations with linear models. In International Conference on Learning Representations (ICLR), External Links: 2211.15661 Cited by: §1, §2. Bai et al. (2023) Y. Bai, F. Chen, H. Wang, C. Xiong, and S. Mei Transformers as statisticians: provable in-context learning with in-context algorithm selection. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2306.04637 Cited by: §2. Bhan et al. (2023) M. Bhan, N. Achache, V. Legrand, A. Blangero, and N. Chesneau Evaluating self-attention interpretability through human-grounded experimental protocol. In Explainable Artificial Intelligence: First World Conference, xAI 2023, Communications in Computer and Information Science, Vol. 1903, p. 26–46. External Links: Document Cited by: §4.3. Bibal et al. (2022) A. Bibal, R. Cardon, D. Alfter, R. Wilkens, X. Wang, T. François, and P. Watrin Is attention explanation? An introduction to the debate. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3889–3900. Cited by: §4.3. Biloš et al. (2026) M. Biloš, J. T. Wilson, A. Schneider, and Y. Nevmyvaka A mechanistic study of tabular foundation models. arXiv preprint arXiv:2605.21288. External Links: 2605.21288 Cited by: §2. Collins et al. (2024) L. Collins, A. Parulekar, A. Mokhtari, S. Sanghavi, and S. Shakkottai In-context learning with transformers: softmax attention adapts to function Lipschitzness. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 37, p. 92638–92696. External Links: 2402.11639 Cited by: §1, §2, §3.1. Dai et al. (2023) D. Dai, Y. Sun, L. Dong, Y. Hao, S. Ma, Z. Sui, and F. Wei Why can GPT learn in-context? Language models implicitly perform gradient descent as meta-optimizers. In Findings of the Association for Computational Linguistics: ACL 2023, p. 4005–4019. External Links: 2212.10559 Cited by: §2. Erickson et al. (2025) N. Erickson, L. Purucker, A. Tschalzev, D. Holzmüller, P. Mutalik Desai, D. Salinas, and F. Hutter TabArena: a living benchmark for machine learning on tabular data. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, External Links: 2506.16791, Link Cited by: §2. Google Research (2026) Google Research TabFM source code. Note: GitHub repository External Links: Link Cited by: §B.1. Grinsztajn et al. (2025) L. Grinsztajn, K. Flöge, O. Key, F. Birkel, P. Jund, B. Roof, B. Jäger, D. Safaric, S. Alessi, A. Hayler, M. Manium, R. Yu, F. Jablonski, S. B. Hoo, A. Garg, J. Robertson, M. Bühler, V. Moroshan, L. Purucker, C. Cornu, L. C. Wehrhahn, A. Bonetto, B. Schölkopf, S. Gambhir, N. Hollmann, and F. Hutter TabPFN-2.5: advancing the state of the art in tabular foundation models. arXiv preprint arXiv:2511.08667. External Links: 2511.08667 Cited by: Table 4, §4.1. Grinsztajn et al. (2026) L. Grinsztajn, K. Flöge, O. Key, F. Birkel, et al. TabPFN-3: technical report. arXiv preprint arXiv:2605.13986. External Links: 2605.13986 Cited by: Table 4, §4.1. Grinsztajn et al. (2022) L. Grinsztajn, E. Oyallon, and G. Varoquaux Why do tree-based models still outperform deep learning on typical tabular data?. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, External Links: 2207.08815 Cited by: §2. Gupta et al. (2026a) A. Gupta, D. Kumar, M. Mandal, and S. Deshpande Where computation lives inside TabPFN: causal localisation of attention head function. arXiv preprint arXiv:2606.12917. External Links: 2606.12917 Cited by: §2. Gupta et al. (2026b) A. Gupta, A. Sethi, and D. Kumar TabPFN through the looking glass: an interpretability study of TabPFN and its internal representations. arXiv preprint arXiv:2601.08181. External Links: 2601.08181 Cited by: §2. Hendel et al. (2023) R. Hendel, M. Geva, and A. Globerson In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 9318–9333. External Links: 2310.15916 Cited by: §2. Hollmann et al. (2025) N. Hollmann, S. Müller, L. Purucker, A. Krishnakumar, M. Körfer, S. B. Hoo, R. T. Schirrmeister, and F. Hutter Accurate predictions on small data with a tabular foundation model. Nature 637 (8045), p. 319–326. External Links: Document Cited by: §2. Hosseinzadeh et al. (2026) R. Hosseinzadeh, A. Labach, Z. Xue, S. Han, V. Thomas, and A. L. Caterini TabDPT-Turbo: efficient in-context learning for tabular prediction. arXiv preprint arXiv:2608.01400. External Links: 2608.01400 Cited by: §B.1, §B.2, Table 4, §4.1. Hu and Ghelichi (2026) J. Hu and M. Ghelichi Noise immunity in in-context tabular learning: an empirical robustness analysis of TabPFN’s attention mechanisms. Note: arXiv:2604.04868 Cited by: §2. Jain and Wallace (2019) S. Jain and B. C. Wallace Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 3543–3556. Cited by: §4.3. Kong and Das (2026) W. Kong and A. Das Introducing TabFM: a zero-shot foundation model for tabular data. Note: Google Research blog External Links: Link Cited by: Table 4, §4.1. Kossen et al. (2024) J. Kossen, Y. Gal, and T. Rainforth In-context learning learns label relationships but is not conventional learning. In International Conference on Learning Representations (ICLR), External Links: 2307.12375 Cited by: §2. Lee (2013) J. M. Lee Introduction to smooth manifolds. 2nd edition, Graduate Texts in Mathematics, Vol. 218, Springer. Cited by: §A.1. Ma et al. (2025) J. Ma, V. Thomas, R. Hosseinzadeh, A. Labach, H. Kamkari, J. C. Cresswell, K. Golestan, G. Yu, A. L. Caterini, and M. Volkovs TabDPT: scaling tabular foundation models on real data. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2410.18164 Cited by: §B.2, §4.1. Miftachov et al. (2026) R. Miftachov, B. Charron, and S. Valentin Interpretable tabular foundation models via in-context kernel regression. arXiv preprint arXiv:2602.02162. External Links: 2602.02162 Cited by: §C.6, §2. Min et al. (2022) S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer Rethinking the role of demonstrations: what makes in-context learning work?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 11048–11064. External Links: 2202.12837 Cited by: §2. Müller et al. (2022) S. Müller, N. Hollmann, S. Pineda Arango, J. Grabocka, and F. Hutter Transformers can do Bayesian inference. In International Conference on Learning Representations (ICLR), External Links: 2112.10510 Cited by: §2. Nagler (2023) T. Nagler Statistical foundations of prior-data fitted networks. In Proceedings of the 40th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 202, p. 25660–25676. External Links: 2305.11097 Cited by: §2. Olsson et al. (2022) C. Olsson, N. Elhage, N. Nanda, N. Joseph, et al. In-context learning and induction heads. Transformer Circuits Thread. External Links: 2209.11895 Cited by: §2. Panwar et al. (2024) M. Panwar, K. Ahuja, and N. Goyal In-context learning through the Bayesian prism. In International Conference on Learning Representations (ICLR), External Links: 2306.04891 Cited by: §2. Purucker et al. (2026) L. Purucker, A. Tschalzev, N. Erickson, G. Blayer, D. Holzmüller, A. Arazi, A. Pfefferle, M. Tajjar, G. Varoquaux, and F. Hutter Beyond IID: how general are tabular foundation models, really?. arXiv preprint arXiv:2606.30410. External Links: 2606.30410, Link Cited by: §2. Qu et al. (2025) J. Qu, D. Holzmüller, G. Varoquaux, and M. Le Morvan TabICL: a tabular foundation model for in-context learning on large data. In Proceedings of the 42nd International Conference on Machine Learning (ICML), External Links: 2502.05564 Cited by: §2. Qu et al. (2026) J. Qu, D. Holzmüller, G. Varoquaux, and M. Le Morvan TabICLv2: a better, faster, scalable, and open tabular foundation model. In Proceedings of the 43rd International Conference on Machine Learning (ICML), External Links: 2602.11139 Cited by: Table 4, §4.1. Rezaei Balef et al. (2026) A. Rezaei Balef, M. Koshil, and K. Eggensperger Is one layer enough? Understanding inference dynamics in tabular foundation models. In Proceedings of the 43rd International Conference on Machine Learning (ICML), External Links: 2605.06510 Cited by: §2. Todd et al. (2024) E. Todd, M. L. Li, A. S. Sharma, A. Mueller, B. C. Wallace, and D. Bau Function vectors in large language models. In International Conference on Learning Representations (ICLR), External Links: 2310.15213 Cited by: §2. Tsai et al. (2019) Y. H. Tsai, S. Bai, M. Yamada, L. Morency, and R. Salakhutdinov Transformer dissection: an unified understanding for transformer’s attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 4344–4353. External Links: 1908.11775 Cited by: §2. von Oswald et al. (2023) J. von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov Transformers learn in-context by gradient descent. In Proceedings of the 40th International Conference on Machine Learning (ICML), External Links: 2212.07677 Cited by: §1, §2. Wiegreffe and Pinter (2019) S. Wiegreffe and Y. Pinter Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 11–20. Cited by: §4.3. Xie et al. (2022) S. M. Xie, A. Raghunathan, P. Liang, and T. Ma An explanation of in-context learning as implicit Bayesian inference. In International Conference on Learning Representations (ICLR), External Links: 2111.02080 Cited by: §2. Ye et al. (2025) H. Ye, S. Liu, and W. Chao A closer look at TabPFN v2: understanding its strengths and extending its capabilities. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2502.17361 Cited by: §2. Zhang et al. (2024) R. Zhang, S. Frei, and P. L. Bartlett Trained transformers learn linear models in-context. Journal of Machine Learning Research 25 (49), p. 1–55. External Links: Link Cited by: §1, §2, §3.1. Zhou et al. (2024) Z. Zhou, X. Lin, X. Xu, A. Prakash, D. Rus, and B. K. H. Low DETAIL: task demonstration attribution for interpretable in-context learning. In Advances in Neural Information Processing Systems (NeurIPS), External Links: 2405.14899 Cited by: §2. Appendix A Proofs and computation of the certificates This appendix proves the two lemmas that Section 3.2 states: Lemma 1 for the corrected curvature ATA_T, and Lemma 2 for the residual XsepX_sep. It then gives the linear-algebra computation that turns the definition of XsepX_sep into a number, extends the same manifold correction to the first-derivative test of Section 3.1, and calibrates these statistics against maps whose class is known in advance. A.1 Proof of Lemma 1 The lemma must show that a map which predicts exactly like a fixed-weight map on the standardization manifold ℳM can still differ from it in its second derivatives, but only in a restricted way: the tangent curvature it can produce is the same in every tangent direction. The idea is that such a map differs from the fixed-weight map only through terms built out of the two functions whose common zero set is ℳM, and each such term adds curvature only along the two directions normal to ℳM, which the tangent projector PTP_T removes, except for a single scalar rescaling of PTP_T itself that survives the removal. Write u⁡(~)=⊤~u( y)=1 y for the sum of the standardized labels and w⁡(~)=∥~∥2−r2w( y)= y ^2-r^2 for the deviation of their squared norm from r2r^2, so that ℳ=u=0,w=0M=\u=0,\,w=0\ is exactly their common zero set (Section 3.2). Since 1 and ~ y are linearly independent whenever r>0r>0, the gradients ∇u=∇ u=1 and ∇w=2~∇ w=2 y are independent on ℳM, so ℳM is a regular level set of the pair (u,w)(u,w). If g agrees on ℳM with the affine map b+a~⊤~b+ a y, Hadamard’s lemma applied in local coordinates adapted to (u,w)(u,w) writes the difference between g and this affine map as a combination of u and w with smooth coefficient functions p and q defined near each point of ℳM (Lee, 2013, Ch. 5 and Thm. C.15): g=b+a~⊤~+p⁡(~)u(~)+q⁡(~)w(~).g=b+ a y+p( y)\,u( y)+q( y)\,w( y). Because u and w vanish exactly on ℳM, this equation only restates that g and the affine map coincide there, while attaching a smooth multiplier to each of the two ways of moving off ℳM. At a point of ℳM, u=w=0u=w=0 while their gradients need not vanish, so differentiating gives ∇g ∇ g =a~+p 1+2q~, = a+p\,1+2q\, y, H H =2qI+∇p⊤+∇p 1⊤+2~∇q⊤+2∇q~⊤. =2qI+1∇ p +∇ p\,1 +2 y∇ q +2∇ q\, y . The gradient of g on ℳM is therefore the fixed weight vector a~ a plus a term along each of the two normal directions, and the Hessian keeps the isotropic term 2qI2qI and picks up four more terms, each proportional to 1 or to ~ y. The normal space of ℳM at ~ y is spanned by 1 and ~ y, which are orthogonal because the standardized labels have zero mean, ⊤~=01 y=0. The tangent projector PT=I−⊤/n−~~⊤/r2P_T=I-11 /n- y y /r^2 of Section 3.2 therefore has rank n−2n-2 and annihilates both normal directions, PT=PT~=0P_T1=P_T y=0. Projecting the Hessian above on both sides removes all four extra terms, since each carries a factor of 1 or ~ y, and leaves PTHPT=2qPT,P_THP_T=2q\,P_T, which is the isotropic form the lemma claims, with λ⁡(~)=2q(~)λ( y)=2q( y). This proves Lemma 1. The intrinsic Riemannian Hessian of g restricted to ℳM is the curvature computed from the values that g takes on ℳM alone, and not from how g extends off the manifold. The gradient identity above shows that it differs from PTHPTP_THP_T only by the isotropic correction −(~⊤∇g/r2)PT-( y ∇ g/r^2)P_T. Since this correction is itself proportional to PTP_T, it is removed along with every other isotropic term once the trace is subtracted, so the traceless part of PTHPTP_THP_T equals the traceless part of the intrinsic Hessian. Writing traceless⁡(B)=B−tr⁡(B)PT/(n−2)traceless(B)=B-tr(B)P_T/(n-2) for a tangent-projected operator B=PTBPTB=P_TBP_T, this traceless part is exactly T=traceless⁡(PTHPT)T=traceless(P_THP_T) of Eq. (3). Hence T depends only on the predictions that g makes on ℳM and not on how g is defined away from it, while the isotropic part λ⁡(~)λ( y) cannot be identified from on-manifold behavior alone, since any value of λ is compatible with the same restriction of g to ℳM. A.2 Proof of Lemma 2 The lemma must show that the same argument extends to the larger class of maps that transform each label separately before combining the results, so that the corrected curvature such a map can produce always lies in the space =traceless⁡(PTDPT):D diagonalS=\traceless(P_TDP_T):D diagonal\ of Section 3.2, and the residual XsepX_sep of Eq. (5) is zero. Let f=b+a⊤~+∑kϕk(y~k)f=b+a y+ _k _k( y_k) agree with g on ℳM, where each ϕk _k acts on a single standardized label as in ga,ϕg_a,φ (Eq. (4)). The difference g−fg-f vanishes identically on ℳM, so applying the decomposition of the previous proof to g−fg-f shows that its own tangent curvature is isotropic. The map f itself produces the tangent curvature PTDPTP_TDP_T, where D=diag⁡(ϕk′(y~k))D=diag( _k ( y_k)) collects the curvature of each ϕk _k and the linear term a⊤~a y contributes none. Adding this curvature back gives PTHgPT=PTDPT+2qPTP_TH_gP_T=P_TDP_T+2q\,P_T for the tangent curvature of g itself. Removing the trace cancels the isotropic term 2qPT2qP_T and leaves T=traceless⁡(PTDPT)T=traceless(P_TDP_T), which is a member of S by definition, so T∈T and hence Xsep=∥T−PT∥F=0X_sep= T-P_ST _F=0. As in the proof of Lemma 1, the traceless projected Hessian of any difference that vanishes on ℳM is itself zero, so T, and with it XsepX_sep, does not depend on which off-manifold extension g happens to have. Both certificates are one-sided: a value of zero is consistent with the corresponding class of maps, fixed-weight for ATA_T and one-label-at-a-time for XsepX_sep, but it does not by itself prove that a model belongs to that class, since some other map outside the class could in principle also give a zero value on the labels tested. A.3 Computing the projection onto S Evaluating XsepX_sep requires the orthogonal projection PTP_ST of the measured curvature T onto S, the space of corrected curvatures that a one-label-at-a-time map can produce. Because S is spanned by one operator per label, this projection reduces to a small linear solve rather than a search over label maps. For each label k, let EkkE_k be the matrix with a single 11 in position (k,k)(k,k) and zeros elsewhere, so that a diagonal matrix D is a combination of the EkkE_k, and set Bk=traceless⁡(PTEkkPT)B_k=traceless(P_TE_kP_T) for the corrected curvature that this single-label bump produces. The operators B1,…,BnB_1,…,B_n therefore span S. Write β=diag⁡(T)β=diag(T) for the vector of diagonal entries of the measured curvature. Because T is already tangent-projected and traceless, its inner product with each basis operator reduces to a single diagonal entry, ⟨T,Bk⟩=Tkk T,B_k =T_k, and the projection coefficients solve the normal equations of this basis, Gc=βGc=β, where the Gram matrix of inner products between basis operators is Gjk=(PT)jk2−(PT)jj(PT)kkn−2.G_jk=(P_T)_jk^2- (P_T)_j(P_T)_kn-2. The orthogonal projection is then PT=∑kckBkP_ST= _kc_kB_k with c=G+βc=G^+β, using the Moore–Penrose pseudoinverse because the basis is not independent, and Xsep=∥T−∑kckBk∥FX_sep= T- _kc_kB_k _F. This solve stays well posed despite the rank deficiency of G. If v lies in the null space of G, the corresponding combination of basis operators vanishes, ∑kvkBk=0 _kv_kB_k=0, so its inner product with T is also zero, v⊤β=0v β=0, and the system Gc=βGc=β therefore remains consistent on the subspace where G is singular. One such null direction is known in closed form, G=0G1=0, because summing the EkkE_k over every label gives the identity and traceless⁡(PTIPT)=traceless⁡(PT)=0traceless(P_TIP_T)=traceless(P_T)=0. The pseudoinverse handles this direction by returning the minimum-norm solution among those that solve the system. An independent implementation, which projects onto S using an orthonormal basis rather than the BkB_k above, gives the same residual to a relative error of at most 2×10−162× 10^-16 on the coupled control below and on model Hessians of all five models. On the two null controls, both implementations return values at the floating-point floor of about 3×10−143× 10^-14. A nonzero XsepX_sep is possible only when S is a proper subspace of the space in which T lives. S is the image of the diagonal matrices under D↦traceless⁡(PTDPT)D (P_TDP_T), so its dimension is at most n−1n-1 because this map has a nontrivial null space. The traceless symmetric operators on the tangent space, where T lives, form a space of dimension (n−2)(n−1)/2−1(n-2)(n-1)/2-1, which exceeds n−1n-1 once n≥5n≥ 5. For n≥5n≥ 5, S is therefore always a proper subspace, a condition met by a wide margin at the context size n=256n=256 that the paper uses for the certificates. For smaller contexts, S fills the whole space at most label vectors but not at all of them: at n=4n=4 and ~∝(1,1,−1,−1) y (1,1,-1,-1), S has dimension one inside a space of dimension two. A.4 The corrected perturbation test Section 3.1 describes a first test for fixed-weight prediction: perturb the context labels and compare the label sensitivities ai=∂g⁡(~)/∂y~ia_i=∂ g( y)/∂ y_i before and after. Section 3.2 shows that this test falls into the standardization trap, as does the second-derivative test built from the Hessian: a map can change its sensitivities under a permutation of the labels while still making exactly the same predictions on every valid input. This section defines the naive statistic precisely and repairs it with the same manifold correction used for ATA_T and XsepX_sep. Writing a(~)=∇~g(~)a( y)= _ yg( y) for the vector collecting these sensitivities and PπP_π for the matrix that permutes the context labels according to a permutation π, the naive permutation statistic averages the relative change of this vector over queries x∗x_* and permutations π: Δperm=x∗,π[∥a⁡(Pπ~)−a⁡(~)∥2∥a⁡(~)∥2]. _perm=E_x_*,π\! [ a(P_π y)-a( y) _2 a( y) _2 ]. (7) A fixed-weight map keeps a unchanged under any permutation of the labels, so Δperm=0 _perm=0 for such a map exactly. But Δperm _perm can be nonzero for a map that is nonetheless a fixed-weight map on every valid input. For gag_a (Eq. (2)), whose sensitivities are a+2q~a+2q y (Section 3.2), a permutation changes them by a⁡(Pπ~)−a⁡(~)=2q(Pπ~−~),a(P_π y)-a( y)=2q\,(P_π y- y), a vector that lies entirely in the span of ~ y and Pπ~P_π y, and Δperm _perm reports this change even though gag_a makes the same predictions as the fixed-weight map b+a⊤~b+a y everywhere it is ever evaluated. The repair projects this span out before comparing sensitivities, the same correction that PTP_T performs for the second-derivative test. For each of six permutations πk _k, let a0=∇g(~)a_0=∇ g( y) and ak=∇g(Pπk~)a_k=∇ g(P_ _k y) denote the sensitivities at the identity and permuted labels, and let QkQ_k project orthogonal to span⁡,~,Pπk~span\1, y,P_ _k y\. The intersection-projected displacement averages the residual change left after this projection: Δperm⟂=16∑k=16∥Qk(ak−a0)∥2∥Qka0∥2+η. _perm = 16 _k=1^6 Q_k(a_k-a_0) _2 Q_ka_0 _2+η. A second statistic asks whether a single fixed weight vector can fit the tangent-projected sensitivities at the identity and all six permutations at once. With Pk=PT(Pπk~)P_k=P_T(P_ _k y) the tangent projector evaluated at the k-th permuted point, and P0P_0 the tangent projector at the identity, the least-squares residual is LS=(min⁡∑k=06a^⁡∥Pk(a^−ak)∥22)1/2(∑k=06∥Pkak∥22)1/2+η.LS= ( _ a _k=0^6 P_k( a-a_k) _2^2 )^1/2 ( _k=0^6 P_ka_k _2^2 )^1/2+η. Both statistics use the numerical guard η=10−300η=10^-300 to keep the ratio finite when its denominator is itself close to zero. Both statistics vanish for every map that is affine on ℳM, by the gradient identity from the proof of Lemma 1: on ℳM, the gradient of such a map differs from the fixed weight vector a~ a only by components along 1 and ~ y, which at a permuted point become components along 1 and Pπk~P_ _k y, exactly the directions that QkQ_k removes and that the single choice a^=a~ a= a absorbs once the sensitivities are projected by PkP_k. Reported values average the per-query statistics within each setting. A.5 Calibration on maps with known behavior We check the lemmas and the statistics above against three constructed maps whose class is fixed by design, rather than only against the trained models, so that a nonzero or a zero value can be read against a known answer before it is trusted on a model whose class is unknown. All three controls pass through the same estimator as the models, so the check exercises the same computation that produces the numbers reported for the models and not a separate shortcut. The affine control on ℳM is a fixed-weight map, built like gag_a (Eq. (2)). The separable control on ℳM transforms one label at a time, built like ga,ϕg_a,φ (Eq. (4)). The coupled control lets labels interact. Table 3 reports all six statistics on the three controls. The two raw statistics are far from zero on the affine control: the naive permutation statistic Δperm _perm and the norm of the tangent curvature before its isotropic part is removed. This is the standardization trap of Section 3.2 made numerical, since a fixed-weight map still produces a large apparent change under these uncorrected measurements. The corrected curvature ATA_T removes this artifact and is zero, to within floating-point error, on the affine control, but it is not zero on the separable control, which does transform each label nonlinearly and so has genuine tangent curvature. XsepX_sep is zero on both the affine and the separable control, because Lemma 2 covers the affine control as the special case ϕk=0 _k=0 of the affine-plus-separable class, and it is nonzero on the coupled control, where every statistic in the table is nonzero. The separable control’s combination of a nonzero ATA_T and a zero XsepX_sep is the pattern that the main claim of Section 4.2 rests on: a model can fail the fixed-weight test and still pass the one-label-at-a-time test, and the calibration confirms that the estimator reports exactly this pattern when it is the true answer. Float32 calibration puts a floor under the zeros in the table, AT≤1.1×10−5A_T≤ 1.1× 10^-5 on the affine null and Xsep≤9.3×10−6X_sep≤ 9.3× 10^-6 on the separable null, which corresponds to a share of the squared curvature X¯sep X_sep (Eq. (6)) of 7×10−117× 10^-11. A separate, descriptive check perturbs pairs of labels directly rather than computing the Hessian. Because a bump to one label that preserves the mean and the norm of the labels changes the standardized values of other rows as well, such pairwise perturbations can still respond to the separable control even though it transforms only one label at a time, so they remain a descriptive check and not a substitute for XsepX_sep. Table 3: Calibration on maps with known behavior. Values use the same estimator as the models, at n=256n=256 in float64. The raw statistics Δperm _perm and ∥PTHPT∥F P_THP_T _F can be nonzero even when the map is affine on ℳM, which is the trap of Section 3.2 made numerical. Only XsepX_sep is also zero on a map that transforms one label at a time. The first-order statistics LS and Δperm⟂ _perm are zero on every map that is affine on ℳM. Control Δperm _perm ∥PTHPT∥F P_THP_T _F ATA_T XsepX_sep LS Δperm⟂ _perm Affine on ℳM 1.211.21 38.738.7 ×10−144\!×\!10^-14 ×10−143\!×\!10^-14 ×10−146\!×\!10^-14 ×10−147\!×\!10^-14 Separable on ℳM 1.231.23 9.79.7 1.111.11 ×10−143\!×\!10^-14 0.360.36 0.530.53 Coupled 0.960.96 0.730.73 0.720.72 0.720.72 0.750.75 0.960.96 Appendix B Experimental details This appendix documents the models, the datasets, and the Hessian computations behind the results of Sections 3 and 4, in the order a reader needs them: which models we evaluate and where each one first receives the context labels (Appendix B.1), which datasets and how many repetitions every experiment in the paper uses (Appendix B.2), and how we compute and validate the exact Hessians that the two certificates depend on (Appendix B.3). B.1 Models and where the labels enter This subsection specifies the five models evaluated in Section 4.2, the estimator inside each package whose output our derivatives target, and the point in each architecture where the context labels first enter the computation, so that the certificates of Section 3 can be traced back to a documented instrument. Table 4 lists the five models together with the readout we differentiate in each one: the specific function of the network’s output that supplies the internal prediction y^int y_int of Section 3.1. We differentiate this single readout inside each package rather than its deployed default prediction, because that default combines several such estimators, and Appendix C.2 tests the default predictions separately. Table 4: The five evaluated models and the readout we differentiate. The identifier names the checkpoint or model version that we load, and the readout names the function of the network’s output that supplies the internal prediction y^int y_int rather than the package’s default combined prediction. Every derivative computation in this paper uses this single readout, except the readout-head check of Appendix C.1. Model Identifier Readout TabICL-v2 (Qu et al., 2026) tabicl-regressor-v2 linear mean-over-quantiles TabPFN-v3 (Grinsztajn et al., 2026) ModelVersion.V3 bucket-softmax TabPFN-v2.6 (Grinsztajn et al., 2025) ModelVersion.V2_6 bucket-softmax TabFM 1.0 (Kong and Das, 2026) tabfm-1.0.0 (regression) scalar regression output TabDPT v1.2 (Hosseinzadeh et al., 2026) tabdpt1_2.safetensors bucket-softmax The five models also differ in where the context labels first reach the network, which matters for reading the results of Section 4.2. TabICL-v2 introduces the standardized label vector at two points in its computation, once before the column-wise set transformer that builds per-feature representations from the context, and once more at the in-context stage that forms the prediction from those representations, so that the same label vector we differentiate feeds both sites in a full forward pass. TabPFN-v3 follows the same two-stage design: it adds an encoding of each context label to the cell embeddings of that row before the attention that builds the row representations, and a second encoding before its in-context stage, and the same label vector feeds both sites. TabPFN-v2.6 injects the labels once, as an additional column that its alternating attention across features and rows then shares across the context. According to its released code (Google Research, 2026), TabFM also introduces the labels at two points: it adds an encoding of each context label to every feature cell of that row before attention across features and rows, and a second encoding to the compressed row representation before the in-context stage, and the same label vector feeds both sites. Released TabDPT v1.2 (Hosseinzadeh et al., 2026) instead re-encodes the labels into the values that attention combines at every layer, so its queries and keys receive no direct label injection even though its deeper residual streams still carry label-derived information indirectly. This description holds for the released TabDPT v1.2 inference code that we evaluate. We run each package with its default feature preprocessing for a single estimator, except that we turn off the feature normalization of TabFM, and we run every derivative computation in float32. We differentiate with respect to the internal standardized label vector ~ y of Section 3.1, and we treat the output conversion back to raw label units as a fixed map that plays no part in the derivative. Package versions, constructors, runtime settings, and the digests that identify the exact model state loaded for each experiment are recorded in the supplementary material. Forward passes and first-order derivatives use each package’s own attention computation, with mixed precision and the optional fused kernels of TabICL-v2 and TabDPT turned off, and the Hessian experiments select the standard, non-fused attention backend for the whole run. Second derivatives need an attention implementation with a second backward pass, so every Hessian in this paper substitutes standard (eager) softmax attention, and Appendix B.3 validates this substitution against the deployed kernels directly. Under eager attention, the first derivatives of the readout match those computed with the native kernels to a relative error of at most 8×10−58× 10^-5 for all five models, and the forward prediction of TabFM matches its native prediction to 1.2×10−51.2× 10^-5 standard deviations of the output. B.2 Datasets and protocol This subsection lists the datasets behind every experiment in the paper and the grid and replication each one uses, so that Table 5 serves as a single reference for what a given result covers and how many times we repeat the measurement. Datasets. The main experiment of Section 4.2 evaluates four datasets: a synthetic regression task we call synthetic_mi (the generator paragraph below), and the OpenML datasets kin8nm (189), pol (201), and superconduct (43174). An eight-dataset core grid extends this set with ailerons (296), abalone (44956), concrete (44959), and house_16H (574), and supplies the settings for the first-order tests and for the surrogate-replacement panel, except that the seven-dataset surrogate panel excludes abalone. Table 5 records the grid and the replication of every experiment in the paper, writing Stat 4 for the four-dataset main grid above, Core 8 for the eight-dataset grid, and Core 7 for its seven-dataset restriction, each crossed with the stated number of models. The rows with four models do not include TabFM, which enters only the exact-Hessian experiments and the Hessian validation of Appendix B.3. Table 5: Grid and replication of every experiment in the paper. Each row names an experiment and gives its grid (the datasets and models it covers) and its replication. A setting is one model evaluated on one dataset. Stat 4, Core 8, and Core 7 are the dataset grids defined in the text. n is the number of context rows, nqn_q the number of query points, M the number of label draws used to fit a surrogate, and probes the number of random tangent directions used to estimate a Hessian statistic without forming the full matrix (Appendix B.3). Experiment Grid and settings Replication First-order tests Core 8, 4 models, 32 n=256n=256, 3 contexts, nq=8n_q=8, 6 permutations Projected Hessian Stat 4, 4 models, 16 nq=4n_q=4, 3 contexts, 16 paired probes Probe-based certificates Stat 4, 4 models, 16 n=256n=256, 3 contexts, 64 probes, nq=8n_q=8 Forward second differences Stat 4, 4 models, 16 n=256n=256, 3 contexts, first query, 4 tangent directions per context, 3 step sizes Exact Hessian Stat 4, 5 models, 20 n=256n=256, 3 contexts, nq=4n_q=4 Hessian validation 2 datasets, 5 models, 10 n=256n=256 Readout head 2 datasets, 2 models, 4 n=256n=256, 2 contexts, nq=2n_q=2 Default predictions 2 datasets, 3 models, 6 2 contexts, and the feature orderings of TabDPT on the same 2 datasets Surrogate replacement Core 7, 3 models, 21 n=96n=96, M=400M=400, 300 test queries Observed-feature comparator Core 7, 4 models, 28 n=96n=96, 300 test queries Task-preserving surrogate 3 datasets, 4 models, 12 n=64n=64, nq=6n_q=6, M=210M=210 (wild bootstrap: kin8nm, pol, superconduct) Constructed local model 3 datasets, 3 models, 9 n=128n=128, nq=5n_q=5, 24 label draws (construction: 3 synthetic tasks at n=96n=96) Context size, RTR_T 2 datasets, 3 models, 6 n=128n=128–10241024, 1 context, nq=2n_q=2, 8 probes Context size, X¯sep X_sep 2 datasets, 2 models, 4 n=128n=128, 256256, 512512, 2 contexts, nq=2n_q=2 Random copies 2 datasets, 5 models, 10 12 copies per model, each evaluated on both datasets Random copies, Δperm _perm 2 datasets, 4 models, 8 n=128n=128 (TabICL-v2: 256256), 12 copies per model Attention-entropy control 2 datasets, 2 models, 4 3 copies per model, each evaluated on both datasets All-layer route patch Stat 4, 4 models, 16 n=256n=256, nq=8n_q=8, 6 permutations Controlled models Gaussian-process task 3 seeds, and a five-seed extension of 75 runs Pretraining overlap. The evaluated TabDPT v1.2 checkpoint (TabDPT-Turbo) was pretrained on 1,4451,445 OpenML datasets, from which its authors removed possible duplicates of the C18, CTR23, and TabArena benchmarks, and no list of these datasets is released (Hosseinzadeh et al., 2026). We therefore cannot check which of our real datasets it saw during pretraining. The 123123-dataset manifest of the original TabDPT (Ma et al., 2025) contains ailerons directly and pol and house_16H only through classification variants. Such an overlap does not change the measured response of the evaluated map, but it limits how far TabDPT’s results on these datasets can be read as evidence of generalization independent of pretraining. Metrics. The sensitivities ai=∂y^int/∂y~ia_i=∂ y_int/∂ y_i that the certificates use are local derivatives, and they describe a rate of change rather than an additive contribution to a finite prediction change. The first-order tests permute the standardized labels inside the model, so the context rows stay aligned without any further check. The surrogate experiments of Appendix C.6 fit an affine surrogate to a model’s predictions and evaluate it on label draws held out from fitting. The surrogate-replacement panel fits it on 400400 label permutations, choosing the ridge regularization from a grid spanning 10−310^-3 to 10210^2 on an 85/15 split of these draws, and evaluates it on 100100 fresh permutations. The task-preserving surrogate fits it on 210210 of 300300 label draws, choosing the regularization from a grid spanning 10−310^-3 to 10310^3 by leave-one-out cross-validation, and evaluates it on the remaining 9090. The task-preserving draws of that appendix change the moments of the labels and are fitted in raw label coordinates, unlike the derivative measurements elsewhere in this appendix, which use standardized labels. Because the fit uses finitely many draws and a regularized estimator, a surrogate R2R^2 below 11 does not by itself establish that the residual gap reflects the underlying map rather than fitting noise or the chosen regularization. Synthetic generator. Features follow x∼[0,1]16x [0,1]^16. Rows of A∈ℝnidx×16A ^n_idx× 16 are drawn from a standard normal. The experiments on the certificate grid of Table 8 scale each row to unit Euclidean norm. These are the exact and probe-based certificates, the first-order tests, the forward second differences, the readout-head check, the context-size check of X¯sep X_sep, the default-prediction check, the certificates of the random copies, and the Hessian validation on TabICL-v2 and TabFM. The other experiments, including the context-size check of RTR_T, divide A by its largest column sum of absolute values. The target is f⁡(x)=∑k=1nidx∑j=16wkjexp⁡[−(Ak⊤x−ckj)22(0.2)2],y=f⁡(x)+0.05ε,f(x)= _k=1^n_idx _j=1^6w_kj \! [- (A_k x-c_kj)^22(0.2)^2 ], y=f(x)+0.05 , where wkj,ε∼⁡(0,1)w_kj, (0,1) and ckj=0.5∑iAki+ukjc_kj=0.5 _iA_ki+u_kj with ukj∼⁡(−0.6,0.6)u_kj (-0.6,0.6). The core task uses four indices. B.3 Computing and validating the Hessians Second derivatives need attention implemented with eager softmax, because the fused attention kernels that packages deploy do not support a second backward pass (Appendix B.1), so every exact Hessian in this paper comes from a different computational path than the predictions and first-order sensitivities that accompany it. Before treating these Hessians as evidence, we must show that substituting eager attention at this one step leaves the computed derivatives unchanged, which is what this subsection validates. Assembling and estimating the Hessians. We assemble each exact Hessian one column at a time. Differentiating the full sensitivity vector a=∇~g(~)a= _ yg( y) with respect to a single label y~j y_j gives Hej=∂a/∂y~jHe_j=∂ a/∂ y_j, the j-th column of H~H_ y, where eje_j is the j-th standard basis vector in ℝnR^n, and stacking its n columns recovers the exact Hessian. We compute each column twice on the same contexts, once from the full network and once with derivatives blocked from passing through the attention scores, the intervention of Section 4.3. Forming every column of a full Hessian at every setting in the paper would be costly to repeat, so several of the appendix analyses that do not need the exact matrix instead estimate ATA_T from a small set of shared tangent Rademacher probes. These probes are random tangent-space directions with independent ±1± 1 entries, and we use 88, 1616 or 6464 of them per setting to estimate the projected norm and the trace of the tangent curvature PTH~PTP_TH_ yP_T. The squared trace uses a split-product estimator, which multiplies two independent probe-based estimates of the trace rather than squaring one, so that the noise of a single estimate does not enter as a bias. Because a probe-based estimate of ATA_T can occasionally fall outside the range that a true squared norm allows, we clamp it to its admissible range, which introduces a small null bias in return: in a 6464-probe simulation of the affine control gag_a (Eq. 2), this bias has mean 0.480.48 and 9595th percentile 1.601.60, small against the projected norm of 38.738.7 measured in the same simulation. The exact Hessians reported in Table 2 do not use this clamp. We also re-center the standardized labels inside each tangent projector, which compensates for small floating-point differences between how different packages implement the standardization wrapper of Section 3.1. Validating the eager-attention path. The functions we differentiate compose linear maps, softmax, smooth activations (GELU, SiLU), and normalization layers (LayerNorm, RMSNorm) away from their degenerate points, and the discrete steps of the standardization wrapper sit outside the standardized-label leaf that these derivatives are taken against, so nothing discontinuous enters the derivative itself. We compare the exact Hessian-vector products computed under eager attention with central finite differences of the gradient returned by each package’s native fused kernel: a⁡(~+εv)−a⁡(~−εv)2ε, a( y+ v)-a( y- v)2 , where v is a probe direction and ε the finite-difference step size. We use the same seeded Rademacher directions for both computations, together with unit-vector directions that validate individual Hessian columns directly, and we select the step ε⋆ at a plateau in the relative error where adjacent step sizes give converged answers. At n=256n=256, every one of the ten model–dataset settings we tested meets our validation thresholds: cosine similarity of at least 0.990.99, relative error of at most 0.050.05, and a matched-column fraction of at least 0.900.90 between the two computations. The agreement is in practice closer than these thresholds require. Every median cosine rounds to 1.00001.0000, the highest median relative error is 0.00340.0034, every matched-column fraction reaches 1.001.00, and the selected steps are 10−310^-3 or 3×10−33× 10^-3, except for TabFM, whose float32 finite differences become noisy below 10−210^-2. Table 6 reports these ten settings, and the supplementary material holds the full validation and probe records. Table 6: Validating the eager-attention Hessians against the native fused-kernel gradient, at n=256n=256 on synthetic_mi and kin8nm, two of the four datasets in the main grid. For each model–dataset setting, ε⋆ is the selected finite-difference step, cos and rel-L2L^2 are the cosine similarity and relative L2L^2 error between the eager Hessian-vector product and the central finite difference of the native gradient along the same probe direction, reported as the median over probes and queries with the maximum in parentheses, matched is the fraction of Hessian columns on which the two computations agree, and ∥H∥F/∥a∥ H _F/ a gives the scale of the Hessian relative to the sensitivity vector for context, not a validation criterion. Every setting meets the thresholds stated in the text. Model Dataset ε⋆ cos rel-L2L^2 med (max) matched ∥H∥F/∥a∥ H _F/ a TabICL-v2 synthetic_mi ×10−33\!×\!10^-3 1.00001.0000 0.0008(0.0012)0.0008\ (0.0012) 1.001.00 5.515.51 TabICL-v2 kin8nm ×10−33\!×\!10^-3 1.00001.0000 0.0010(0.0020)0.0010\ (0.0020) 1.001.00 4.624.62 TabPFN-v3 synthetic_mi ×10−33\!×\!10^-3 1.00001.0000 0.0014(0.0046)0.0014\ (0.0046) 1.001.00 7.297.29 TabPFN-v3 kin8nm ×10−33\!×\!10^-3 1.00001.0000 0.0020(0.0063)0.0020\ (0.0063) 1.001.00 4.994.99 TabPFN-v2.6 synthetic_mi ×10−33\!×\!10^-3 1.00001.0000 0.0004(0.0006)0.0004\ (0.0006) 1.001.00 3.023.02 TabPFN-v2.6 kin8nm ×10−33\!×\!10^-3 1.00001.0000 0.0004(0.0005)0.0004\ (0.0005) 1.001.00 3.143.14 TabFM synthetic_mi ×10−21\!×\!10^-2 1.00001.0000 0.0024(0.0033)0.0024\ (0.0033) 1.001.00 22.122.1 TabFM kin8nm ×10−21\!×\!10^-2 1.00001.0000 0.0034(0.0065)0.0034\ (0.0065) 1.001.00 5.785.78 TabDPT synthetic_mi ×10−31\!×\!10^-3 1.00001.0000 0.0003(0.0004)0.0003\ (0.0004) 1.001.00 8.408.40 TabDPT kin8nm ×10−31\!×\!10^-3 1.00001.0000 0.0004(0.0006)0.0004\ (0.0006) 1.001.00 6.936.93 Appendix C Additional results C.1 Certificates in every setting This subsection supports Table 2 of the main text (Section 4.2), which reports each certificate averaged over four datasets per model. It adds three things to that table: (1) the per-setting values behind its means, (2) a first-order version of the fixed-weight test that does not depend on differentiating the sensitivities twice, and (3) checks that the interaction found by the second-order certificates is not an artifact of the readout head or of the exact-Hessian instrument used to compute it. It closes with a look at how the certificate behaves at context sizes other than the n=256n=256 rows used in the main experiments. First-order tests. A first-order counterpart to the fixed-weight test avoids differentiating the sensitivities twice. Appendix A defines the intersection-projected displacement Δperm⟂ _perm and the least-squares residual LS, two statistics built from the tangent gradient alone that are zero for every map that agrees with a fixed-weight map on ℳM. Both reject the affine null on all 3232 core settings, with LS residuals between 0.7220.722 and 0.9250.925, which shows that the departure from fixed weights does not depend on the second-order instrument. Table 7: First-order and probe-based certificate results, by setting. Each row averages three sampled contexts and eight queries per setting at n=256n=256. The columns AT/∥PTa∥A_T/ P_Ta , A¯T A_T, and Att. drop are estimated from tangent probes (Appendix B.3). AT/∥PTa∥A_T/ P_Ta scales the certificate by the norm of the tangent gradient, so that models with larger raw sensitivities do not appear to have a proportionally larger certificate. Att. drop is the relative fall of ATA_T when derivatives cannot pass through the attention scores (Section 4.3), and the brackets enclose the three per-context paired-probe 95%95\% bootstrap intervals. Δperm⟂ _perm and LS are the corrected first-order statistics of Appendix A, which likewise vanish for every map that is affine on ℳM. Model Dataset AT/∥PTa∥A_T/ P_Ta A¯T A_T Att. drop [interval] Δperm⟂ _perm LS TabICL-v2 synthetic_mi 5.08 0.991 0.874 [0.82,0.930.82,0.93] 1.08 0.893 TabICL-v2 kin8nm 4.40 0.993 0.814 [0.74,0.860.74,0.86] 1.02 0.865 TabICL-v2 pol 35.75 0.972 0.874 [0.79,0.920.79,0.92] 1.12 0.783 TabICL-v2 superconduct 10.05 0.987 0.749 [0.68,0.820.68,0.82] 0.99 0.789 TabPFN-v3 synthetic_mi 8.76 0.996 0.933 [0.88,0.970.88,0.97] 1.01 0.915 TabPFN-v3 kin8nm 4.82 0.993 0.850 [0.81,0.910.81,0.91] 1.00 0.907 TabPFN-v3 pol 57.03 0.960 0.882 [0.84,0.910.84,0.91] 1.52 0.885 TabPFN-v3 superconduct 36.32 0.996 0.880 [0.72,0.960.72,0.96] 1.14 0.907 TabPFN-v2.6 synthetic_mi 3.14 0.988 0.912 [0.86,0.950.86,0.95] 1.21 0.899 TabPFN-v2.6 kin8nm 2.71 0.985 0.849 [0.83,0.870.83,0.87] 1.10 0.885 TabPFN-v2.6 pol 16.30 0.949 0.964 [0.95,0.970.95,0.97] 2.52 0.922 TabPFN-v2.6 superconduct 7.51 0.989 0.932 [0.91,0.950.91,0.95] 1.38 0.901 TabDPT synthetic_mi 9.27 0.994 0.840 [0.75,0.890.75,0.89] 1.00 0.925 TabDPT kin8nm 6.71 0.995 0.762 [0.73,0.810.73,0.81] 1.01 0.922 TabDPT pol 31.04 0.948 0.655 [0.50,0.740.50,0.74] 1.33 0.810 TabDPT superconduct 10.32 0.991 0.568 [0.50,0.610.50,0.61] 0.99 0.879 Shares of the squared curvature in every setting. The exact normalized certificate X¯sep X_sep is 0.2390.239 to 0.9160.916 across the 2020 settings, with a mean of 0.630.63 (Table 8). The exact A¯T A_T ranges from 0.9520.952 to 0.9970.997. On the 1616 settings where both instruments were run, it ranges from 0.9520.952 to 0.9960.996 and closely matches the cheaper probe estimate of the same quantity, 0.9480.948 to 0.9960.996, which checks that the probe-based instrument used in Table 7 agrees with the exact Hessian used elsewhere. Table 8: Exact certificate of joint processing, by setting. Behind the per-model means of Table 2, each row reports X¯sep X_sep for one setting as the mean over three sampled contexts and four queries at n=256n=256, with the range of the three context means (Range). This certificate is exactly zero for every map that is affine plus one-label-at-a-time on ℳM (Section 3), and its float32 implementation floors it at 7×10−117× 10^-11. A¯T A_T also comes from the exact Hessian, whereas Table 7 estimates it from probes. Xsep/∥PTa∥X_sep/ P_Ta scales XsepX_sep by the norm of the tangent gradient. The two Att. drop columns give 1−AT(Hdet)/AT(H)1-A_T(H_det)/A_T(H) and 1−Xsep(Hdet)/Xsep(H)1-X_sep(H_det)/X_sep(H), the relative falls of ATA_T and XsepX_sep when derivatives cannot pass through the attention scores. Att. drop Model Dataset X¯sep X_sep Range A¯T A_T Xsep/∥PTa∥X_sep/ P_Ta ATA_T XsepX_sep TabICL-v2 synthetic_mi 0.8640.864 [0.836,0.900][0.836,0.900] 0.9900.990 4.804.80 0.8780.878 0.8950.895 TabICL-v2 kin8nm 0.7720.772 [0.700,0.813][0.700,0.813] 0.9930.993 4.174.17 0.8200.820 0.8480.848 TabICL-v2 pol 0.3220.322 [0.306,0.353][0.306,0.353] 0.9770.977 20.720.7 0.8850.885 0.8380.838 TabICL-v2 superconduct 0.5660.566 [0.552,0.575][0.552,0.575] 0.9900.990 5.945.94 0.6810.681 0.7000.700 TabPFN-v3 synthetic_mi 0.8880.888 [0.844,0.922][0.844,0.922] 0.9960.996 8.498.49 0.9350.935 0.9680.968 TabPFN-v3 kin8nm 0.7450.745 [0.721,0.770][0.721,0.770] 0.9940.994 5.255.25 0.8640.864 0.9190.919 TabPFN-v3 pol 0.4960.496 [0.439,0.530][0.439,0.530] 0.9530.953 36.136.1 0.7830.783 0.8300.830 TabPFN-v3 superconduct 0.7020.702 [0.662,0.757][0.662,0.757] 0.9950.995 30.530.5 0.8520.852 0.8840.884 TabPFN-v2.6 synthetic_mi 0.8370.837 [0.772,0.896][0.772,0.896] 0.9810.981 2.822.82 0.8910.891 0.9520.952 TabPFN-v2.6 kin8nm 0.7740.774 [0.741,0.818][0.741,0.818] 0.9820.982 2.362.36 0.8570.857 0.9310.931 TabPFN-v2.6 pol 0.5670.567 [0.547,0.599][0.547,0.599] 0.9540.954 12.712.7 0.9620.962 0.9680.968 TabPFN-v2.6 superconduct 0.7580.758 [0.739,0.788][0.739,0.788] 0.9900.990 6.486.48 0.9260.926 0.9610.961 TabFM synthetic_mi 0.9160.916 [0.882,0.977][0.882,0.977] 0.9950.995 11.411.4 0.9210.921 0.9410.941 TabFM kin8nm 0.8290.829 [0.817,0.843][0.817,0.843] 0.9910.991 5.255.25 0.8930.893 0.9280.928 TabFM pol 0.2510.251 [0.242,0.260][0.242,0.260] 0.9850.985 30.430.4 0.5810.581 0.6280.628 TabFM superconduct 0.7030.703 [0.672,0.718][0.672,0.718] 0.9970.997 8.438.43 0.7340.734 0.7890.789 TabDPT synthetic_mi 0.4470.447 [0.366,0.553][0.366,0.553] 0.9920.992 6.376.37 0.8400.840 0.9140.914 TabDPT kin8nm 0.4350.435 [0.417,0.446][0.417,0.446] 0.9960.996 4.324.32 0.7850.785 0.8990.899 TabDPT pol 0.2390.239 [0.174,0.292][0.174,0.292] 0.9520.952 16.316.3 0.5500.550 0.5270.527 TabDPT superconduct 0.4960.496 [0.423,0.567][0.423,0.567] 0.9920.992 7.377.37 0.5460.546 0.6730.673 The interaction is present before the readout head. Three of the models, the two TabPFN models and TabDPT, turn the network’s logits over a grid of label bins into a prediction through a softmax (the bucket-softmax readout of Table 4). Such a head is itself a nonlinear map of the logits, so it can contribute curvature even when the logits are an affine function of the labels, through the chain-rule term Jℓ⊤(∇2h)JℓJ_ (∇^2h)J_ , where JℓJ_ is the Jacobian of the logits with respect to the labels and h is the head. Because this term would inflate the certificates for a reason unrelated to joint label processing inside the network, we recompute both certificates from a frozen linear functional of the pre-head logits, which removes the term. On the two TabPFN models, evaluated on two datasets and two contexts each, all eight context-settings keep nonzero certificates under this readout, with X¯sep X_sep between 0.500.50 and 0.970.97. Their ratio to the values obtained with the mean readout is 0.95±0.130.95± 0.13, with a minimum of 0.700.70, which places most of the measured interaction upstream of the bucket head on these settings. The differentiated curvature matches the curvature of actual predictions. The certificates above come from differentiating the internal prediction at the observed labels. As a check that this differentiated curvature matches the curvature that the network actually shows along the manifold, we compare it with a purely forward quantity: the second difference of the prediction itself, evaluated at points of ℳM near the observed labels. For a tangent direction v, the second derivative of a prediction function F along the renormalized label path is BF(v)=v⊤HFv−∥v∥2r2~c⊤∇F,B_F(v)=v H_Fv- v ^2r^2\, y_c ∇ F, (8) where ~c y_c is the centered standardized-label vector, HFH_F and ∇F∇ F are the Hessian and the gradient of F, and the second term removes the curvature that the path picks up from staying on the sphere, the same correction that builds T in Section 3. We compare BF(v)B_F(v) with central differences of the prediction along four unit tangent directions per context at the first query, for the four models other than TabFM, with steps of 0.050.05, 0.10.1, and 0.20.2. The full internal readout reaches a median setting-level relative error of 0.0460.046, and 1515 of the 1616 settings fall below the descriptive threshold of 0.50.5, with only TabDPT on pol above it at 0.5670.567. Context-size dependence. Because the certificates in Table 2 use n=256n=256 context rows, we also check whether the interaction strengthens or weakens with more context. For this check we use the ratio RT=∥PTHPT∥F/∥PTa∥R_T= P_THP_T _F/ P_Ta . Unlike ATA_T, this ratio keeps the isotropic part of the tangent curvature and simply scales its norm by the tangent gradient norm. For TabPFN-v3, TabPFN-v2.6 and TabDPT on synthetic_mi and kin8nm, RTR_T varies by at most a factor of 2.42.4 and shows no consistent trend from 128128 to 10241024 rows, so the same ratio divided by n n declines, and this descriptor does not show a strengthening of the curvature with context size. Two of the largest-context TabPFN-v2.6 settings exceeded available memory. For TabICL-v2 and TabPFN-v2.6 on the same two datasets, the exact normalized certificate X¯sep X_sep ranges over the context means, with two contexts and two queries at each size, from 0.800.80 to 0.960.96 at n=128n=128, 0.720.72 to 0.910.91 at n=256n=256, and 0.670.67 to 0.910.91 at n=512n=512, a mild decline that leaves the certificate present at every tested size. C.2 The default predictions are not affine in the labels The two certificates of Section 3 are computed on a single internal estimator inside each package, since this is the quantity for which the network’s own derivatives are well defined (Section 4.1). By contrast, each package returns a complete prediction that averages several such estimators together. This subsection checks separately whether that complete prediction is also non-affine in the labels, the check that Section 4.2 of the main text cites. Construction. The test perturbs the labels along two orthogonal tangent bumps δi _i and δj _j, each of scale ε , chosen so that δi⊤δj=0 _i _j=0. Starting from the standardized labels ~ y, a step v is applied through the exactly renormalized path m⁡(v)=y~¯+(~c+v)r/∥~c+v∥m(v)= y1+( y_c+v)r/ y_c+v , which keeps the mean and the norm of the labels fixed at every step, so that m⁡(v)m(v) always lies on the standardization manifold ℳM of Section 3.2. The test evaluates the four-point difference Dij=g⁡(m⁡(δi+δj))−g⁡(m⁡(δi))−g⁡(m⁡(δj))+g⁡(~).D_ij=g(m( _i+ _j))-g(m( _i))-g(m( _j))+g( y). For a map that is affine on ℳM, the order-ε2 ^2 term that the four-point difference could otherwise pick up from the curved path is proportional to δi⊤δj _i _j, so the chosen orthogonality sets this term to zero and the affine null becomes O⁡(ε3)O( ^3). The orthogonality of δi _i and δj _j is required for this cancellation, and a one-label-at-a-time map can still produce a nonzero response under this test. The native test applies the affine image of these paths in raw-label coordinates and evaluates the complete prediction returned by each package’s default inference procedure, with both the label mean and the label scale held fixed. Results. We calibrate the test on setting-specific controls: an affine map, and an impostor that agrees with an affine map on the fixed-moment set but not outside it, in the manner of gag_a. A third control, a map that lets the labels interact, checks that the test fires. With the member seeds held fixed, the measured interaction exceeds the floor set by the first two controls by a factor of 4444 to 16,90016,900 on 1212 context-settings, spanning three models, two datasets, and two contexts. We had required in advance that this floor stay below 5%5\% of the interaction of the third control. The two kin8nm contexts miss this requirement, with floors at 66 to 9%9\%, and they give the smallest factors, 4444 to 279279, which still place the measured interactions far above the floor. Averaged over the two contexts of each model and dataset, the ratio between the complete and the single-estimator response is 0.410.41 to 1.031.03 (0.300.30 to 1.371.37 for single contexts). TabDPT additionally averages over several feature orderings, and this separate averaging axis exceeds its calibrated floor by a factor of 5555 to 616616 on four context-settings, with ratios of 0.540.54 to 1.541.54 between the averaged and the single response. Scope. These functional checks support the finding that the default predictions are not affine in the labels. They do not exclude a complete prediction that transforms each label one at a time and combines the results without interaction, since such a map is also non-affine. The exact certificate XsepX_sep therefore remains scoped to the internal, single-estimator paths measured in Section 4.2. C.3 Random copies of the trained models Section 4.3 of the main text asks whether the interaction between labels comes with the architecture itself or appears only after training, and reports that the normalized certificate X¯sep X_sep separates trained networks from untrained ones while ATA_T does not (Section 4.3). This subsection gives the construction and the full range of values behind that comparison, together with a further control that matches the entropy of the attention scores between trained and random networks. Random copies. For each of the five models, we build 1212 random copies and evaluate each copy on the same two datasets, synthetic_mi and kin8nm. Each copy is built by calling the reset method of every module of the network, and we draw every parameter that these methods do not reach from a Gaussian with standard deviation 0.020.02. For TabFM, we also redraw from a standard normal the learned Fourier frequencies that its released code stores outside the parameter list. The copy is restored only after the standardization wrapper has been fit to the context, so that only the network’s own parameters differ from the trained model. Results. For the four models other than TabFM, the trained values are the means over the four queries of the first context of the grid behind Table 8, and each random copy is evaluated on the same context rows and labels with the first two of these queries. On kin8nm, the feature scaling of the two inputs also differs slightly, because it is fit on the context and query rows together. For TabFM, the trained values come instead from the first context and the first two queries of that grid, which match the shared context exactly on synthetic_mi and differ from it on kin8nm only by a feature rescaling of at most 0.020.02. This comparison separates the training-associated share of the curvature from the generic curvature that any nonlinear network carries even before training. Every query of a trained model gives X¯sep≥29.5% X_sep≥ 29.5\%, with means of 44%44\% to 98%98\%, whereas the random copies stay at most 1.7%1.7\% as a two-query mean and at most 3.4%3.4\% for a single query. The gradient-scaled certificate Xsep/∥PTa∥X_sep/ P_Ta shows the same separation: 2.492.49 to 21.721.7 for the trained models against a maximum of 0.0370.037 to 0.320.32 across the random copies. Random copies still carry generic curvature, with a mean AT/∥PTa∥A_T/ P_Ta of 2.02.0 to 2.32.3, so the corrected curvature norm ATA_T is nonzero for trained and random networks alike and, as a yes-or-no certificate, cannot distinguish them. For the four models other than TabFM, the naive permutation statistic Δperm _perm of the trained models falls within the range of 1212 separately drawn random copies on all eight settings (at n=128n=128 for all models but TabICL-v2), so this simpler statistic cannot separate trained from random either, in line with the same limit noted for ATA_T in the main text. Attention-entropy control. A further control asks whether matching the entropy of the attention scores, rather than leaving random copies at their default entropy, closes the gap. It covers three random copies per architecture on two architectures (TabICL-v2 and TabPFN-v2.6) and two datasets, for 1212 evaluations. Matching the attention entropy between the random copies and the trained network raises X¯sep X_sep only to 0.160.16 for TabICL-v2 and to 0.0040.004 for TabPFN-v2.6, and raises the gradient-scaled certificate to at most 1.331.33, far short of the trained values above. The TabDPT entropy comparison is descriptive rather than a matched control, because scaling its logits did not match its entropy to the trained network’s. Taken together, these controls support a training-associated change in how labels interact, without isolating that change as the only one that training introduces. Appendix C.5 follows the label response of small transformers through training, in models whose architecture we control. C.4 Paths through the attention scores Attention combines the representations with weights that a softmax computes from the attention scores. Section 4.3 shows that blocking derivatives through them removes most of the measured interaction between labels while leaving the prediction unchanged (Figure 1b), and Table 2 reports the resulting fall in ATA_T and XsepX_sep for the exact Hessian. This subsection reports that comparison in full, gives a direct comparison between the intervened and the full curvature operator, states the scope of this derivative-path intervention, and then reports a second intervention that changes the forward computation itself. We apply the intervention at every attention step the network executes: the attention scores keep their forward values unchanged, so the prediction does not change, while derivatives with respect to the labels are blocked from passing through them. Writing the resulting Hessian as HdetH_det, the difference H−HdetH-H_det isolates the curvature carried by the blocked derivative paths. For TabICL-v2, one label vector feeds both the column-wise set transformer and the label encoder of the in-context stage (Appendix B.1), and a replay of the full pipeline lets the derivatives pass through both sites, with forward-agreement checks confirming that the replay leaves the prediction unchanged. TabPFN-v3 and TabFM likewise receive one label vector at both of their label sites, and we differentiate their full forward pass, so their derivatives also pass through both sites. Table 9: Attention-path norm reductions. Every entry is the relative fall of a norm under the intervention described in the text, not a share of squared curvature and not a loss of accuracy. Values average over four datasets, with three contexts per dataset and four queries per context (eight for the probe estimate of ATA_T) at n=256n=256 context rows. The probe columns estimate the tangent curvature norm ∥PTHPT∥F P_THP_T _F from 1616 shared probes and ATA_T from 6464 probes, and the exact columns use Hessians assembled column by column on all 2020 settings. We measured TabFM with the exact Hessian only, so the probe means cover the other four models. Probe estimate Exact Hessian Model ∥PTHPT∥F P_THP_T _F ATA_T ATA_T XsepX_sep TabICL-v2 82.1%82.1\% 82.8%82.8\% 81.6%81.6\% 82.0%82.0\% TabPFN-v3 87.7%87.7\% 88.6%88.6\% 85.9%85.9\% 90.0%90.0\% TabPFN-v2.6 91.5%91.5\% 91.4%91.4\% 90.9%90.9\% 95.3%95.3\% TabFM – – 78.2%78.2\% 82.1%82.1\% TabDPT 69.6%69.6\% 70.6%70.6\% 68.0%68.0\% 75.3%75.3\% Mean 82.7%82.7\% 83.4%83.4\% 80.9%80.9\% 85.0%85.0\% A direct comparison of the full and the intervened curvature operator supports the same reading. The part of the curvature attributable to the blocked derivative paths, ∥PT(H−Hdet)PT∥F/∥PTHPT∥F P_T(H-H_det)P_T _F/ P_THP_T _F, reaches 0.960.96, 0.990.99, 1.001.00, 0.940.94, and 0.930.93 for TabICL-v2, TabPFN-v3, TabPFN-v2.6, TabFM, and TabDPT, in that order, while the part that survives the intervention, ∥PTHdetPT∥F/∥PTHPT∥F P_TH_detP_T _F/ P_THP_T _F, is 0.190.19, 0.140.14, 0.090.09, 0.220.22, and 0.320.32. Averaged over the twelve Hessians of each setting, the cosine between the two parts lies between −0.12-0.12 and 0.280.28, consistent with a small remainder and limited cancellation between them. This decomposition is relative to the evaluated computational graph, and an algebraically equivalent implementation could redistribute the derivative paths between the two terms. The intervention also only blocks derivatives, so the forward computation and the prediction stay unchanged. It therefore does not by itself validate a model in which the attention scores are frozen at their base-label values and never recomputed as the labels change. Such a forward intervention would cache every attention-score tensor at the base labels and reuse it on every perturbed forward pass, and measuring its own curvature would require that frozen map’s own gradient in the forward second-difference formula of Eq. (8), together with a comparison against that map’s own prediction differences. Recomputing the attention scores with a stop-gradient supplies neither. The ambient quotient kept in the supplementary records omits these gradient terms as well, so it does not identify a signed share of the intrinsic curvature either. The norm reductions reported above therefore keep their scope as comparisons between derivative paths in the computational graph as evaluated. Replacing the cached attention maps. We report a second intervention that does change the forward computation, as a stress test of how much the attention scores contribute to the prediction itself, rather than only to its derivatives. We run two forward passes. The first pass computes the network on one set of labels and caches the attention scores of every attention step in the order the network computes them. The second pass recomputes the network’s hidden states from a second, possibly different set of labels, but at every attention step it replaces its own attention scores with the corresponding cached ones, in the same order. When both passes use the same labels, this hybrid computation reproduces the native internal forward pass exactly on all 1616 settings of the four models other than TabFM, which this intervention does not cover. Replacing the cached scores by those computed from a zero label vector instead shifts the prediction by 0.600.60 to 0.970.97 standard deviations of the model’s own native predictions, averaged within each model. Because the hybrid computation applies later-layer cached scores to hidden states that differ from the ones on which those scores were computed once the two passes receive different labels, the exact agreement obtained under identical labels validates only that the replay reproduces the native forward pass correctly. It does not by itself show that varying the cached scores isolates a causal effect of the attention scores. An affine map fitted from the replayed internal outputs to the package’s own predictions in the original label units reproduces these predictions to within 0.96%0.96\% relative error. Replacing the cached scores worsens the normalized mean squared error (NMSE, the mean squared error divided by the variance of the query targets) on 1313 of the 1616 settings and improves it on the remaining three, with a median ratio of 5.125.12 between the patched and the native error. Interpolating between the true-label scores and the zero-label scores increases the displacement from the native prediction monotonically on 1515 of the 1616 settings. Replacing the cached scores at a single attention call can produce a larger displacement than replacing them at every call, which shows cancellation across calls and excludes an explanation in which each call’s attention scores contribute additively to the total displacement. Per-setting results are in the supplementary material. C.5 Controlled models The pretrained models evaluated above cannot be retrained under a different wiring, so answering which components the measured interaction needs, and whether attention scores that depend on the labels are needed at all for a model to use labels well, requires models whose architecture we control ourselves. This subsection trains small transformers in which we vary how labels enter the network, whether LayerNorm is present, and whether the attention scores are allowed to depend on the labels. A controlled transformer on a Gaussian-process task. We train a three-layer, 6464-dimensional row-token encoder for 40004000 Adam steps on a two-dimensional radial-basis-function Gaussian-process prior, under either Gaussian observation noise or 15%15\% outlier contamination. Across three seeds, the held-out loss falls from about 1.31.3 to 0.020.02 under Gaussian noise and 0.030.03 under contamination within 10001000 steps. These models receive the raw labels without standardization, so every label vector is a valid input and the ambient derivatives are determined by the predictions, which makes the correction of Section 3.2 unnecessary here. We track two descriptors across training, computed directly on these derivatives: the permutation response Δperm _perm (Eq. 7), which measures how much the sensitivity vector changes under a permutation of the context labels, and the off-diagonal ratio ∥H−diag⁡(H)∥F/∥H∥F H-diag(H) _F/ H _F of the ambient Hessian. The permutation response grows from Δperm≈0.35 _perm≈ 0.35 at initialization to 0.85±0.500.85± 0.50 under Gaussian noise and 1.38±0.291.38± 0.29 under contamination, and the final off-diagonal ratio reaches 0.440.44 to 0.500.50. Because the Gaussian task fixes the kernel and the noise level, its Bayes-optimal predictor, the Gaussian-process posterior mean, is exactly linear in the labels, so this learned nonlinearity alone does not establish that it is useful, and the higher response under contamination is suggestive but not conclusive, given the spread across seeds. This small controlled model supports no claim about scale. LayerNorm is not necessary. Because LayerNorm’s normalizing statistics are themselves computed from representations that already mix the labels, we ask whether blocking derivatives through the attention scores is confounded with blocking derivatives through LayerNorm, or whether the two make separate contributions. In a transformer of the same size trained on the contaminated task, blocking derivatives through the attention scores removes 92±2%92± 2\% of the ambient off-diagonal norm across three seeds, and blocking derivatives through both the attention scores and the LayerNorm statistics removes 98%98\%. Blocking the LayerNorm statistics alone removes 85%85\%, because normalization follows label mixing, so the two interventions overlap rather than acting on separate parts of the interaction. Yet a model trained with no LayerNorm at all reaches a comparable loss (0.0280.028 against 0.0260.026 for the model with LayerNorm), keeps an off-diagonal ratio of 0.410.41, and still loses 97%97\% of its off-diagonal norm when derivatives cannot pass through the attention scores. LayerNorm is therefore not necessary for the interaction, since removing it leaves both the interaction and its concentration in the attention scores in place. Useful label processing without label-dependent attention scores. We next ask whether a model can process labels nonlinearly, and use that nonlinearity effectively under contamination, without ever letting the attention scores depend on the labels. A three-seed comparison matches depth, width, number of heads, task, and training budget across four ways of wiring labels into the network: injecting them into the same tokens from which the attention scores are computed, as in four of the five pretrained models above (token injection), injecting them only into the values that attention combines, as in TabDPT (value-only injection), computing the attention scores from the features alone while a separate nonlinear stream processes the labels, so that the labels cannot influence the attention scores at all (the score-blocked model), and a fully label-linear control. Under symmetric contamination, the score-blocked model reaches MSE 0.0215±0.00080.0215± 0.0008, within 9%9\% of token injection’s 0.0198±0.00090.0198± 0.0009, while the label-linear control reaches 0.0388±0.00030.0388± 0.0003, 96%96\% worse than token injection. Clean-task MSE ranges from 0.01450.0145 for token injection to 0.02140.0214 for value-only injection. In the score-blocked model, a smaller share of the curvature comes from interaction that no one-label-at-a-time map can produce, X¯sep=0.085±0.145 X_sep=0.085± 0.145 against 0.243±0.0800.243± 0.080 for token injection. A five-seed extension of this comparison covers 7575 runs with saved training curves. Under symmetric contamination, the score-blocked and label-linear losses are respectively 5%5\% and 91%91\% above token injection. Because the initial score-blocked model has more parameters than token injection (175,937175,937 against 100,865100,865), we also train a slim version with 107,993107,993 parameters, 7%7\% more than token injection’s count, which reaches MSE 0.0243±0.00160.0243± 0.0016: within 20%20\% of token injection and 1.61.6 times better than the label-linear control. One-sided outlier contamination reproduces the same ordering, with the score-blocked model 8%8\% above token injection against 142%142\% for the label-linear control. The share of the curvature norm that passes through the attention scores also changes with how labels enter the network, but its direction reverses across contamination regimes: the share for value-only injection is below token injection’s under symmetric contamination (0.8290.829 against 0.9140.914) and above it under one-sided contamination (0.9860.986 against 0.9310.931). These constructed models establish a useful alternative to label-dependent attention scores, without identifying a necessary pathway in the deployed models. C.6 How close a fixed-weight map comes Section 4.2 shows that no fixed-weight map reproduces any of the five models exactly, since A¯T A_T is nonzero in every setting (Table 2). Section 5 nonetheless reports that a fixed-weight surrogate fitted to a model’s predictions reaches a median held-out R2R^2 of 0.720.72, so failing the exact test does not mean that a model is far from such a map on the labels it meets in practice. This subsection reports the comparators behind that finding: an observed-feature comparator tuned on the context, a surrogate fitted under a task-preserving label distribution, and a constructed local-linear model with an explicit label-adaptive gain. Figure 3 relates the fidelity of a permutation-fitted surrogate to the predictive advantage of the models. An observed-feature comparator. A first comparator tunes Gaussian and dot-product kernel ridge regression, k-nearest-neighbor regression, and local-linear regression on a held-out fifth of the context rows, and we report the best of the four on the held-out queries, a choice that favors the comparator. Each candidate map is label-linear, while the tuning itself need not be. On 2828 settings, four models and seven datasets at n=96n=96 context rows, the selected comparator comes within 10%10\% of the model’s accuracy on 99 of the 2828 settings, with a median held-out NMSE gap of +0.066+0.066. A ridge surrogate fitted directly to the model’s predictions performs worse than this comparator on 2727 of the 2828 settings. This comparator differs from the jointly trained, label-conditioned embeddings of Miftachov et al. (2026). Task-preserving approximation. To keep the label distribution close to the one seen in practice rather than far from it, we draw fresh observation noise for the synthetic task and bootstrap the residuals of a gradient-boosted decision-tree (GBDT) fit on the real datasets: y(k)=m^(X)+resample⁡(y−m^(X)),y^(k)= m(X)+resample (y- m(X) ), where m m is the GBDT fit to the data and resample⁡(⋅)resample(·) redraws residuals with replacement. We fit raw-label surrogates on 210210 of 300300 label draws and evaluate them on the other 90, at n=64n=64 context rows and six queries per setting. Table 10 reports a median held-out R2R^2 of 0.7230.723 under this task-preserving resampling, against 0.480.48 under label permutations, while the label-linear Nadaraya–Watson control is recovered exactly, at R2=1.0000R^2=1.0000 on all 1212 settings. Table 10: Held-out surrogate fidelity. Held-out R2R^2, averaged over the six queries, of a surrogate that is affine in y, fitted to each model’s predictions under three label-draw distributions: Rtask2R^2_task under the task-preserving resampling described in the text, Rperm2R^2_perm under label permutations, and RNW-lin2R^2_NW-lin when the same fitting protocol is applied to a label-linear Nadaraya–Watson control instead of a model. A value of 11 means that the surrogate reproduces the predictions exactly on held-out draws, so the control column shows that the protocol recovers a genuinely affine map whenever one is present. The bottom row gives the median over the 1212 settings. Model Dataset Rtask2R^2_task Rperm2R^2_perm RNW-lin2R^2_NW-lin TabICL-v2 synthetic_mi 0.97 0.68 1.00 TabICL-v2 kin8nm 0.71 0.65 1.00 TabICL-v2 pol 0.68 0.56 1.00 TabPFN-v3 synthetic_mi 0.93 0.28 1.00 TabPFN-v3 kin8nm 0.75 0.55 1.00 TabPFN-v3 pol 0.57 0.13 1.00 TabPFN-v2.6 synthetic_mi 0.99 0.42 1.00 TabPFN-v2.6 kin8nm 0.69 0.53 1.00 TabPFN-v2.6 pol 0.74 0.28 1.00 TabDPT synthetic_mi 0.95 0.05 1.00 TabDPT kin8nm 0.57 0.03 1.00 TabDPT pol 0.71 0.62 1.00 median 0.72 0.48 1.00 Residual resampling of this kind removes the feature-dependent structure of the noise. A separate check on kin8nm, pol and superconduct instead uses wild-bootstrap multipliers, Rademacher and Mammen, which rescale each row’s own residual by a random factor, of magnitude one for the Rademacher multiplier. The median R2R^2 is 0.7080.708 under i.i.d. residual resampling, 0.7520.752 under the Rademacher multiplier, and 0.7710.771 under the Mammen multiplier, with a median span of 0.0970.097 across the three within each setting. These resampling schemes are proxies for the task’s label distribution, distinct from the fixed-moment manifold ℳM that the certificates of Section 3.2 use. Because permutations may lie farther from TabDPT’s real-data pretraining distribution, comparing near-task fidelity across models is the safer choice here than comparing permutation fidelity. A local Taylor-remainder check failed numerical stability on 22 of the 1212 settings, so it supplies no claim about the remainder curve. Figure 3: Approximation fidelity and predictive cost. The same 2121 settings from three prior-fitted models show (a) the held-out permutation-surrogate R2R^2, (b) the increase in NMSE when the model’s own predictions are replaced by this surrogate, and (c) the relation between the surrogate’s error and the model’s predictive advantage, which is the held-out NMSE of the best of three tuned label-linear smoothers (k-nearest neighbors, kernel ridge regression and local-linear regression) minus the model’s own NMSE, so that a negative value means that a smoother predicts better than the model. Replacement increases the error on all 2121 settings, but the correlation between the two magnitudes is −0.03-0.03, so this is an approximation comparison and not a causal ablation. TabDPT is not plotted here. A separate permutation fit with 200200 label draws at n=96n=96 gives it an R2R^2 of −0.92-0.92 and −3.18-3.18 on synthetic_mi and kin8nm, against 0.050.05 and 0.030.03 under the protocol of Table 10, so its permutation fidelity depends strongly on the protocol. A constructed local model. A third comparator asks whether a simple, interpretable label-adaptive map can reproduce the models’ behavior, rather than only matching their predictions. It weights each context row by a Gaussian kernel in the features together with an exponential gain in that row’s own deviation from the mean context label, before a local-linear readout that uses these weights: w~i=exp⁡(−∥x∗−xi∥22h2)exp⁡(λ⁡(yi−y¯)), w_i= \! (- x_*-x_i ^22h^2 ) (λ(y_i- y) ), where h is the kernel bandwidth and λ sets the strength of the label-adaptive gain. Choosing (h,λ)(h,λ) reproduces the aggregate response profile on three synthetic tasks with 33, 88, and 1616 indices at n=96n=96, and setting λ=0λ=0 removes the label adaptation while still allowing the local-linear readout to carry signed weights. This construction nonetheless fits the trained models poorly. On the nine settings of TabPFN-v3, TabPFN-v2.6 and TabDPT on synthetic_mi, kin8nm and pol at n=128n=128, its median held-out R2R^2 is 0.020.02, against approximately one for maps that already belong to this class. A variant that repeats the reweighting, recomputing the gain of each row from its deviation from the previous prediction for up to eight rounds, does not fit better, since its fit selects a single round on 88 of the 99 settings.

Instant research discovery