Paper deep dive
RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions
Rishabh Ranjan, Vignesh Kothapalli, Harshvardhan Agarwal, Charilaos Kanatsoulis, Roshan Upendra, Tom Palczewski, Carlos Guestrin, Jure Leskovec
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Foundation models for tables and relational databases are increasingly used for predictive tasks across various scientific and industrial domains. However, adapting existing models to unseen datasets requires tens of thousands of examples. Here we introduce RT-J, an 85M-parameter relational foundation model (RFM) that makes accurate predictions even with hundreds of in-context examples or fewer. To achieve this, we first assemble the Join, a large collection of 639 web-crawled databases with 6,023 LLM-generated forecasting tasks. We pretrain with long contexts of up to 8,192 database cells, using in-context examples from a novel lightweight random-walk retriever. Finally, we leverage context tuning and ensembling to scale up test-time compute. On in-context regression tasks from RelBench, RT-J requires 10x fewer examples on average to match strong feature engineering pipelines with LLM-agent or RDBLearn featurizers and TabICLv2 or LightGBM predictors. Overall, our work pushes the frontier for context-efficient predictions on relational data and highlights the importance of real-world semantics, context engineering and large-scale pretraining for RFMs.
Tags
Links
- Source: https://arxiv.org/abs/2610.17867v1
- Canonical: https://arxiv.org/abs/2610.17867v1
Trouble viewing inline? Open PDF directly →
Full Text
139,423 characters extracted from source content.
Expand or collapse full text
RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions Rishabh Ranjan Vignesh Kothapalli Harshvardhan Agarwal Charilaos Kanatsoulis Roshan Upendra Affiliation: Stanford University SAP https://star-project.stanford.edu/rt-j Tom Palczewski Affiliation: Stanford University SAP https://star-project.stanford.edu/rt-j Carlos Guestrin Jure Leskovec Abstract Foundation models for tables and relational databases are increasingly used for predictive tasks across various scientific and industrial domains. However, adapting existing models to unseen datasets requires tens of thousands of examples. Here we introduce RT-J, an 8585M-parameter relational foundation model (RFM) that makes accurate predictions even with hundreds of in-context examples or fewer. To achieve this, we first assemble the Join, a large collection of 639639 web-crawled databases with 6,0236,023 LLM-generated forecasting tasks. We pretrain with long contexts of up to 8,1928,192 database cells, using in-context examples from a novel lightweight random-walk retriever. Finally, we leverage context tuning and ensembling to scale up test-time compute. On in-context regression tasks from RelBench, RT-J requires 10×10× fewer examples on average to match strong feature engineering pipelines with LLM-agent or RDBLearn featurizers and TabICLv2 or LightGBM predictors. Overall, our work pushes the frontier for context-efficient predictions on relational data and highlights the importance of real-world semantics, context engineering and large-scale pretraining for RFMs. †footnotetext: Correspondence to: <ranjanr,vigneshk,guestrin,jure>@cs.stanford.edu Figure 1: RT-J pretraining recipe. (1) We sample a database and prediction task from the Join. The task specifies the target table and column. (2) We sample a target row from the target table. Rows after the target row’s timestamp are dropped; kept target-table rows form the candidate in-context examples. (3) From the target row, many parallel random walks rank candidate examples by visit count, on the fly. (4) We process candidate examples in descending visit-count order, using each as the root of a local bounded-width breadth-first-search (BFS) over primary–foreign key links. The context size and BFS hyperparameters are sampled randomly. (5) The collected rows form the context window for the relational transformer (RT), which treats cells as tokens. The target table rows in the context enable in-context learning. The target cell is always masked; non-target cells may be masked optionally. (6) RT-J predicts all masked cells and is trained against their ground-truth values. 1 Introduction Foundation models have transformed how machine learning systems are adapted to new tasks [1, 2]. In language and vision, pretrained models can often solve new tasks using only a small number of examples in context [3, 4]. Yet this capability remains largely out of reach for relational data, even though relational databases and data warehouses store much of the world’s structured enterprise, scientific, and operational data [5, 6]. Today, strong relational predictors typically require task-specific training [7, 8, 9], or substantial labeled data [10, 11, 12, 13, 14, 15, 16]. This limits predictive modeling precisely in the settings where it is most valuable, including cases where labeled examples are scarce, and users need rapid ad hoc predictions for decision-making. Tabular foundation models (TFMs) like TabPFN [17, 18] and TabICL [19, 20] provide in-context learning for single-table prediction, and systems such as RDBLearn [21] extend them to relational databases by aggregating relational neighborhoods into engineered features. Such an approach can be effective with many labeled examples, but it flattens the database and removes the explicit relational structure needed to effectively reason over entities, relationships, and cross-table dependencies. Natively relational architectures like RelGT [8] and RelGNN [7] operate directly on multi-table databases by representing rows as nodes and primary–foreign-key links as edges, but require training for each schema and task. More recent schema-agnostic relational foundation models (RFMs), such as Griffin [22], KumoRFM [10, 11], and relational transformers (RTs) [23], pretrain across many databases and adapt to unseen ones at inference time. However, these models have not yet been tailored for strong prediction with only a few labeled examples. See Appendix C for related work. This inability is not primarily an architectural limitation, but a mismatch between existing RFM training recipes and inference with limited in-context examples. RT [23] provides a natural starting point because it operates directly over database cells, schema metadata, and relational links. However, the original RT recipe was not designed to turn the architecture into a strong in-context learner. There are three main gaps. First, publicly available real world relational pretraining data remains scarce [24, 25]. Second, existing pretraining recipes target zero-shot or fine-tuning regimes, which precludes in-context learning. Third, standard context construction either ignores relational structure or expands local neighborhoods naively, without preferentially retrieving rows relevant to the target. Figure 2: In-context regression. Examples are matched. RT-J at train context size of 81928192 cells fits around 225225 examples on average. At all example counts, RT-J is >2>2 nMAE points better than the best baseline; at all performance levels, RT-J needs ≳10× 10× fewer examples. In this paper, we show that a pretrained RT can become a strong context-efficient predictor when the recipe for training, data, context retrieval and test-time compute is designed for this regime. Our goal is not to introduce a new backbone architecture, but to identify the ingredients that turn an RT model into an effective in-context learner for relational databases. First, we develop a pretraining recipe (Figure 1) built on stochastic context sizes and hyperparameters, which exposes the model to long and diverse contexts. Second, to train with high-quality in-context examples, we propose a lightweight random-walk retriever (Figure 1) that fills the context window with rows selected for both relational proximity to the target and likelihood of carrying useful predictive signal. Third, we leverage context tuning and context ensembling to scale test-time compute. With this recipe, we first obtain RT-PluRel, an 8686M-parameter RT model pretrained on synthetic PluRel data [26]. With task-specific fine-tuning, RT-PluRel ranked first among all system and model submissions on the RelArena-α [27] leaderboard (Appendix A). Finally, to improve generalization to real data, we assemble the Join, the largest open pretraining corpus of real-world relational data to date, comprising 66K LLM-generated forecasting tasks and 1111K autocomplete tasks across 639639 diverse web-crawled databases. We refer to the model obtained by continued pretraining (CPT) of RT-PluRel on the Join with our recipe as RT-J. Compared to the original RT pretraining [23], our pretraining used 11×11× more total tokens with up to 8×8× larger contexts and a 4×4× larger model, amounting to roughly 70×70× more pretraining FLOPs. Together, these ingredients make RT-J far more context-efficient than strong feature-engineering pipelines (Figure 2). We compare RT-J against pipelines that compose an LLM agent or RDBLearn featurizer with a TabICLv2 or LightGBM predictor, each given the same examples as in RT-J’s context. Other RFMs such as KumoRFM [10, 11], Griffin [22] and RDB-PFN [28] do not support this regime, hence could not be compared against (Appendix B). On RelBench regression, RT-J fits 225225 examples on average at its max training context size of 81928192 database cells, and achieves a normalized mean absolute error (nMAE) which the strongest baseline matches only with roughly 10×10× as many examples. Moreover, with only 1515 examples, RT-J already beats the best error any pipeline reaches with 225225 examples. At matched example counts, RT-J leads at every context size by 2.32.3–5.05.0 nMAE points. On classification, RT-J leads by 3.13.1 AUROC points when examples are scarcest, and the strongest pipeline catches up only at the largest context size. We attribute this efficiency to RT-J processing the raw relational context natively, which feature-engineering pipelines flatten into a feature table via aggregation techniques. Our results demonstrate that context-efficient relational prediction can be attained with a pretraining recipe matched to this regime and scaling up along the axes of data, context and test-time compute. Overall, this is a step toward RFMs that work for databases the way LLMs work for text. In essence, pretrained once, reused across domains, and adapted to new tasks with only a few examples. 2 Background: Relational Transformers for Predictive Tasks on Databases Relational databases and relational structure. Let D denote a relational database with schema graph =(,ℰ)G=(T,E), where each node T∈T is a table and each directed edge (T~,T)∈ℰ( T,T) indicates that table T contains a foreign key column referencing the rows of the parent table T~ T. Each table T contains rows indexed by primary key(s) and columns consisting of numerical, categorical, textual, or temporal values, and optional foreign keys. Thus, the relational structure of D is characterized by the schema graph along with row-level bipartite connectivity via primary–foreign key links [6, 29, 26]. Prediction tasks on relational data. A prediction task specifies a target table T⋆T , a target column c⋆c , and a set of prediction rows ℐI in T⋆T . For each row r∈ℐr , the target value is yr=xr,c⋆y_r=x_r,c . The goal of a foundation model fθf_θ is to predict yry_r from the observed relational context around row r. For boolean or categorical targets, fθf_θ outputs a class distribution. For numerical targets, it outputs a real-valued prediction. This formulation covers both (1) autocompletion and (2) forecasting tasks. In an autocompletion task, T⋆T is an existing database table and c⋆c is one of its columns (such as imputing the price of a transaction). In a forecasting task, T⋆T may be a curated prediction table whose rows define future prediction queries (such as forecasting revenue for an entity in the next quarter). When either task is temporal, the context is restricted to rows available before the prediction time. In-context learning for prediction tasks. In a classical in-context learning approach, fθf_θ predicts the target column c⋆c for a query row r by conditioning on a support set rS_r of labeled rows from the target table T⋆T . Each support row i∈ri _r has an observed target value yi=xi,c⋆y_i=x_i,c . A local relational context iC_i is then constructed around each support row and a context rC_r around the query row r (with its target value masked). The model fθf_θ predicts yry_r using the support and query context as: y^r=fθ((i,yi):i∈r,r). y_r=f_θ\! (\(C_i,y_i):i _r\,C_r ). (1) Here, in-context examples are sampled explicitly as labeled rows, and each example is represented by its own relational context. However, support rows often retrieve overlapping cells through primary–foreign key links. For example, labeled rows for people who purchased the same shoe after seeing the same advertisement may include the same cells from the shoe table in their contexts. This duplication consumes the context budget and limits the number of examples in context. Context window of relational transformers. RT [23] instantiates the context preparation idea differently. Rather than separately sampling support rows and constructing an independent context around each one, RT samples a single query-centered context rC_r around the query row r (see stage (4) in Figure 1). This context is obtained by expanding from r along primary–foreign key links while keeping yry_r masked. During this expansion, the context may naturally include other rows from the target table whose target values are observed. Thus, the support examples are not sampled separately. They are induced implicitly by the relational context around the query row. Therefore, the prediction of an RT model is given by y^r=fθ(r) y_r=f_θ(C_r). The benefit is that overlapping relational neighborhoods are represented only once inside a single query-centered context, rather than being duplicated across independently sampled support examples. Masked token prediction with relational transformers. RT treats each cell in rC_r as a token and predicts the masked target cell with a masked token prediction objective. Thus, heterogeneous prediction tasks are reduced to a common ‘cell prediction’ problem over any database in a schema-agnostic manner. RT computes the input token embedding by combining the cell value, datatype-specific information, and frozen text embeddings of the corresponding table and column names. During the forward pass, each transformer block of RT applies three attention layers [26] in sequence: (1) column attention, where tokens attend within the same column, (2) feature attention, where tokens attend within the same row, and (3) neighbor attention, where tokens attend across rows connected by primary–foreign key links. This attention structure enables information sharing across rows, columns, and tables while preserving the relational structure of the database. See Appendix F for details. 3 The Join: A Large-Scale Pretraining Corpus of Relational Data Existing collections of relational datasets, such as RelBench [29, 24], CTU [25], BIRD [30], and Spider [31, 32] comprise diverse databases but fail to capture the structural and distributional variety of predictive tasks needed for training RFMs. To address this issue, we introduce the Join, a corpus comprising 639 relational databases across diverse domains (e-commerce, sports, media, finance, healthcare, etc.) and 6K forecasting tasks spanning various levels of difficulty, including rare events and cold-start entities. We follow a four-stage approach to curate the corpus: (1) collection of raw databases from the web, (2) standardization of data types in tables, (3) task generation using LLMs, and (4) task filtering using an XGBoost model. See Appendix D for details. 3.1 Collection We collected the raw data from five disjoint source classes: (1) curated relational repositories (CTU, OGB, OpenAlex), (2) public APIs (SEC EDGAR, CFPB, World Bank, EIA), (3) domain specific portals (Lahman, ATP, StatsBomb, Cricsheet, OpenFlights), (4) community dumps (Stack Exchange), and (5) benchmarks (Kaggle, Spider/BIRD). Every contributing source is either a bulk archive published under a permissive license or a public API whose published terms permit programmatic research use. The classes carry qualitatively different relational structures: CTU schemas are small, clean, and follow third-normal-form, whereas Stack Exchange dumps are wide and denormalized. As a preliminary quality filter, we only collect databases that have at least two tables joined by a foreign key, and at least one column that can be parsed as a timestamp. Table 4 lists the domains included in the corpus. To avoid contamination, we exclude databases that overlap with ones in RelBench [29], RelBench v2 [24] and 4DBInfer [33] using source identity rather than schema similarity. 3.2 Standardization The databases collected in their raw form are usually messy, with inconsistencies across column data types, timestamps, and primary-foreign key links being the major issues. For example: natively boolean columns are represented as strings "true"/"false", dates are represented as ISO strings, and foreign keys exist only as a column-naming convention with no explicit declaration. To address such issues, we apply five idempotent detector-and-transformer fixes that recover the canonical relational structure: (1) boolean coercion, rewriting string and uint8 columns whose values fall within a canonical truth-set as booleans, (2) numeric coercion, casting string-encoded numerics to integers or floats with an identifier blacklist guarding zip codes and ISBNs, (3) foreign-key inference by name match plus an ≥80%≥\!80\% value-containment test against parent primary keys, (4) temporal normalization, electing a single time column per table from regex, name-hint, and integer-year candidates; and (5) entity-axis subsampling of oversized databases, retaining all activity rows for sampled entities to preserve their full activity histories. 3.3 Task generation While cell imputation tasks are naturally defined by simply masking a cell/column in an existing database, in forecasting tasks the target cell/column (e.g., revenue next month) does not appear in the database and has to be explicitly generated. To obtain forecasting tasks we proceed as follows. Audit and eligibility. Each database is first parsed into a typed schema graph with two named roles: an entity table (whose primary key is referenced by a foreign key (FK) from another table, identifying a long-lived object such as a customer, player, or hospital) and an activity table (with both a timestamp column and an FK to an entity table, so each row is a timestamped event attributable to an entity). A database is marked as eligible for task generation if it has at least one entity table, at least one activity table, ≥5≥\!5 unique timestamps in that activity table, and ≥100≥\!100 rows. The first two conditions ensure the schema can express a per-entity forecasting task. The timestamp and row count conditions allow us to perform XGBoost-based filtering of the generated tasks for quality control. Propose and generate. Proposing forecasting tasks for a database requires subject-matter expertise and a scalable way to verify correctness. We use an LLM (Claude Opus 4.6) for the former and a lightweight deterministic checker for the latter. For every eligible database, the LLM is given the schema description in text and is asked to write a set of task proposals in JSON. Each proposal selects one of 2525 registered template families, grouped into eight reasoning regimes. These include aggregation, threshold/existence, trend/change, behavior/churn, ranking, adoption/novelty, streak, and tail-event tasks. The checker then iterates through each proposal and drops it if: (1) a referenced column does not exist, (2) the entity column is not a foreign key, or (3) the requested pattern is not one of the registered templates. The LLM proposer is run multiple times per database, where each pass sees the tasks already accepted and is asked specifically for templates with low coverage. 3.4 Task filtering Finally, we score each task using an XGBoost model over window-aggregation features. We retain tasks that are either learnable, where XGBoost beats the lightweight historical baselines, or where the signal is too weak for these baselines but still act as harder learning objectives for stronger models. Overall, this process gives us 6,0236,023 forecasting tasks, where each task is defined by its description and a table with timestamped labels (e.g., future revenue for each quarter)11 1 Our code is publicly available at: https://github.com/stanford-star/agentic-task-gen. See Appendix D for case studies and a manual audit of the tasks for their practicality and realism. 4 Pretraining Recipe Here we describe our key improvements to the pretraining recipe of the original RT paper [23], which enable large-scale pretraining and context-efficient predictions (Figure 1). 4.1 Long-context training The context size L is the number of cells presented to RT and is a key scaling axis to improve prediction quality. However, training at large L is computationally expensive. Further, the same pretrained model should be able to handle both small and large contexts at test-time, so that inference compute can be traded off against quality. Hence, we train at a mix of context sizes. For each training batch, we sample L∼Unif(ℒ)L (L) from a fixed set ℒL of context sizes. We keep batch size B constant irrespective of L for stable optimization. As GPU memory scales as L×BL× B, we use gradient accumulation adaptively to prevent out-of-memory (OOM) errors at large L. 4.2 Retrieval The original RT recipe [23] samples the relational context rC_r with a bounded-width breadth-first search (BFS) over primary and foreign key links starting from the target row r. As the context length L grows, BFS can exhaust the local neighborhood, retrieve less relevant rows, and miss useful support rows from the target table. Retrieval is especially important for relational data because contexts are heterogeneous. For example, when predicting whether a user will click on a shoe ad, plain BFS may spend the context budget on generic user history or similar product cells. We use a two-stage sampling approach based on random walks to first retrieve similar rows and then expand the local neighborhood of each such row to prepare the context. Our approach can hence prioritize nearby target rows repeatedly reached through shared users, shoes, and ads tables for high quality contexts. Stage 1: Random-walks for global retrieval. The first stage uses random walks inspired by Pixie [34] and PinSAGE [35]. We run W random walks of length K from the target row r and record visit counts for rows in the target table T⋆T . We rank these rows by visit count, optionally using timestamps when the recency bias parameter ρ is set to 11. The walk parameters W and K approximate the random-walk ordering efficiently and are tuned for latency. Unlike vector-similarity retrieval, this sampler requires no pre-computation and can run on the fly across large databases in a highly parallelizable manner. For example, we use W=10,000W=10,000 random walks, each of K=20K=20 steps, with zero data loading overhead as the context sampler running on the CPU is ready with the next batch before the forward and backward passes have completed on the GPU. While random walks add a small latency overhead at test-time, our sensitivity analysis in Appendix I shows that lower values of W and K reduce this overhead significantly while largely retaining accuracy. Stage 2: BFS for local retrieval. For the second stage, we use the ranked rows as seeds for local BFS with context size ℓ and width bound b until the total context size reaches L. The full procedure is given in Algorithm 1 (Appendix I). Along with L, the retrieval parameters (ℓ,b,ρ)( ,b,ρ) can be tuned at test time. To allow for this, we need to expose the model to diverse contexts during training. Hence, we randomly sample (ℓ,b,ρ)∼Unif(ℒℓ×ℬ×ℛ)( ,b,ρ) (L_ ×B×R) using the values listed in Section 5. 5 Pretraining Results Pretraining details. We use an 8686M-parameter RT model with 12 layers, 512 hidden size and 8 attention heads, incorporating the improvements of [26] (such as QK-Norm) over the original architecture of [23]. During pretraining, we sample context size L from ℒ=512,…,8192L=\512,…,8192\, the local context size ℓ from ℒℓ=128,…,8192L_ =\128,…,8192\, the BFS width-bound b from ℬ=8,…,256B=\8,…,256\ (all powers of two) and the recency parameter ρ from ℛ=0,1R=\0,1\, and mask only the target cell (see Appendix H for experiments with multi-cell masking). We pretrain in two phases, first on the synthetic PluRel corpus [26] (86,21186,211 tasks over 1,9001,900 databases)22 2 The dataset is publicly available at: https://huggingface.co/datasets/stanford-star/plurel and then on 13,24313,243 tasks from a 523523-database subset of the Join (excluding the largest DBs due to memory bottlenecks). We optimize with Muon [36] at a constant learning rate after warmup and select the best checkpoint after stochastic weight averaging (SWA) [37] as recommended by Ranjan et al. [38]. The same checkpoint is used for all classification and regression tasks. Each phase takes about a day on a single node with 8×8×A100 GPUs. See Appendix G for details. Evaluation. We evaluate on RelBench [29], which comprises 2121 forecasting tasks across 77 databases (99 regression and 1212 binary classification tasks). These tasks span rare-event, cold-start, heavy-tailed and long-horizon regimes (see Appendix E for task descriptions). All databases used at evaluation are held out from pretraining. Following the official RelBench leaderboard, we use Z-score normalized mean absolute error (nMAE, lower is better) for regression and AUROC (higher is better) for classification tasks and report the average metrics. 5.1 Baseline comparisons Baselines. We compare RT-J against the strongest publicly available in-context learning (ICL) pipelines for relational prediction. A pipeline consists of a featurizer followed by a predictor. We consider two featurizers: (1) RDBLearn [21], which aggregates relational neighborhoods into a flat feature table and (2) an LLM Agent (Claude Code with Opus 4.6), which explores the database and writes SQL queries to perform feature engineering before applying a tabular predictor. We use two predictors: (1) TabICLv2, an in-context tabular foundation model [19, 20], and (2) LightGBM [39], gradient-boosted decision trees at the library’s default hyperparameters. Other RFMs such as KumoRFM [10, 11], Griffin [22], RDB-PFN [28] and TabPFN-Rel [15, 16] could not be used as baselines for reasons detailed in Appendix B. We compare against RT-PluRel in Section 5.2. Setup. For every task, we sweep the test-time context size L∈256,512,1024,2048,4096,8192L∈\256,512,1024,2048,4096,8192\ cells for RT-J, holding all other context hyperparameters fixed, ensuring that the 256256-cell context is a subset of the 512512-cell context and so on. For a fair label-matched evaluation of the baselines, we extract the task rows from the context prepared for RT-J, but do not further constrain the feature extraction or processing from the source database. Any checkpoint selection or hyperparameter tuning is done on the validation split, and all numbers are reported on the full test splits. Figure 3: RT-J vs baselines on the full RelBench test splits. On regression (left), RT-J dominates baselines at every label count, by up to 5.05.0 nMAE points over the strongest one. On classification (right), RT-J leads by up to 3.13.1 AUROC points. Per-task plots are in Appendix J. On regression, RT-J beats every baseline at every context size, by 2.32.3–5.05.0 nMAE points (see Figure 3). It is also far more example-efficient: with 1515 in-context examples it already beats the best error any baseline reaches with 225225, and the strongest pipeline needs around 23002300 examples, 10×10× more, to surpass it. Since all methods see the same examples, we hypothesize that the gap comes from RT-J better exploiting the relational context around them, which baselines flatten into a set of features. On classification, RT-J leads by 3.13.1 AUROC points when examples are scarcest, but the baselines improve steeply with more examples and the strongest catches up at the rightmost end. Among the baselines, the predictor matters more than the featurizer: TabICLv2 beats LightGBM at every context size on both metrics, while LLM Agent and RDBLearn features perform similarly. 5.2 Impact of synthetic and real pretraining phases Setup. We isolate the contribution of each pretraining phase by comparing three RT models with the same architecture: randomly initialized RT, RT-PluRel (PluRel phase only) and RT-J (PluRel followed by the Join), on both fine-tuning and in-context learning regimes. These pipelines include test-time scaling (Section 6) and reflect our submissions to the official RelBench leaderboard. Appendix A details both pipelines and reports per-task results. Observations. Table 1 shows the results. Synthetic pretraining on PluRel improves fine-tuning over a randomly initialized RT by 1.71.7 nMAE and 0.80.8 AUROC points, and continued pretraining on the Join adds 0.50.5 nMAE and 0.50.5 AUROC points on top. The real-data phase matters most in-context, where RT-J improves over RT-PluRel by 0.60.6 nMAE and 1.51.5 AUROC points. Table 1: Regression nMAE and classification AUROC on RelBench for RT models with no pretraining (RT), synthetic-only pretraining (RT-PluRel) and synthetic + real pretraining (RT-J), for both fine-tuning and in-context learning settings. Fine-tuned In-context Model Pretraining data nMAE % ↓ AUROC % ↑ nMAE % ↓ AUROC % ↑ RT none (random init) 29.329.3 77.877.8 – – RT-PluRel PluRel 27.627.6 78.578.5 32.932.9 73.073.0 RT-J PluRel → the Join 27.127.1 79.079.0 32.432.4 74.574.5 6 Test-Time Compute Scaling Considering the fixed context window of RT-J, choosing informative input cells for a predictive task becomes crucial. As the optimal context hyperparameters can be task dependent, and sampling cells with a particular configuration is inherently random, we employ (1) context tuning and (2) context ensembling as two complementary ways to improve predictions of RT-J. The compute, time and label overheads of tuning and ensembling approaches are compared in Appendix J. Here we use a subsampled test set for computational feasibility. 6.1 Context hyperparameter tuning Intuitively, during inference with RT-J, different tasks might prioritize width vs. depth of BFS, local vs. global neighborhoods, and recent vs. long-term patterns differently. We therefore tune (L,ℓ,b,ρ)(L, ,b,ρ) separately for each task on its validation split and apply the best configuration on the test split. The full protocol and the selected per-task configurations are given in Table 13 (Appendix J). Observations. From Figure 5, we notice that context tuning is primarily useful for regression. With a single context seed, it reduces mean nMAE from 33.433.4 to 32.532.5, and improves eight of the nine regression tasks. The largest gains occur on rel-trial/site-success (91.091.0 →88.6→ 88.6) and on rel-hm/item-sales (14.014.0 →11.8→ 11.8), the latter by selecting a smaller 512512-cell context. For classification, tuning hurts a single seed on average (72.972.9 →72.4→ 72.4 AUROC), as the validation choice transfers to test on only five of twelve tasks (Table 13 in Appendix J). Moreover, among the selected configurations, three of nine regression tasks (rel-amazon/item-ltv, rel-amazon/user-ltv, rel-hm/item-sales) and four of twelve classification tasks (rel-avito/user-visits, rel-event/user-ignore, rel-event/user-repeat, rel-stack/user-engagement) prefer a context with L<8192L<8192 cells. 6.2 Context ensembling Both the random-walk retrieval and the bounded-width local BFS stages of our context sampler are stochastic, with different random seeds yielding different yet equally viable contexts. Context ensembling simply averages the predictions made from nensn_ens independently sampled contexts. Ensembling reduces prediction noise due to retrieval by averaging over contexts and can be computed in parallel across nensn_ens. We sweep over nens∈1,2,4,8,16n_ens∈\1,2,4,8,16\ using different sampler seeds with a default configuration of L=8192L=8192, ℓ=256 =256, b=32b=32, ρ=1ρ=1 and showcase the benefits. Observations. Figure 5 shows the results. With the per-task-tuned configuration, increasing nensn_ens from 11 to 1616 yields a +1.3+1.3 AUROC point aggregate gain for classification tasks, while at the default context it is slightly lower at +1.1+1.1 AUROC points. Regression tasks show smaller but consistent improvements (0.20.2 nMAE points on average at the default context). Based on the per-task curves in Appendix J, the largest default-context gains accrue to rel-trial/study-outcome with +5.3+5.3, rel-event/user-repeat with +2.6+2.6 and rel-hm/user-churn with an improvement of +2.4+2.4 AUROC points from nens=1n_ens=1 to 1616, while rel-avito/user-clicks degrades by 3.03.0 AUROC points. On regression tasks the gains top out at 0.70.7 nMAE points on rel-event/user-attendance. 7 Ablation Studies Here we isolate the impact of the key components of our recipe. The schema-semantics (Section 7.1) and context-retrieval (Section 7.2) ablations swap a single component of the full RT-J pipeline at inference time, evaluated on a fixed 81928192-row subsampled test set per task. Table 2 summarizes the gain from various components. More pretraining ablations are in Appendix H. Figure 4: Context ensembling at test time, for the default (L=8192L=8192 cell) and per-task-tuned contexts. Ensembling improves both metrics for both configurations. Per-task plots are in Appendix J. Figure 5: Schema-semantics ablation. Shuffling table and column names at evaluation degrades results increasingly with context. Per-task plots are in Appendix J. 7.1 Schema semantics RT-J consumes language model (LM) embeddings of table/column names in addition to cell values (see also the cell embedding design in Appendix F). Figure 5 shows a comparison with and without such semantics, where semantics are removed by shuffling table/column names at evaluation time. Observations. Shuffling table/column names degrades classification at every point of the sweep, by up to 2.12.1 AUROC points, and regression by up to 0.80.8 nMAE points. Notably, the gap widens with more in-context examples, which suggests that schema text provides prior information that the examples do not replace. These observations highlight that RT-J has evidently absorbed world knowledge attached to real table and column names during pretraining on the Join. Figure 6: Context-retrieval ablation with each retriever being swapped into the pretrained RT-J at inference time. Within each metric group, the left panel plots the metric against context size L and the right panel plots it against the mean number of in-context labeled task rows delivered at L=8192L=8192. Our random-walk variant (red) is the strongest retriever on regression at small L and within 0.80.8 nMAE and 0.90.9 AUROC of the best at every L. Per-task plots are in Appendix J. 7.2 Alternatives to random-walk retrieval We compare our random-walk retrieval approach (Section 4) against five alternatives at inference: uniformly random task-row selection, BFS of width 32,256\32,256\ rooted at the target (the original scheme of [23]), and two vector-similarity retrievers that score task rows by RDBLearn features or RT-J embeddings. In particular, the RT-J embedding of a task row is the final hidden state at its masked target cell, computed over a 256256-cell BFS context around the row. Both similarity retrievers rank labeled task rows dated before the target by cosine similarity and use them, in that order, as seeds for the same local BFS as random-walk retrieval. Observations. Figure 6 shows that our random walk approach and similarity search over RDBLearn’s precomputed features are well ahead of the other retrievers and effectively tie at L=8192L=8192 (33.433.4 versus 33.533.5 nMAE and 72.972.9 versus 73.073.0 AUROC). Random walk leads regression at the two smallest contexts (L=256L=256, by 0.70.7 nMAE, and L=512L=512) and again at L=8192L=8192, whereas the similarity retriever is ahead by 0.40.4–0.80.8 nMAE at L=1024L=1024–40964096 and by at most 0.90.9 AUROC. The scatter panels in Figure 6 separate label count from label relevance. RDBLearn-feature similarity packs the most labels into the context (394394) and reaches parity with random walk performance (that uses 223223 labels at L=8192L=8192). However, uniform random selection retrieves a similar count (208208) and trails by 4.14.1 nMAE and 6.16.1 AUROC. RT-J-embedding similarity stays 2.32.3–3.23.2 nMAE and 3.63.6–5.45.4 AUROC behind random walk at every L, yet similarity over RDBLearn’s engineered features keeps improving with L, highlighting the importance of the embedding quality for ranking relevant task rows. The two BFS variants are restricted to their fixed local neighborhoods and come only within 1.61.6–1.81.8 nMAE of random walk with 2.72.7–5.6×5.6× fewer labels (8484 and 4040) respectively. Overall, our random walk retriever sits at the label-efficient end of this trade-off as it does not need per-task feature pipelines that RDBLearn requires, while providing higher-quality contexts for predictions. Table 2: Summary of gains, in nMAE (regression) and AUROC (classification) points. Comparison Reg. Clf. Comparison Reg. Clf. Pretraining data Context retrieval RT-J vs. RT-PluRel +0.6+0.6 +1.5+1.5 random walk vs. width-256 BFS +1.6+1.6 +3.9+3.9 the Join-only vs. PluRel-only +1.8+1.8 +2.8+2.8 random walk vs. uniform random +4.1+4.1 +6.1+6.1 Test-time compute tuned vs. default context +0.8+0.8 −0.5-0.5 16 context ensemble vs. 1 context +0.2+0.2 +1.1+1.1 8 Discussion and Conclusion Our results show that RT-J can make accurate predictions on new datasets with as few as a hundred examples, in a single forward pass. This makes it practical to solve a wide range of predictive tasks without building dedicated feature engineering pipelines, or relying on extensive labeled data. Examples include flagging likely churn or fraud in a small-scale enterprise without a large customer base; predicting outcomes of novel clinical trials with small cohorts; and answering one-off predictive questions from an analyst. RT-J could be exposed directly in the database system, allowing users to request quick ad-hoc forecasts without an expensive labeling step. It could also be offered as a tool for LLM agents to call when a question requires predicting an unknown value, unlike transforming existing ones with SQL. The Join is a first step toward capturing the immense diversity of real-world applications of relational foundation models (RFMs) like RT-J. Beyond synthetic pretraining corpora prevalent for structured data foundation models, the Join offers hundreds of real databases across science, healthcare, finance, sports, etc., with semantically meaningful predictive tasks spanning rare-event, cold-start, heavy-tailed and long-horizon regimes. Conclusion. We developed a pretraining recipe for relational transformers (RTs) [23] with long-context training, random-walk retrieval of high-quality in-context examples, and test-time compute scaling via context tuning and ensembling. With this recipe, we obtain RT-J by first pretraining on PluRel [26] synthetic data and then on the Join, our large-scale real data corpus. RT-J matches strong feature engineering pipelines with as much as 10×10× fewer examples. Experiments highlight the role of real data pretraining, schema semantics and context engineering. Our work moves RFMs closer to practical systems that can solve novel predictive tasks over databases on the fly. Limitations and future work. RT-J has been trained up to 8,1928,192 context cells; future work can explore efficient longer context training, which would allow both for more in-context examples, and larger relational contexts for them. RT-J is unable to solve link prediction and recommendation tasks, sharing this limitation with RT [23]. Our evaluation is limited to RelBench, with only 2121 tasks on 77 databases—inspired by the Join, future work can expand the scope of real-world relational benchmarks. We release (1) the RT-PluRel and RT-J models, (2) the Join data, (3) the LLM agent task curation pipeline, and (4) our pretraining, finetuning and inference code at star-project.stanford.edu/rt-j, hoping to accelerate future research and adoption of RFMs. Acknowledgments and Disclosure of Funding We thank Anvita Gupta, Joe Meyer, Johannes Hoffart, Jonas De Schouwer, Lata Nair, Mahmoud Mohammadi, Marcel Roed, Mark Znidar, Matthias Fey, Michael Bereket, Rok Sosic, Ryan Welch, Valter Hudovernik, Vijay Prakash Dwivedi, Viswa Ganapathy, Yanay Rosen, and Yangyi Shen for discussions, feedback and support. We also gratefully acknowledge the support of NSF under Nos. CCF-1918940 (Expeditions), DMS-2327709 (IHBEM), IIS-2403318 (I); NIH under No. 1U24NS146314-01, Gates Foundation, Stanford Data Applications Initiative, Wu Tsai Neurosciences Institute, Stanford Institute for Human-Centered AI, Chan Zuckerberg Initiative, Amazon, Genentech, SAP, and SCBX. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities. References [1] Rishi Bommasani et al. On the opportunities and risks of foundation models, 2022. URL https://arxiv.org/abs/2108.07258. [2] Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. International Journal of Machine Learning and Cybernetics, pages 1–65, 2024. [3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.c/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. [4] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems, volume 35, pages 23716–23736, 2022. [5] Liane Vogel, Benjamin Hilprecht, and Carsten Binnig. Towards foundation models for relational databases [vision paper]. In NeurIPS 2022 First Table Representation Workshop, 2022. [6] Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, and Jure Leskovec. Position: Relational deep learning - graph representation learning on relational databases. In Forty-first International Conference on Machine Learning, 2024. [7] Tianlang Chen, Charilaos Kanatsoulis, and Jure Leskovec. RelGNN: Composite message passing for relational deep learning. In Forty-second International Conference on Machine Learning, 2025. [8] Vijay Prakash Dwivedi, Sri Jaladi, Yangyi Shen, Federico López, Charilaos I. Kanatsoulis, Rishi Puri, Matthias Fey, and Jure Leskovec. Relational graph transformer, 2025a. URL https://arxiv.org/abs/2505.10960. [9] Jakub Peleška and Gustav Šír. Transformers meet relational databases, 2024. URL https://arxiv.org/abs/2412.05218. [10] Matthias Fey, Vid Kocijan, Federico Lopez, Jan Eric Lenssen, and Jure Leskovec. Kumorfm: A foundation model for in-context learning on relational data, 2025. [11] Valter Hudovernik, Federico López, Vid Kocijan, Akihiro Nitta, Jan Eric Lenssen, Jure Leskovec, and Matthias Fey. KumoRFM-2: Scaling foundation models for relational learning, 2026. URL https://arxiv.org/abs/2604.12596. [12] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794, 2016. [13] Hoang Thanh Lam, Beat Buesser, Hong Min, Tran Ngoc Minh, Martin Wistuba, Udayan Khurana, Gregory Bramble, Theodoros Salonidis, Dakuo Wang, and Horst Samulowitz. Automated data science for relational data. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pages 2689–2692. IEEE, 2021. [14] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84–90, 2022. [15] Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Mihir Manium, Shi Bin Hoo, Magnus Bühler, Anurag Garg, Dominik Safaric, Jake Robertson, Benjamin Jäger, Simone Alessi, Adrian Hayler, Vladyslav Moroshan, Lennart Purucker, Philipp Singer, Alan Arazi, Julien Siems, Jan Hendrik Metzen, Georg Grab, Nick Erickson, Siyuan Guo, Eliott Kalfon, Simon Bing, David Salinas, Clara Cornu, Lilly Charlotte Wehrhahn, Diana Kriuchkova, Kursat Kaya, Lydia Sidhoum, Marie Salmon, Jerry Chen, Madelon Hulsebos, Yann LeCun, Samuel Müller, Bernhard Schölkopf, Sauraj Gambhir, Noah Hollmann, and Frank Hutter. Tabpfn-3: Technical report, 2026. URL https://arxiv.org/abs/2605.13986. [16] Benjamin Jäger, Nick Erickson, Léo Grinsztajn, Felix Birkel, Klemens Flöge, Oscar Key, Kürşat Kaya, Jonas Kübler, Adèle Frankel, Tobias Schröder, Anurag Garg, Jan Hendrik Metzen, David Salinas, Simon Bing, Kristina Collins, Tuana Çelik, Vahid Balazadeh, Lydia Sidhoum, Tomás Pereda, Brendan Roof, Andrej Tschalzev, Siyuan Guo, Philipp Singer, Lennart Purucker, Jake Robertson, Marie Salmon, Philipp Jund, Jerry Chen, Diana Kriuchkova, Arthur Cahu, Eliott Kalfon, Adrian Hayler, Georg Grab, Vitor Monteiro, Lilly Wehrhahn, Dominik Safaric, Clara Cornu, Alan Arazi, Rylee Grace, Simone Alessi, Mihir Manium, Bernhard Schölkopf, Yann LeCun, Madelon Hulsebos, Sauraj Gambhir, Noah Hollmann, and Frank Hutter. Tabpfn-3.5: Technical report, 2026. URL https://arxiv.org/abs/2609.17895. [17] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. In The Eleventh International Conference on Learning Representations, 2023. [18] Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319–326, 2025. [19] Jingang QU, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICL: A tabular foundation model for in-context learning on large data. In Forty-second International Conference on Machine Learning, 2025. [20] Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICLv2: A better, faster, scalable, and open tabular foundation model, 2026. [21] Yanlin Zhang, Linjie Xu, Quan Gan, David Wipf, and Minjie Wang. RDBLearn: Simple in-context prediction over relational databases, 2026. URL https://arxiv.org/abs/2602.18495. [22] Yanbo Wang, Xiyuan Wang, Quan Gan, Minjie Wang, Qibin Yang, David Wipf, and Muhan Zhang. Griffin: Towards a graph-centric relational database foundation model. In Forty-second International Conference on Machine Learning, 2025. [23] Rishabh Ranjan, Valter Hudovernik, Mark Znidar, Charilaos I. Kanatsoulis, Roshan Upendra, Mahmoud Mohammadi, Joe Meyer, Tom Palczewski, Carlos Guestrin, and Jure Leskovec. Relational transformer: Toward zero-shot foundation models for relational data. In The Fourteenth International Conference on Learning Representations, 2026. [24] Justin Gu, Rishabh Ranjan, Charilaos Kanatsoulis, Haiming Tang, Martin Jurkovic, Valter Hudovernik, Mark Znidar, Pranshu Chaturvedi, Parth Shroff, Fengyu Li, et al. Relbench v2: A large-scale benchmark and repository for relational data. arXiv preprint arXiv:2602.12606, 2026. [25] Jan Motl and Oliver Schulte. The ctu prague relational learning repository. arXiv preprint arXiv:1511.03086, 2015. [26] Vignesh Kothapalli, Rishabh Ranjan, Valter Hudovernik, Vijay Prakash Dwivedi, Johannes Hoffart, Carlos Guestrin, and Jure Leskovec. Plurel: Synthetic data unlocks scaling laws for relational foundation models. In Forty-third International Conference on Machine Learning, 2026a. URL https://openreview.net/forum?id=rM9r9qyRSA. [27] Adrian Hayler, Klemens Flöge, Alan Arazi, Rishabh Ranjan, Jure Leskovec, Felix Birkel, Brendan Roof, Anurag Garg, Kristina Collins, Lydia Sidhoum, et al. Advancing open and reproducible relational learning: RelArena-α, TabPFN-Rel and RPI. arXiv preprint arXiv:2608.16319, 2026. [28] Yanbo Wang, Jiaxuan You, Chuan Shi, and Muhan Zhang. Relational in-context learning via synthetic pre-training with structural prior. In Forty-third International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2603.03805. [29] Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, Jiaqi Han, Alejandro Dobles, Matthias Fey, Jan Eric Lenssen, Yiwen Yuan, Zecheng Zhang, et al. Relbench: A benchmark for deep learning on relational databases. Advances in Neural Information Processing Systems, 37:21330–21341, 2024. [30] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36:42330–42357, 2023. [31] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 3911–3921, 2018. [32] Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin SU, ZHAOQING SUO, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows. In The Thirteenth International Conference on Learning Representations, 2025. [33] Minjie Wang, Quan Gan, David Wipf, Zhenkun Cai, Ning Li, Jianheng Tang, Yanlin Zhang, Zizhao Zhang, Zunyao Mao, Yakun Song, et al. 4dbinfer: A 4d benchmarking toolbox for graph-centric predictive modeling on rdbs. Advances in Neural Information Processing Systems, 37:27236–27273, 2024. [34] Chantat Eksombatchai, Pranav Jindal, Jerry Zitao Liu, Yuchen Liu, Rahul Sharma, Charles Sugnet, Mark Ulrich, and Jure Leskovec. Pixie: A system for recommending 3+ billion items to 200+ million users in real-time. In Proceedings of the 2018 world wide web conference, pages 1775–1784, 2018. [35] Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 974–983, 2018. [36] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/. [37] P Izmailov, AG Wilson, D Podoprikhin, D Vetrov, and T Garipov. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pages 876–885, 2018. [38] Rishabh Ranjan, Saurabh Garg, Mrigank Raman, Carlos Guestrin, and Zachary Lipton. Post-hoc reversal: Are we selecting models prematurely? Advances in Neural Information Processing Systems, 37:91460–91491, 2024. [39] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, 2017. [40] Charilaos I Kanatsoulis, Evelyn Choi, Stephanie Jegelka, Jure Leskovec, and Alejandro Ribeiro. Learning efficient positional encodings with graph neural networks. In The Thirteenth International Conference on Learning Representations, 2025. [41] Vijay Prakash Dwivedi, Charilaos Kanatsoulis, Shenyang Huang, and Jure Leskovec. Relational deep learning: Challenges, foundations and next-generation architectures. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 5999–6009, 2025b. [42] Marco Spinaci, Marek Polewczyk, Johannes Hoffart, Markus C. Kohler, Sam Thelin, and Tassilo Klein. Portal: Scalable tabular foundation models via content-specific tokenization, 2024. URL https://arxiv.org/abs/2410.13516. [43] Myung Jun Kim, Leo Grinsztajn, and Gael Varoquaux. Carte: Pretraining and transfer for tabular learning. In International Conference on Machine Learning, pages 23843–23866. PMLR, 2024. [44] Marek Wydmuch, Łukasz Borchmann, and Filip Graliński. Tackling prediction tasks in relational databases with llms, 2024. URL https://arxiv.org/abs/2411.11829. [45] Fang Wu, Vijay Prakash Dwivedi, and Jure Leskovec. Large language models are good relational learners, 2025. URL https://arxiv.org/abs/2506.05725. [46] Jun Xia, Chengshuai Zhao, Bozhen Hu, Zhangyang Gao, Cheng Tan, Yue Liu, Siyuan Li, and Stan Z Li. Mole-bert: Rethinking pre-training graph neural networks for molecules. In The Eleventh International Conference on Learning Representations, 2023. [47] Dominique Beaini, Shenyang Huang, Joao Alex Cunha, Zhiyi Li, Gabriela Moisescu-Pareja, Oleksandr Dymov, Samuel Maddrell-Mander, Callum McLean, Frederik Wenkel, Luis Müller, et al. Towards foundational models for molecular learning on large-scale multi-task datasets. In The Twelfth International Conference on Learning Representations, 2024. [48] Qian Huang, Hongyu Ren, Peng Chen, Gregor Kržmanc, Daniel Zeng, Percy S Liang, and Jure Leskovec. Prodigy: Enabling in-context learning over graphs. Advances in Neural Information Processing Systems, 36:16302–16317, 2023. [49] Jianan Zhao, Zhaocheng Zhu, Mikhail Galkin, Hesham Mostafa, Michael M Bronstein, and Jian Tang. Fully-inductive node classification on arbitrary graphs. In The Thirteenth International Conference on Learning Representations, 2025a. [50] Mikhail Galkin, Xinyu Yuan, Hesham Mostafa, Jian Tang, and Zhaocheng Zhu. Towards foundation models for knowledge graph reasoning. In The Twelfth International Conference on Learning Representations, 2024. [51] Qifang Zhao, Weidong Ren, Tianyu Li, Hong Liu, Xingsheng He, and Xiaoxiao Xu. Graphgpt: Generative pre-trained graph eulerian transformer. In Forty-second International Conference on Machine Learning, 2025b. [52] Maya Bechler-Speicher, Yoel Gottlieb, Andrey Isakov, David Abensur, Ami Tavory, Daniel Haimovich, Ido Guy, and Udi Weinsberg. Billion-scale graph foundation models, 2026. [53] Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for LLM training, 2025. URL https://arxiv.org/abs/2502.16982. [54] Changxin Tian, Jiapeng Wang, Qian Zhao, Kunlong Chen, Jia Liu, Ziqi Liu, Jiaxin Mao, Wayne Xin Zhao, Zhiqiang Zhang, and Jun Zhou. WSM: Decay-free learning rate schedule via checkpoint merging for LLM pre-training, 2025. URL https://arxiv.org/abs/2507.17634. [55] Craig Kapfer, Kurt Stine, Balasubramanian Narasimhan, Christopher Mentzel, and Emmanuel Candes. Marlowe: Stanford’s gpu-based computational instrument, January 2025. URL https://doi.org/10.5281/zenodo.14751899. [56] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019. [57] Mingze Dong, Leda Wang, and Yuval Kluger. Understanding and enhancing mask-based pretraining towards universal representations. In Advances in Neural Information Processing Systems, 2025. [58] Vignesh Kothapalli, Hamed Firooz, and Maziar Sanjabi. Cot-icl lab: A synthetic framework for studying chain-of-thought learning from in-context demonstrations. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14620–14642, 2025. [59] Vignesh Kothapalli, Ata Fatahibaarzi, Hamed Firooz, and Maziar Sanjabi. To think or not to think: The hidden cost of meta-training with excessive cot examples. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15619–15644, 2026b. Appendix A Leaderboard Results RelArena-α. RelArena-α [27] re-runs every submission on the 2121 entity-level forecasting tasks of RelBench v1 under a single API, data state and evaluation protocol, and aggregates per-task results into Elo ratings. It separates models, tuned under a standardized, compute-matched tuning regime, from systems, which follow the same data states and evaluation but bring their own tuning protocol. Figure 7 reproduces the combined model + system leaderboard from Hayler et al. [27]. RT-PluRel, submitted as a system with its own sequential tuning regime, ranks first with an Elo of 18611861, ahead of TabPFN-Rel (API) at 18261826 and of all relational deep learning baselines (GraphSAGE, RelGT, RelGNN) and RDBLearn. RT-J was not part of this release. Figure 7: Combined model + system leaderboard of RelArena-α, reproduced from Hayler et al. [27]: Elo over the 2121 RelBench v1 forecasting tasks (anchored at 10001000 for the global constant predictor). RT-PluRel (hatched) is a system submission with its own tuning protocol; TabPFN-Rel variants in pink, other baselines in blue, constant predictors in grey. RelBench leaderboard. Table 3 lists the per-task and mean test metrics of the RT family on the RelBench leaderboard, both fine-tuned on each task and in-context. In-context, RT-J improves over RT-PluRel on both regression (32.3532.35 vs. 32.9532.95 nMAE) and classification (74.4574.45 vs. 72.9872.98 AUROC). The gains from the Join carry over to fine-tuning, where RT-J again has the best means of the three RT models (27.1127.11 nMAE and 79.0279.02 AUROC, against 27.5727.57 and 78.5378.53 for RT-PluRel and 29.2629.26 and 77.7677.76 for RT). We detail the pipelines used for our submissions below. Fine-tuning pipeline. Each model is fine-tuned separately on each of the 2121 tasks in four stages. (1) A selection run trains on the train split for up to 5050K steps, evaluating every 100100 steps on 10241024 validation rows with a 44-seed context ensemble and stopping after 1010K steps without improvement. (2) A context search scores 9696 context configurations (L,ℓ,b,ρ)(L, ,b,ρ) with L≤1024L≤ 1024 on 40964096 validation rows. (3) A refit trains on train and validation together for the selected number of steps, scaled by the ratio of train-plus-validation to train rows. (4) The SWA weights of the refit predict the full test split under the selected context hyperparameters with an 88-seed context ensemble. RT-PluRel and RT-J train an additive delta on their frozen pretrained weights, whereas RT trains all weights from random initialization. All runs use Muon with learning rate 5×10−45× 10^-4 and weight decay 0.10.1, batches of 256256 contexts of up to 10241024 cells, binary cross-entropy for classification and L1 loss for regression. In-context pipeline. The pretrained checkpoint is frozen, and no RelBench database is seen in pretraining. Per task, we score 120120 context configurations (L,ℓ,b,ρ)(L, ,b,ρ) with L≤8192L≤ 8192 on 40964096 validation rows with a 44-seed context ensemble, then run each of the four best configurations with four context seeds on the full test split and average the 1616 predictions per row. Table 3: RT models on the RelBench leaderboard. Per-task test metrics of RT [23], RT-PluRel and RT-J, fine-tuned on each task or used in-context, as listed on the RelBench leaderboard. All metrics in percent. Best per row within each group in bold; lower is better for nMAE, higher is better for AUROC. Fine-tuned In-context DB Task RT (rand init) RT-PluRel RT-J RT-PluRel RT-J Regression (nMAE %, lower is better) rel-amazon item-ltv 7.507.50 7.267.26 7.197.19 8.658.65 7.817.81 rel-amazon user-ltv 24.1524.15 24.5824.58 24.2924.29 29.6029.60 29.1429.14 rel-avito ad-ctr 38.5338.53 37.8637.86 36.0336.03 41.7141.71 42.0142.01 rel-event user-attendance 32.5232.52 31.6431.64 31.6331.63 43.7843.78 42.6942.69 rel-f1 driver-position 48.7548.75 39.7239.72 37.6737.67 42.4142.41 38.7538.75 rel-hm item-sales 7.757.75 7.537.53 7.577.57 11.4411.44 10.1110.11 rel-stack post-votes 13.7713.77 13.6213.62 12.7912.79 14.7014.70 14.7414.74 rel-trial site-success 78.3678.36 75.1775.17 76.5476.54 87.6587.65 89.6289.62 rel-trial study-adverse 12.0612.06 10.7410.74 10.2810.28 16.5716.57 16.3316.33 Mean 29.2629.26 27.5727.57 27.1127.11 32.9532.95 32.3532.35 Classification (AUROC %, higher is better) rel-amazon item-churn 82.9682.96 83.1983.19 83.1683.16 74.4074.40 78.6178.61 rel-amazon user-churn 71.3071.30 71.3471.34 71.4671.46 66.2466.24 67.2067.20 rel-avito user-clicks 68.5968.59 65.8965.89 66.8166.81 46.2046.20 51.3951.39 rel-avito user-visits 66.2166.21 66.7566.75 66.8966.89 61.4461.44 64.0964.09 rel-event user-ignore 81.4181.41 80.8580.85 85.6185.61 76.7876.78 80.9780.97 rel-event user-repeat 78.9778.97 80.2980.29 79.8479.84 77.6577.65 79.1279.12 rel-f1 driver-dnf 79.9379.93 82.0982.09 82.7782.77 81.8981.89 83.0883.08 rel-f1 driver-top3 83.8483.84 90.0190.01 91.7191.71 89.6989.69 91.0091.00 rel-hm user-churn 70.2470.24 70.6170.61 70.4970.49 65.7765.77 65.6665.66 rel-stack user-badge 89.3389.33 89.2589.25 89.2189.21 83.7883.78 81.1181.11 rel-stack user-engagement 90.7490.74 90.9190.91 90.6590.65 87.8687.86 86.7686.76 rel-trial study-outcome 69.5569.55 71.2171.21 69.5769.57 64.0864.08 64.4364.43 Mean 77.7677.76 78.5378.53 79.0279.02 72.9872.98 74.4574.45 Appendix B Other Relational Foundation Models Beyond the pipelines of Section 5, we looked into comparing RT-J with recent relational foundation models (RFMs), but none could be evaluated under our protocol, in which every method sees the same in-context examples. KumoRFM [10]: closed weights, and API access does not allow controlling the in-context examples, precluding a fair comparison. KumoRFM-2 [11]: closed weights. API access was released only a few weeks before our submission deadline, and no longer works due to the ongoing migration of Kumo to NVIDIA. Griffin [22]: the released checkpoints are incapable of in-context learning, and pretraining Griffin with our recipe is not feasible, as the recipe is engineered around transformer-based architectures, which Griffin is not. RDB-PFN [28]: supports only classification tasks, whereas our evaluation covers both regression and classification. TabPFN-Rel with TabPFN-3 and TabPFN-3.5 [15, 16]: released after submission. Appendix C Related Work Relational deep learning and benchmarks. Relational deep learning (RDL) is a machine learning paradigm that operates directly on relational databases, avoiding the need to manually flatten a database into a single feature table [6]. A common formulation represents a database as a heterogeneous relational entity graph: rows are nodes, tables define node types, and primary–foreign-key links define typed edges [29]. Graph neural networks can then propagate information across related rows and tables, enabling end-to-end prediction over multi-table data [29, 7]. Recent work has improved the efficiency and expressiveness of relational message passing [7], and transformer-based relational architectures have also been proposed [9, 8]. Kanatsoulis et al. [40] introduce an efficient positional encoding framework based on lightweight message-passing operations, providing structural features that have been shown to improve relational deep learning models. Broader surveys and position papers frame these models as part of a larger agenda for relational deep learning [41]. Benchmarking has been central to this progress. RelBench [29] introduced a standardized suite of relational prediction tasks, while 4DBInfer [33] and RelBench v2 [24] expand the scale and diversity of relational benchmarks, including broader task families and database domains. These benchmarks make it possible to compare relational models under realistic multi-table and temporal constraints. Relational foundation models. Recent work has begun to develop foundation models specifically for relational data. These models seek to transfer across schemas and tasks rather than training a separate predictor for every database. KumoRFM and KumoRFM-2 study in-context learning for relational data at scale [10, 11]. Griffin develops a graph-centric relational foundation model for transfer across relational databases [22]. The Relational Transformer (RT) takes a complementary cell-level approach: database cells are treated as tokens, schema semantics are included in the input, and relational attention masks encode primary–foreign-key structure [23]. PluRel [26] enables RT pretraining with synthetic data. It synthesizes multi-table databases from scratch, including schemas, primary–foreign-key links, and feature distributions, enabling controlled studies of synthetic data scaling for relational foundation models. Our work builds directly on RT because its cell-level formulation is well-suited to context-efficient relational prediction. Labeled task rows, unlabeled rows, schema metadata, and linked evidence can all appear in the same context window, making masked cell prediction a natural interface for both pretraining and inference. However, the original RT work primarily establishes a schema-agnostic architecture; it does not develop a scalable pretraining and inference recipe. We therefore use RT as the backbone and study how to make it effective when only a small number of in-context examples are available. Tabular foundation models and relational tabularization. Tabular foundation models aim to bring pretraining and in-context learning to single-table prediction. TabPFN and its successors show that transformers can perform strong in-context prediction when labeled rows from a tabular task are provided in the context [17, 18]. More recent models, including TabICLv2 and other large-scale tabular foundation models, extend this paradigm to larger datasets, richer feature spaces, and broader tabular distributions [19, 20, 42, 43]. These methods provide strong evidence that structured prediction can benefit from foundation-model-style training. Relational databases, however, contain structure that is not present in ordinary single tables. Predictive evidence may lie several joins away from the target row, and the relevant signal may involve temporal histories, linked entities, or multi-hop dependencies. One way to use tabular foundation models in this setting is to first transform the database into a single feature table. RDBLearn follows this strategy by aggregating relational neighborhoods into engineered features and then applying a tabular in-context learner [21]. This approach is effective and provides a strong baseline, but the flattening step changes the input representation: the model no longer reasons directly over cells, schemas, timestamps, primary–foreign-key links, or multi-table structure. Large language models for relational data. Another line of work applies large language models to structured and relational data by serializing tables or database neighborhoods into text [44]. This has the advantage of using powerful pretrained language models without designing a specialized architecture. However, serialization can be inefficient and brittle. Relational structure must be expressed as text, long database contexts can exceed the model context window, and the resulting input format may differ substantially from the data distribution seen during language-model pretraining. Hybrid approaches that combine graph or relational encoders with LLMs can partially address these issues, but they often still depend on task-specific components or limited context construction [45]. Graph foundation models. Several papers have paved the way toward graph foundation models. Mole-BERT [46] pretrains GNNs with masked atom prediction and triplet-based contrastive learning for node- and graph-level drug-discovery tasks. Beaini et al. [47] further show that molecular pretraining benefits from scale, using large multi-task datasets with billions of labels and demonstrating that combining quantum and biological supervision improves low-resource performance. Recent work has extended pretraining graph models beyond molecules. PRODIGY [48] introduces prompt-based graph representations for in-context learning. GraphAny [49] studies zero-shot node classification across graphs with disjoint feature and label spaces, using LinearGNN ensembles and inductive attention inspired by linear least-squares estimation. ULTRA [50] learns transferable relational representations for zero-shot reasoning over unseen knowledge graphs. GraphGPT [51] converts graphs into reversible token sequences using Eulerian paths, enabling scalable generative Transformer pretraining for graph-structured data. Most recently, Bechler-Speicher et al. [52] scale the graph foundation model paradigm to heterogeneous billion-scale graphs, proposing a billion-parameter Transformer architecture and a recipe for link prediction pretraining. Appendix D Additional Details on the Join D.1 Agentic execution Constructing the Join corpus manually would require months of human engineering, particularly for web collection as each source presents different download mechanisms, file formats, schema conventions, parsing failures, and data-quality issues. We therefore used Claude Code agents as the execution layer for corpus construction. The agents identify and inspect candidate sources, implement and debug domain-specific collectors, adapt standardization routines to newly observed edge cases, run the pipeline, inspect logs and intermediate artifacts, and iterate until each stage reaches its completion criteria. The agents were used for engineering and orchestration, while task validation, scoring, filtering, baseline computation, loadability checks, and provenance tracking were performed by deterministic scripts based on human-defined criteria. D.2 Phase 1: Collection Collecting from heterogeneous sources. The first phase builds a raw relational snapshot from a broad set of public data sources. This is difficult to automate with a single scraper because different sources expose data in very different ways: as SQL dumps, SQLite databases, CSV bundles, parquet or JSON exports, archive mirrors, or paginated APIs. We therefore use source-specific collectors. Each collector downloads the data, parses it in its native format, recovers table and key metadata when available, and converts the result into a common parquet-based representation. We also exclude databases that exactly overlap with established benchmark sources in RelBench, RelBench v2 and 4DBInfer, using source identity rather than schema similarity as the criterion. Respecting upstream constraints. The collectors are designed to be reproducible while remaining respectful of the sources. API-based collectors follow the documented rate limits and include the required identifying information. Bulk-download sources are handled at the archive level, and for sources that publish periodic snapshots, the refresh cadence of our corpus is limited by the cadence of the upstream release. Normalizing the collected data. Before any database is passed to later stages, the collector performs basic normalization. For example, (1) primary and foreign keys are converted to integer indices, (2) timestamps and booleans are stored in consistent types, and (3) mixed-type columns are either normalized or rejected. We also store primary-key, foreign-key, and time-column metadata alongside each Parquet table, so that downstream stages do not rely on source-specific assumptions. Table 4 lists the domains included in the corpus. Table 4: Per-domain corpus breakdown across all collected domains, grouped by italic subject-area headers and sorted by descending DB count. DBs: raw count. The License(s) and Ref. cells list one (license, reference) pair per line with rows aligning across the two columns. Domain Source DBs License(s) Ref. Commerce & Online Platforms tech community Stack Exchange / GH Archive / HN 75 C BY 4.0 Open public API C BY-SA 4.0 gharchive.org hn.algolia.com archive.org recommendations Amazon / MovieLens / IMDb / UCI 34 MIT (Amazon ToS) Research-use Research-use Research-use Research-use Public Domain C BY 4.0 github.com data.imdb.com cseweb.ucsd.edu grouplens.org grouplens.org research.gov archive.ics.uci.edu prediction market Manifold / Polymarket 2 ToS-restricted ToS-restricted manifold.markets polymarket.com ecommerce RecSys Challenge 2015 1 C BY-NC-ND 4.0 kaggle.com consumer product Open Food Facts 1 ODbL world.openfoodfacts.org food reviews Food.com (Kaggle) 1 Kaggle ToS kaggle.com real estate Zillow Research 1 Research-use zillow.com Sports, Culture & Media motorsport Wikipedia / Racing-Reference / MotoGP 9 C BY-SA 4.0 C BY-SA 4.0 C BY-SA 4.0 C BY-SA 4.0 Public unofficial API Research-use C BY-SA 4.0 en.wikipedia.org en.wikipedia.org racing-reference.info en.wikipedia.org motogp.com racing-reference.info en.wikipedia.org sports ATP / Cricsheet / Lahman / StatsBomb / OpenFlights / PKDD 6 C BY-NC-SA 4.0 Research-use C0 C BY-SA 3.0 ODbL Research-use github.com relational.fel.cvut.cz cricsheet.org sabr.org openflights.org github.com food SNAP / Open Food Facts / USDA 4 Research-use Research-use ODbL Public Domain snap.stanford.edu snap.stanford.edu world.openfoodfacts.org fdc.nal.usda.gov music FMA / Last.fm / ListenBrainz 3 C BY 4.0 Research-use C0 github.com grouplens.org listenbrainz.org museum Met / MoMA / Tate 3 C0 C0 C0 github.com github.com github.com gaming EVE Online / Lichess / Steam 3 Open public API C0 Research-use developers.eveonline.com database.lichess.org cseweb.ucsd.edu cultural Europeana / Wikidata 2 C0 C0 europeana.eu wikidata.org cultural surveys D-PLACE / Pew 2 C BY 4.0 Public-use data (Pew) d-place.org pewresearch.org podcast Apple Podcasts (Kaggle) 1 C BY-NC-SA 4.0 kaggle.com history Chronicling America 1 Public Domain chroniclingamerica.loc.gov Government, Finance & Energy government us BTS / FDA / FDIC / FEC / HMDA / NHTSA 11 Public Domain usa.gov energy BOEM / Ember / OPSD / PHMSA / Sodir 6 Public Domain C BY 4.0 C BY 4.0 C BY 4.0 Public Domain NLOD data.boem.gov github.com ember-energy.org open-power-system-data.org phmsa.dot.gov norge.no international Eurostat / GDELT / UNHCR / World Bank 4 Eurostat re-use Open (attribution) C BY 4.0 C BY 4.0 ec.europa.eu gdeltproject.org unhcr.org datacatalog.worldbank.org finance CFPB / SEC EDGAR 2 Public Domain Public Domain consumerfinance.gov sec.gov politics Our World in Data / V-Dem 2 C BY 4.0 C BY 4.0 ourworldindata.org v-dem.net eia forms EIA Form-860 / 923 2 Public Domain eia.gov civic NYC OpenData (311) 1 Public Domain opendata.cityofnewyork.us energy opsd Open Power System Data 1 C BY 4.0 open-power-system-data.org crypto CoinGecko 1 API ToS (free tier) coingecko.com economics World Bank WDI 1 C BY 4.0 datacatalog.worldbank.org energy infrastructure GeoNuclearData 1 ODbL github.com labor ILOSTAT 1 C BY 4.0 ilostat.ilo.org legal OpenSanctions 1 C BY-NC 4.0 opensanctions.org patent USPTO / PatentsView 1 C BY 4.0 data.uspto.gov uk NHS prescriptions (OGL) 1 OGL v3.0 nationalarchives.gov.uk Health & Public Services clinical ChEMBL / openFDA 4 C BY-SA 3.0 Public Domain chembl.gitbook.io fda.gov biology ClinVar / GBIF / PubChem / Reactome 4 Public Domain C BY 4.0 Public Domain C0 ncbi.nlm.nih.gov gbif.org ncbi.nlm.nih.gov reactome.org healthcare quality CMS Provider Data 2 Public Domain cms.gov education OULAD (Open University) 1 C BY 4.0 research.stem.open.ac.uk public health WHO GHO 1 Open access who.int transport GTFS feeds 1 Per-agency mobilitydata.org agriculture USDA NASS 1 Public Domain quickstats.nass.usda.gov biology advanced Unpaywall 1 C0 unpaywall.org disaster OpenFEMA 1 Public Domain fema.gov skills O*NET 1 C BY 4.0 onetcenter.org Scientific & Reference text2sql Spider / BIRD / Spider 2.0 / LiveSQL 259 C BY-SA 4.0 Public access (Oracle) MIT C BY-SA 4.0 bird-bench.github.io livesql.oracle.com github.com yale-lily.github.io ctu CTU Prague Repository 114 Research-use MS sample DB MIT MySQL sample DB Research-use Research-use C BY-SA 4.0 MS sample DB MS sample DB BSD TPC license TPC license TPC license TPC license relational.fel.cvut.cz microsoft.com github.com mysqltutorial.org data.imdb.com grouplens.org sabr.org microsoft.com microsoft.com dev.mysql.com tpc.org tpc.org tpc.org tpc.org temporal network SNAP / KONECT 12 BSD C BY 4.0 snap.stanford.edu zenodo.org network SNAP / KONECT (raw) 10 BSD snap.stanford.edu academic OpenAlex / DBLP / CORDIS / NIH / Citi Bike 7 Free use C BY 4.0 C BY 4.0 ODC-BY Public Domain C0 citibikenyc.com europa.eu europa.eu dblp.org reporter.nih.gov openalex.org geospatial SNAP / Foursquare / GeoNames / OSM / Overture 6 BSD Academic-use only C BY 4.0 BSD ODbL CDLA-Permissive snap.stanford.edu sites.google.com geonames.org snap.stanford.edu openstreetmap.org overturemaps.org graph learning OGB / FB15k / WN18R / YAGO 5 MS research license ODC-BY WordNet license C BY 3.0 microsoft.com github.com wordnet.princeton.edu yago-knowledge.org fivethirtyeight FiveThirtyEight datasets 4 C BY 4.0 github.com ecology iNaturalist / PBDB / RAM Legacy 3 Various C BY 4.0 C BY 4.0 inaturalist.org paleobiodb.org ramlegacy.org linguistics Glottolog / OpenSubtitles / WALS 3 C BY 4.0 Restricted C BY 4.0 glottolog.org opus.nlpl.eu wals.info security NVD / Exploit-DB / MITRE ATT&CK 3 Public Domain GPL 2.0 Apache 2.0 nvd.nist.gov exploit-db.com github.com astronomy NASA Exoplanet Archive / SDSS 2 Public Domain Free use ipac.caltech.edu sdss.org geonames GeoNames 1 C BY 4.0 geonames.org forecasting Good Judgment Project 1 C0 dataverse.harvard.edu hydrology USGS Water Data 1 Public Domain waterdata.usgs.gov biodiversity GBIF 1 C BY 4.0 gbif.org environment EPA AQS 1 Public Domain epa.gov network infra RIPE / RouteViews 1 Open data ripe.net ocean NOAA NDBC 1 Public Domain ndbc.noaa.gov storm NOAA Storm Events 1 Public Domain ncei.noaa.gov threat intel URLhaus (abuse.ch) 1 abuse.ch ToS urlhaus.abuse.ch D.3 Phase 2: Standardization Creating a common representation. After collection, each database is standardized into a uniform relational format. The purpose of this stage is to make heterogeneous sources loadable and comparable, without altering their semantic content. Each fix is implemented as an idempotent detector–transformer pass over the collected parquet files: a pass first checks whether the target invariant already holds, and only rewrites the data when needed. Recovering relational structure. Many public datasets contain useful structure that is implicit rather than declared. We therefore recover primary keys, foreign keys, time columns, boolean fields, and numeric columns when they can be inferred reliably. These repairs are conservative: candidate keys must satisfy uniqueness checks, foreign keys must pass containment checks, and identifier-like fields such as zip codes, ISBNs, phone numbers, and barcodes are not coerced into numeric values. Normalizing common type issues. The standardization pass resolves common inconsistencies across sources, such as booleans stored as strings, dates stored in mixed textual formats, year columns stored as integers, and numeric values stored as strings. Ambiguous cases are left unchanged or flagged rather than silently modified. This keeps the downstream interface simple while avoiding brittle source-specific assumptions. Controlling scale. For databases that are too large for repeated downstream processing, we apply deterministic entity-axis subsampling: entities are sampled with a fixed seed, all associated activity rows are retained, and keys are reindexed to preserve relational consistency. Preserving auditability. All standardization fixes are performed by mechanical scripts, and each transformation is governed by explicit rules. As a result, downstream task generation and evaluation operate on a consistent representation, while every repair remains reproducible and inspectable. D.4 Phase 3: Task proposal and validation Proposing tasks from schema context. For each eligible database, the proposer receives a structured summary of the database, including its domain, tables, row counts, key relationships, time columns, column types, basic column statistics, and a small number of example rows from activity tables. This context is wrapped in a fixed prompt that specifies the allowed task templates, the required JSON format, and the validation rules. The proposer also receives the names of previously generated tasks for that database, which helps reduce duplicates across passes. Constraining the proposal space. Task proposals are restricted to a registered set of templates, listed in Table 5. This keeps the generated tasks interpretable and mechanically checkable, while still allowing broad coverage across domains. Proposals that refer to unknown templates are rejected before further processing. Validating mechanically. The LLM proposes candidate tasks, but acceptance is decided by a deterministic validator. The validator checks that referenced tables and columns exist, entity columns are valid foreign keys, numeric fields are used where required, generated target names do not collide with existing columns, task names follow the required format, and binary thresholds avoid trivially imbalanced labels. Rejected proposals are logged with their failure reason and are not written to the released task set. Improving diversity across passes. Each database is processed through multiple proposal passes. Early passes tend to recover natural or obvious tasks for the schema, while later passes explicitly target under-represented templates and harder prediction settings, such as rare positives, cold-start behavior, heavy-tailed regression, ranking, and derived-ratio tasks. This multi-pass strategy increases task diversity while preserving the same mechanical validation standard for every accepted proposal. Keeping generation auditable. The proposal stage uses an LLM to suggest candidate tasks, but the final task set is determined by explicit templates and deterministic checks. This separation allows the pipeline to benefit from the LLM’s ability to identify plausible prediction problems from heterogeneous schemas, while keeping task inclusion reproducible, inspectable, and independent of unverified free-form judgments. ⬇ Database: civic/nyc_311 Total: 4 tables, 501223 rows Entity tables: [’agencies’, ’locations’, ’complaint_types’] Schema: agencies (15 rows pkey=agency_id): columns: agency(string), agency_name(string), agency_id(Int64, range=[0..14]) locations (270 rows pkey=location_id): columns: borough(string), incident_zip(Int64, range=[8854..92108]), location_id(Int64, range=[0..269]) complaints (500000 rows pkey=unique_key time_col=created_date fks=’complaint_type_id’: ’complaint_types’, ’agency_id’: ’agencies’, ’location_id’: ’locations’): columns: unique_key(Int64), complaint_type_id(Int64), agency_id(Int64), location_id(Int64), created_date(datetime64[us, UTC]), closed_date(datetime64[us, UTC]), resolution_hours(Float64, mean=49.6, range=[0..1105]), status(string), community_board(string), open_data_channel_type(string) complaint_types (938 rows pkey=complaint_type_id): columns: complaint_type(string), descriptor(string), complaint_type_id(Int64, range=[0..937]) Activity tables (have time_col + FK to entity): complaints: time_col=created_date, entity_fks=’complaint_type_id’: ’complaint_types’, ’agency_id’: ’agencies’, ’location_id’: ’locations’, 500000 rows sample: [’unique_key’: 0, ’agency_id’: 0, ’created_date’: ’2026-04-14 02:33:29+00:00’, ’resolution_hours’: None, ’status’: ’Open’, ’community_board’: ’15 BROOKLYN’, ..., ...] Existing tasks for this DB (do NOT duplicate these): predict-agency-avg-resolution-hours, predict-location-max-resolution-hours, predict-agency-slow-resolution, predict-complaint-type-active, predict-agency-resolution-trend, ... and 24 more TASK: Propose up to 15 high-quality forecasting tasks that a domain expert would find meaningful. Each task predicts a future value for an entity over a time window. PRIORITY: Our existing task suite is heavy on aggregate_numeric, threshold_binary, existence_binary, count_events, churn_activity. Use at least 40% of your proposals from these under-represented patterns (where applicable): time_until_next (REG), streak_length (REG), rank_change (CLF), rank_regression (REG), peer_outperform (CLF), new_category_adoption (CLF), will_repeat_last (CLF), trend_reversal (CLF), volatility_numeric (REG), concentration (REG), derived_ratio (REG). If the schema supports a ‘group_col‘ (race/week/country) -> propose rank_change + rank_regression. If there’s a natural ‘peer_col‘ (category/country/segment) -> propose peer_outperform. If there’s a ‘category_col‘ (product/type/status) -> propose new_category_adoption + will_repeat_last. ALLOWED PATTERNS: - aggregate_numeric (REG): mean/sum/max of numeric col next window. Needs ‘source_col‘, ‘aggregation‘ in [mean, sum, max, min]. - threshold_binary (CLF): aggregate exceeds threshold. Needs ‘source_col‘, ‘aggregation‘, ‘threshold‘, ‘compare‘ in [gt, gte, lt, lte]. - existence_binary (CLF): entity appears in event_table next window. Needs ‘event_table‘, ‘event_entity_col‘, ‘event_time_col‘. ... [more templates omitted] - heavy_tail_regression (REG, HARD TIER): log1p(agg(source_col)) per entity next window. Entity pool = ALL historical entities. Matches rel-amazon/item-ltv, rel-hm/item-sales. HARD REQUIREMENTS (validator rejects violations): 1. entity_col MUST be a foreign key in the activity_table. 2. source_col / numerator_col / denominator_col MUST be numeric columns in the activity_table. 3. target_col must be short and not collide with an existing column. 4. task_name must be kebab-case starting with "predict-". 5. For derived_ratio: pick MEANINGFUL pairs (batting average, completion rate, margins). 6. For threshold_binary: thresholds near the median of the column. 7. Prefer real-world interpretability over mechanical column selection. Return ONLY a JSON array of task objects: [ "task_name": "predict-...", "task_type": "REGRESSION" | "BINARY_CLASSIFICATION", "pattern": "aggregate_numeric" | ..., "entity_table": "...", "entity_col": "...", "activity_table": "...", "time_col": "...", "target_col": "...", "target_description": "...", ... template-specific fields ..., "skip_cols": [], "tables_used": [...], "timedelta_multiplier": 1 ] Figure 8: Full proposer prompt for civic/nyc_311 (one representative database). The schema, FK map, sample rows, and existing-task list are populated per database; the ALLOWED PATTERNS, HARD REQUIREMENTS, and JSON contract blocks are constant across databases (the ALLOWED PATTERNS block in particular is abbreviated here—the full version enumerates all 25 templates with their required fields). ⬇ "task_name": "predict-agency-avg-resolution-hours", "task_type": "REGRESSION", "pattern": "aggregate_numeric", "entity_table": "agencies", "entity_col": "agency_id", "activity_table": "complaints", "time_col": "created_date", "target_col": "avg_resolution_hours", "target_description": "Predict the mean resolution time (hours) for complaints handled by an agency in the next window.", "source_col": "resolution_hours", "aggregation": "mean", "tables_used": ["agencies", "complaints"], "timedelta_multiplier": 1 Figure 9: First accepted proposal for civic/nyc_311, returned by the Claude Opus proposer in response to the prompt of Figure 8. The proposal passes the mechanical validator (the entity_col is a declared FK in the activity_table, source_col is a numeric column of the same table, target_col does not collide with any existing column, and the pattern is registered) and is forwarded to the generator for instantiation. Table 5: The 25 task-template families instantiated by the corpus, organized under italic signal-type headers. Each row gives the prediction target as the model sees it, paired with one concrete domain example. r: regression; c: binary classification. Template Type Prediction target (example) Aggregation—volume / intensity of activity aggregate_numeric r mean / sum of a numeric column over the next window (e.g. a player’s home runs next season) count_events r number of events for the entity in the next window (e.g. orders per customer next month) distinct_count r distinct values of a column the entity touches (e.g. distinct co-authors next year) concentration r entity’s share of total activity (e.g. seller’s share of category sales) Threshold / existence—continuous signal at a decision boundary threshold_binary c whether an aggregate exceeds a threshold (e.g. revenue >>$1M next quarter) existence_binary c whether the entity appears in the event table next window (e.g. user posts at least once) top_k_binary c whether the entity is in the top K%K\% on a metric (e.g. artist enters top 1% of plays) Trend / change—direction and magnitude of change trend_binary c whether a metric rises relative to the current window (e.g. weekly volume rises) trend_reversal c whether the direction of change flips between windows (e.g. sales reverse from rising to falling) change_numeric r numeric delta from current to next window (e.g. change in monthly energy consumption) volatility_numeric r standard deviation of a metric in the next window (e.g. next-week price volatility) Behavior / churn—entity lifecycle churn_activity c entity has no activity in the next window (e.g. subscriber cancels next month) return_binary c dormant entity reactivates next window (e.g. dormant Stack Exchange user posts again) time_until_next r days until the entity’s next event (e.g. days until a player’s next ATP match) will_repeat_last c next event’s category equals the most-recent one (e.g. customer re-buys the same product) first_event_binary c entity experiences a first occurrence next window (e.g. first purchase by a new account) Ranking—relative position within a peer group rank_change c entity’s rank within its group improves by ≥K≥ K (e.g. driver climbs ≥ 5 places) rank_regression r numeric rank within the group (1=1= best) (e.g. predict the driver’s next-race finish position) peer_outperform c entity beats the mean of its peer group (e.g. artist beats peer-group median plays) Adoption / novelty—first-time events new_category_adoption c entity touches a category value it never has before (e.g. first purchase from a new product category) derived_ratio r ratio of two numeric columns over the next window (e.g. new-to-returning customer ratio) Streak—resilience as a single metric streak_length r longest run of consecutive active sub-periods (e.g. consecutive seasons a player remains active) Tail-event (hard-tier)—rare and extreme regimes cold_start_binary c newly-appearing entity has repeat activity in its first window (e.g. new user posts within 24h of joining) rare_positive_binary c rare event (≤10%≤\!10\% positive rate) occurs for the entity (e.g. rare adverse-quality hospital event) heavy_tail_regression r log1 _1p of an aggregate; preserves heavy upper tail (e.g. log1 _1p of next-month spend, large spenders preserved) D.5 Phase 4: Baselines and quality filtering Measuring basic task signal. After task generation, each task is evaluated with a small set of baselines. We first compute two cheap historical baselines: an entity-mean predictor, which uses each entity’s average training label, and a last-value predictor, which uses the entity’s most recent observed label when available. These baselines are inexpensive and provide a simple check for whether the task contains recoverable temporal or entity-level signal. Training a lightweight ML baseline. We also train a fixed XGBoost baseline for each task. For classification tasks we use XGBClassifier, and for regression tasks we use XGBRegressor, with n_estimators=100, max_depth=6, learning_rate=0.1, tree_method="hist", and n_jobs=2. Features are simple window aggregations over the activity table: for the 20 most populous numeric columns, we compute count, mean, sum, max, and min within a lookback window of length 5×timedelta5× timedelta ending at the prediction time. To keep the pass scalable, training is capped at 5,0005,000 rows per task and evaluation at 2,0002,000 rows per task. This baseline is not intended to be a strong relational model and is only intended to provide a consistent and reproducible reference point for filtering tasks across domains. Filtering and difficulty assignment. The quality filter removes tasks that are unlikely to be useful for downstream evaluation, such as tasks that are nearly trivial, exhibit severe distribution drift, or fail during baseline fitting. Tasks with non-trivial baseline signal are retained, and we also retain non-trivial tasks for which XGBoost does not recover signal, since stronger relational models may still exploit structure missed by the lightweight baseline. Retained tasks are assigned a coarse difficulty level from their XGBoost score: easy, medium, or hard. Preparing the release format. After loadability validation, each task is converted into the release format. The train, validation, and test splits are concatenated in chronological order into a single time-ordered training file, while preserving the schema metadata required by downstream loaders. D.6 Asset licenses, Citations, and Compliance Compliance practice. Every contributing source is either (a) a bulk archive that the upstream provider publishes for download under an explicit license, or (b) a public API whose documented terms permit programmatic research use. We do not crawl arbitrary websites, do not bypass paywalls, do not ignore robots.txt, and do not scrape sources whose terms of service forbid automation. Released-corpus license. We intend to release the corpus under C-BY-SA 4.0. Per-database metadata records (i) the upstream source identifier, (i) the upstream license string verbatim, and (i) any usage restrictions inherited from the source (e.g. non-commercial-only for ATP-Tennis, StatsBomb, Yelp, FMA, and ListenBrainz; research-use-only for the CTU repository). Downstream users redistributing derivative works must honor those per-source restrictions in addition to the aggregated C-BY-SA 4.0 license on the corpus assembly. PII. The corpus contains no personally-identifying information beyond what each upstream source has already published under its own license. Where source data uses pseudonymized identifiers (e.g. Stack Exchange user IDs, Lahman player IDs), we preserve those identifiers verbatim and do not attempt to de-anonymize or join across sources. D.7 Case Studies To illustrate what a forecasting task in the Join represents and the diversity of the corpus, we present three case studies from distinct real-world domains. In every case, the prediction target is a quantity monitored in the corresponding domain and instantiates a different label family among the twenty-five task patterns defined in the corpus. Forecasting an account’s peak balance (join-berka) Our first example predicts a continuous quantity from a transaction stream. The Berka dataset from the PKDD’99 Challenge is a complete relational snapshot of a Czech retail bank, including its accounts, the clients who hold them, the permanent orders and loans they carry, and a central ledger of just over a million individual transactions, each stamped with the resulting running balance. Its eight tables form a multi-hop schema (Figure 10), where accounts are linked to its clients through disp, and both accounts and clients are linked to the district in which they are registered. As peak balance is a quantity that retail banks monitor directly, we create the account-max-balance regression task that, for each account and seed time, predicts the highest balance reached during the following window of roughly 17.517.5 days. accounts † 4,500 district 77 disp 5,369 orders 6,471 transactions † 1,056,320 clients 5,369 loans † 682 cards † 892 account-max-balance † 230,011 Figure 10: Schema of join-berka: every table, its row count, and the foreign key links. Arrows point from a foreign key to the table it references; † marks a table carrying a time column. The dashed box is the task table derived from the database. Forecasting new-user retention (join-brightkite) A cold-start task evaluates an entity at its first appearance. This setting is of independent interest because it isolates a model’s ability to generalise from relational context rather than from the entity’s own past. To showcase such a task in the Join, we consider Brightkite, a location-based social network in which users check in at venues and form friendships. The newuser-will-return task considers each user in the window of their first-ever check-in and predicts whether they will check in again within the next two windows, each roughly 6.76.7 days. Because the user’s own label history is unavailable, a model cannot exploit autocorrelation and must instead generalise from side information, including the venues visited and the user’s position in the friendship graph. checkins † 4,747,281 users 58,228 friendships 428,156 venues 772,966 newuser-will-return † 49,601 user_auser_b Figure 11: Schema of join-brightkite. Here friendships and users tables are joined by more than one foreign key, so the edges are drawn separately and labelled with the referencing column. Forecasting who beats their cohort (join-oulad) We consider the Open University Learning Analytics Dataset (OULAD), a relational record of a large online university that links each student to their virtual-learning-environment clickstream and assessment scores. The student-score-beats-cohort task predicts whether a student’s mean assessment score in the next window (of roughly 7.27.2 days) will exceed the mean score of their presentation cohort over the same window. The comparison threshold is therefore the cohort average rather than a fixed constant, and must be computed before the label can be assigned. Because the label is defined relative to a peer aggregate that changes over time, a model cannot score each student in isolation. Instead, it must compute the cohort context at each step and track the student’s trajectory across windows. student_registration † 32,593 courses 22 vle_resources 6,364 students 32,593 student_vle † 10,655,280 assessments 206 student_assessment † 173,912 student-score-beats-cohort † 43,872 Figure 12: Schema of join-oulad. D.8 Manual Audit of Generated Tasks The deterministic checks of Section 3.3 ensure that tasks are valid against the schema, but not that they are meaningful. We therefore manually reviewed the LLM task descriptions and label-generation SQL, and found the following. Targets are semantically named quantities. In join-berka, tasks ignore the meaningless a2–a16 columns of district and instead predict max_balance, goes_negative, days_to_next_txn, etc. Label SQL uses domain concepts, not arbitrary aggregations. In join-oulad, student-score-beats-cohort compares a student to their cohort’s contemporaneous mean. In join-citi-bike, station-time-until-next-trip predicts station idle time, which operators need for inventory planning. Tasks match real practitioner decisions. account-max-balance (join-berka) mirrors overdraft-exposure and credit-limit review; newuser-will-return (join-brightkite) is a standard onboarding-retention metric; movie-new-tag-adoption (join-movielens) tracks metadata coverage. A small number of tasks (about 100100) did aggregate non-semantic columns, e.g., student-feature1-to-feature0-ratio in join-act_mooc. Appendix E RelBench Forecasting Tasks Binary classification tasks. The binary classification tasks in RelBench ask whether a future event or condition will occur for a given entity. They cover churn, future engagement, ad-click behavior, race outcomes, clinical outcomes, and event attendance behavior. See Table 6 for task descriptions. Table 6: Binary classification forecasting tasks in RelBench. Dataset Task Entity Prediction target rel-amazon user-churn User Whether the user does not review any product in the next 3 months. rel-amazon item-churn Item Whether the item receives no reviews in the next 3 months. rel-avito user-visits User Whether the user visits more than one ad in the next 4 days. rel-avito user-clicks User Whether the user clicks more than one ad in the next 4 days. rel-f1 driver-dnf Driver Whether the driver does not finish a race in the next month. rel-f1 driver-top3 Driver Whether the driver qualifies in the top 3 for a race in the next month. rel-hm user-churn User Whether the customer makes no transactions in the next week. rel-stack user-engagement User Whether the user makes any votes, posts, or comments in the next 3 months. rel-stack user-badge User Whether the user receives a new badge in the next 3 months. rel-trial study-outcome Study Whether the trial achieves its primary outcome in the next year. rel-event user-repeat User Whether a previously active user attends another event in the next 7 days. rel-event user-ignore User Whether the user ignores more than two event invitations in the next 7 days. Regression tasks. The regression tasks ask for a continuous or count-valued future outcome. They cover lifetime value, item sales, click-through rate, future votes, race position, adverse-event counts, site success, and event attendance. See Table 7 for task descriptions. Table 7: Regression forecasting tasks in RelBench. Dataset Task Entity Prediction target rel-amazon user-ltv User Value of the products the user buys and reviews in the next 3 months. rel-amazon item-ltv Item Value of the purchases and reviews the item receives in the next 3 months. rel-avito ad-ctr Ad Click-through rate for the ad over the next 4 days. rel-f1 driver-position Driver Average finishing position of the driver across races in the next 2 months. rel-hm item-sales Item Total sales for the article in the next week. rel-stack post-votes Post Number of votes the post receives in the next 3 months. rel-trial study-adverse Study Number of affected patients with severe adverse events in the next year. rel-trial site-success Site Success rate of a trial site in the next year. rel-event user-attendance User Number of events the user responds yes or maybe to in the next 7 days. E.1 Hard Regimes in RelBench the Join places special emphasis on rare-event, cold-start, heavy-tailed and long-horizon tasks. RelBench is too small to serve as a pretraining corpus, but its tasks already cover these regimes (Table 8). Table 8: RelBench tasks in each hard regime. A task can belong to several regimes. Regime RelBench tasks Count rare event rel-avito/user-clicks, rel-stack/user-badge, rel-stack/user-engagement 3 cold start rel-trial/study-outcome, rel-trial/study-adverse, rel-avito/user-clicks 3 heavy-tailed target rel-amazon/item-ltv, rel-avito/ad-ctr 2 long horizon rel-amazon/user-churn, rel-amazon/item-churn, rel-event/user-repeat, rel-hm/user-churn 4 Rare events. Three RelBench tasks sit at or below 5%5\% positive: rel-avito/user-clicks (3.9%3.9\%), rel-stack/user-badge (4.8%4.8\%), and rel-stack/user-engagement (5.0%5.0\%), comparable to the rare end of the Join. Cold start. rel-trial/study-outcome and rel-trial/study-adverse are 100%100\% cold start at test time: they key on nct_id, each trial contributes one row, so no test trial appears in training. This is the most extreme cold-start setting in the suite, beyond the median of the Join’s cold-start tasks. rel-f1/driver-position (79%79\%), rel-f1/driver-dnf (71%71\%), and rel-avito/user-clicks (61%61\%) follow. Heavy tails. rel-amazon/item-ltv has target skew 133133 and kurtosis 29,07929,079; rel-trial/study-adverse has skew 3939; rel-hm/item-sales, rel-stack/post-votes, and rel-amazon/user-ltv sit near skew 2121. About 140140 tasks in the Join are tail-dominated (the top 1%1\% of values hold more than 30%30\% of the mass); the heavy-tailed RelBench tasks are extreme versions of these. Appendix F Relational Transformer Architecture The RT-J model comprises 1212 transformer blocks with hidden dimension d=512d=512, H=8H=8 attention heads of dimension dh=64d_h=64, and feed-forward dimension dff=2048d_f=2048. Each block applies column, feature, and neighbor attention sequentially, followed by a gated feed-forward network. The resulting model contains 8686M trainable parameters, excluding the frozen text encoder. Cell embeddings. Each cell i is associated with a value viv_i, semantic type τi _i, column cic_i, table tit_i, and row rir_i. Boolean values are cast to 0,1\0,1\ and treated as numeric; numeric values are standardized per column, while datetime values use a shared database-wide normalization. Text values and schema names are embedded using the frozen all-MiniLM-L12-v2 encoder ℰ:Text→ℝ384E:Text ^384. The additional schema-based textual information for each cell is given by: i _i =ℰ(ci‖ of ‖ti), =E (c_i\,\|\, of \,\|\,t_i ), (2) where ∥\| denotes string concatenation. For example: "age of user". Let iu_i denote the normalized scalar or text embedding representing the cell value. The initial token embedding is ischema ^schema_i =RMSNorms(Wsi+s), =RMSNorm_s (W_ss_i+b_s ), (3) ivalue ^value_i =τi,if cell i is masked,RMSNormτi(Wτii+τi),otherwise, = casesm_ _i,&if cell i is masked,\\ RMSNorm_ _i (W_ _iu_i+b_ _i ),&otherwise, cases (4) i(0) ^(0)_i =ischema+ivalue∈ℝ512. =e^schema_i+e^value_i ^512. (5) Value encoders and learned mask embeddings τim_ _i are shared across columns of the same semantic type. Primary and foreign keys specify connectivity rather than contributing value embeddings. No positional or learned row-ID embeddings are used in the model. Relational attention. Each of the 1212 transformer blocks sequentially applies column, feature, and neighbor attention, with 88 heads per module. Let (r)P(r) denote the parent rows referenced by the foreign keys of row r. For non-padding cells, the masks are Mijcol M^col_ij =[ci=cj∧ti=tj], =I\! [c_i=c_j t_i=t_j ], (6) Mijfeat M^feat_ij =[ri=rj∨rj∈(ri)], =I\! [r_i=r_j r_j (r_i) ], (7) Mijnbr M^nbr_ij =[ri∈(rj)]. =I\! [r_i (r_j) ]. (8) These modules respectively exchange information within columns, within rows and their referenced parents, and with child rows referencing the query row. Padding cells are excluded from all attention masks. Each module receives pre-normalized inputs i=RMSNormm(i)z_i=RMSNorm_m(x_i), with a separate RMSNorm per module m. For head h, bias-free projections and QK normalization [26] give: ^ih q_ih =RMSNormQ(WQ,hi), =RMSNorm_Q (W_Q,hz_i ), (9) ^jh k_jh =RMSNormK(WK,hj), =RMSNorm_K (W_K,hz_j ), (10) jh _jh =WV,hj. =W_V,hz_j. (11) Queries and keys are RMS-normalized along the head dimension, with separate learned feature-wise gains shared across heads. Normalized queries are additionally multiplied by a learned head-specific scalar and the logarithm of the valid-key count. For a module with mask M, the attention logits are ki k_i =∑jMij, = _jM_ij, (12) ℓijh _ijh =αhlog(max(ki,1))dh^ih⊤^jh,Mij=1,−∞,Mij=0, = cases _h \! ( (k_i,1) )d_h q_ih k_jh,&M_ij=1,\\ -∞,&M_ij=0, cases (13) where αh _h is a learned head-specific scale. For queries with at least one accessible key, aijh a_ijh =softmaxj(ℓijh), =softmax_j( _ijh), (14) i _i =Concath=18(∑jaijhjh), =Concat_h=1^8 ( _ja_ijhv_jh ), (15) i _i =2σ(WGi), =2σ\! (W_Gz_i ), (16) i _i ←i+WO(i⊙i). _i+W_O (g_i _i ). (17) Queries with no accessible keys receive a zero attention contribution. Each attention module has independent parameters. After the three attention modules, a pre-normalized residual SwiGLU network completes the block: i _i =RMSNormffn(i), =RMSNorm_ffn(x_i), (18) i _i ←i+W2[SiLU(W1i)⊙W3i], _i+W_2 [SiLU(W_1z_i) W_3z_i ], (19) with intermediate dimension 20482048. All attention and feed-forward projections are bias-free, and all RMSNorm operations use ϵ=10−6ε=10^-6. Prediction heads. A final RMSNorm and type-specific affine decoders produce i _i =RMSNormout(i(12)), =RMSNorm_out (x^(12)_i ), (20) ^i u_i =Dτii+τiout. =D_ _ih_i+b^out_ _i. (21) Numeric and datetime heads predict normalized scalars, and the text head predicts the 384384-dimensional text embedding. All targets, including binary classification labels, are regressed with the Huber loss (δ=1δ=1, averaged over output dimensions) as ℓτi _ _i. For B contexts with target sets bT_b, the objective is ℒ =1B∑b=1B1max(|b|,1)∑i∈bℓτi(^i,i). = 1B _b=1^B 1 (|T_b|,1) _i _b _ _i ( u_i,u_i ). (22) All prediction-head weights and biases are initialized to zero. The complete model contains 85,562,53085,562,530 trainable parameters, excluding the frozen text encoder. Appendix G Pretraining Details In all our experiments we use an 8686M-parameter RT model with 12 layers, 512 hidden size and 8 attention heads, incorporating the improvements of [26] (such as QK-Norm) over the original architecture of [23]. We sample context size L from ℒ=512,1024,2048,4096,8192L=\512,1024,2048,4096,8192\, the local context size ℓ from ℒℓ=128,256,512,1024,2048,4096,8192L_ =\128,256,512,1024,2048,4096,8192\, the BFS width-bound b from ℬ=8,16,32,64,128,256B=\8,16,32,64,128,256\, and the recency parameter ρ from ℛ=0,1R=\0,1\. The random-walk retriever uses W=10,000W=10,000 walks of length K=20K=20, and only the target cell is masked (pmax=0p_ =0). We pretrain in two phases: first on the synthetic PluRel corpus [26] (86,21186,211 tasks over 1,9001,900 databases), then continuing from that checkpoint on a fixed mixture of 13,24313,243 the Join tasks drawn from the 523523 databases whose preprocessed on-disk footprint stays under 55 GB. Each phase is early-stopped once neither mean validation AUROC nor mean validation nMAE (of the live or SWA weights) has improved for 1010K optimizer steps (at 1818K steps for PluRel and 2626K for the Join), at a global batch size of 10241024 contexts per step using Muon [36] for 22D weight matrices and AdamW for the remaining parameters, at a constant learning rate of 5×10−45× 10^-4 (RMS-matched for Muon as in [53]) after a linear warmup over the first 20002000 steps. To compensate for a lack of decay phase [54], and following the recommendation in [38], we track the stochastic weight average (SWA) performance during training and select the best SWA checkpoint based on validation performance. Specifically, we use an exponential moving average (EMA) for SWA, with momentum 0.99950.9995, which gives an effective window of ≈2000≈\!2000 steps, and find that it produces more stable curves with better results. Each phase takes about a day or less on a single node with 8×8×A100 GPUs, with BFloat16 activations and FP32 master weights [55]. The released RT-J checkpoint is the SWA weights after 99K steps of the the Join phase, selected by mean validation AUROC (74.3674.36); selection runs over the whole SWA trajectory, so the selected step lies well before the step at which the phase stopped. The same single checkpoint is used for every task, classification and regression alike. Appendix H Pretraining Ablations (a) Multi-cell masking ablation (b) Pretraining task-mix ablation Figure 13: Pretraining ablations: multi-cell masking rate and task mix, as validation nMAE / AUROC over pretraining steps. Each arm differs from RT-J’s own pretraining run in one knob and is trained to the same 32,76932,769-step budget. (a) Masking non-target cells at any rate pmax>0p_ >0 costs both metrics; RT-J masks only the target cell. (b) Forecasting tasks alone nearly match the mixture on classification but lose on regression, and autocompletion alone trails on both; the mixture is competitive on both. H.1 Multi-cell masking rate The original RT recipe [23] masks only a single target cell in a given context of length L. Following prior work on masked pretraining and meta-learning [56, 57, 58, 59], the framework also supports masking non-target cells with a hierarchical Bernoulli scheme: per training context i a rate pi∼Unif([0,pmax])p_i ([0,p_ ] ) is drawn, and each non-target, non-missing cell is masked independently with that rate mi,j∼Bernoulli(pi)m_i,j (p_i); RT-J is pretrained with pmax=0p_ =0, i.e., only the target cell is masked. The hyperparameter pmaxp_ of the hierarchical Bernoulli masking rate controls how many non-target cells are masked, and thus how much supervision each context carries. To test whether denser supervision would have been better, we sweep pmaxp_ over 0,0.25,0.5,0.75\0,0.25,0.5,0.75\, with identical data and context-sampling settings, all arms warm-started from the same checkpoint. Observations. Figure 13(a) shows that masking non-target cells hurts both metrics. Over the common window of 1010K–1818K pretraining steps, single-cell masking averages 74.274.2 validation AUROC and 33.333.3 nMAE, while every masked rate is 1.61.6–2.62.6 AUROC points and 0.60.6–1.51.5 nMAE behind (71.671.6/33.933.9 at pmax=0.25p_ =0.25, 71.671.6/34.034.0 at 0.50.5, and 72.572.5/34.834.8 at 0.750.75); the gap holds at each arm’s own best step (74.474.4/32.932.9 against a best of 73.173.1/33.133.1 across the masked arms). Masked non-target cells corrupt the very context the model predicts from, and at this model and data scale that cost is not repaid by the extra supervision per step. Denser supervision is therefore not free: it buys gradient signal by degrading the in-context evidence, and at a fixed step budget the trade is a losing one. H.2 Pretraining task mix the Join supplies two families of pretraining tasks, forecasting and autocompletion (Section 3), and RT-J is pretrained on both. We ablate the mixture by pretraining on each family alone, holding everything else fixed. Observations. From Figure 13(b), the mixture is the only setting competitive on both metrics. Over the same 1010K–1818K window, the mixed objective reaches 74.274.2 AUROC and 33.333.3 nMAE; forecasting alone is close on classification (73.673.6) but 0.40.4 nMAE behind, and autocompletion alone trails badly on classification (71.271.2, three points down) while also being the worst on regression (34.834.8). The single-family arms are not simply slower: autocompletion alone posts the best regression value of any arm early (32.732.7 nMAE at 44K steps) and then degrades steadily, which is what one expects when the pretraining distribution stops resembling the RelBench forecasting tasks the model is evaluated on. Mixing the two families is what keeps both metrics improving together, so we keep it in the recipe. H.3 Compute scaling Figure 14 tracks validation nMAE and AUROC over pretraining for three runs with the same recipe: from scratch on synthetic PluRel data only, from scratch on the Join only, and continued pretraining of RT-PluRel on the Join, which yields RT-J. Real data scales better than synthetic data: the PluRel-only run stops improving after about 88K steps and degrades on regression thereafter, while the the Join-only run keeps slowly improving regression up to 3232K steps. Continued pretraining from RT-PluRel reaches its best validation metrics within about 88K steps and stays ahead of pretraining on the Join from scratch on regression throughout, and on classification at all but a few steps around 2424K, so most of its benefit arrives early. Figure 14: Validation nMAE and AUROC (SWA, 81928192-cell context) over pretraining steps. The two runs from scratch are shown from step 22K; the PluRel-only run stops early at 1818K steps. Appendix I Random-Walk Context Construction This appendix gives the full procedure for context construction summarized in Section 4. For one task item, we score same-table candidate rows by random-walk visit counts (Stage 1), then consume the candidates in order—most recent first with visit counts breaking ties when the recency flag ρ=1ρ=1, or by visit count alone when ρ=0ρ=0—as launch points for the width-bounded BFS cell collector BFSCollect of the original RT, which we reuse unchanged (Stage 2). Walks step uniformly at random along PK and FK edges, restricted to neighbor rows whose timestamp does not exceed the target’s timestamp t⋆t ; this enforces temporal validity at the walk level and avoids leaking future information into the context. Only visits to rows other than the target in the target’s table τ contribute, so the score ranks same-table rows by their personalized random-walk proximity to the target. Algorithm 1 Random-walk context construction (one task item). 1: target row v⋆v in table τ with timestamp t⋆t ; database D; total cell budget L; per-seed cell budget ℓ ; BFS width-bound b; recency flag ρ∈0,1ρ∈\0,1\; walks W; walk length K. 2: cell sequence x with ||≤L|x|≤ L. 3: Stage 1: score same-table candidates by random-walk visits. 4: score[u]←0score[u]← 0 for every row u. 5: for w=1w=1 to W do 6: u←v⋆u← v . 7: for k=1k=1 to K do 8: Move u to a uniformly random PK/FK neighbor whose timestamp is ≤t⋆≤ t ; if no such neighbor exists, end this walk. 9: if u≠v⋆u≠ v and u is in table τ then 10: score[u]←score[u]+1score[u] [u]+1. 11: end if 12: end for 13: end for 14: Stage 2: launch BFS from the ordered candidates. 15: If ρ=1ρ=1, order same-table rows by (timestamp, scorescore) lexicographically descending; otherwise order by scorescore descending. 16: ←x← empty cell sequence. 17: for each row s in this order do 18: if ||≥L|x|≥ L then break. 19: end if 20: Append BFSCollect(,s,ℓ,b,t⋆,L−||)(D,s, ,b,t ,L-|x|) to x. 21: end for 22: return x. I.1 Sensitivity to random-walk parameters The random-walk retriever of Appendix I uses W=10,000W=10,000 walks of length K=20K=20 by default. We evaluate RT-J with the default and three cheaper settings on the 81928192-row test subsample at L=8192L=8192, over three context seeds. Table 9 breaks down the per-example inference cost and Tables 10 and 11 the accuracy. Fewer or shorter walks cut the context-sampling cost from 28.128.1 to 14.314.3–17.917.9 ms per example, while the forward pass stays at about 1010 ms. Classification degrades steadily as the walks shrink, from 72.972.9 to 71.971.9 mean AUROC, driven by rel-trial/study-outcome (59.459.4 to 52.752.7–55.855.8), rel-stack/user-engagement and rel-f1/driver-dnf. Regression is insensitive on average (33.333.3–33.633.6 mean nMAE), although individual tasks move in both directions: rel-hm/item-sales improves with shorter walks (14.114.1 to 11.611.6 nMAE at (10K,10)(10K,10)), while rel-f1/driver-position is best at the default. We keep the default. Table 9: Per-example inference cost in ms at batch size 11 and L=8192L=8192, on one B200 GPU with one AMD EPYC 9565 CPU: per-task means over 100100 timed steps after 55 warmup steps, then mean ± std. over the 2121 RelBench tasks. Stage (10K,20)(10K,20) (10K,10)(10K,10) (1K,20)(1K,20) (1K,10)(1K,10) Context sampler (CPU) 28.06±10.3928.06± 10.39 17.87±8.1417.87± 8.14 14.31±7.9914.31± 7.99 16.00±8.4916.00± 8.49 Forward pass (GPU) 10.03±0.3810.03± 0.38 9.85±0.429.85± 0.42 9.81±0.439.81± 0.43 9.78±0.459.78± 0.45 Total 38.09±10.3138.09± 10.31 27.72±8.1227.72± 8.12 24.11±7.9924.11± 7.99 25.78±8.5225.78± 8.52 Table 10: AUROC (%, higher is better) of RT-J for (W,K)(W,K) on the 81928192-row test subsample at L=8192L=8192, mean ± std. over 33 context seeds; the mean row averages the tasks per seed. Bold: within one standard deviation of the best per row. Task (10K,20)(10K,20) (10K,10)(10K,10) (1K,20)(1K,20) (1K,10)(1K,10) rel-amazon/item-churn 76.67±0.0576.67± 0.05 76.69±0.0676.69± 0.06 76.79±0.1276.79± 0.12 76.91±0.1476.91± 0.14 rel-amazon/user-churn 66.88±0.1066.88± 0.10 67.05±0.1167.05± 0.11 66.69±0.1466.69± 0.14 66.50±0.1266.50± 0.12 rel-avito/user-clicks 52.31±0.9852.31± 0.98 52.83±2.6552.83± 2.65 53.14±3.4153.14± 3.41 53.87±2.6953.87± 2.69 rel-avito/user-visits 61.70±0.2261.70± 0.22 62.68±0.6262.68± 0.62 62.50±0.6262.50± 0.62 62.42±0.5962.42± 0.59 rel-event/user-ignore 79.91±1.9579.91± 1.95 79.55±2.2079.55± 2.20 78.94±2.1678.94± 2.16 77.71±0.8477.71± 0.84 rel-event/user-repeat 77.91±1.1377.91± 1.13 77.48±1.2477.48± 1.24 77.07±1.2377.07± 1.23 75.70±1.7375.70± 1.73 rel-f1/driver-dnf 82.82±0.1782.82± 0.17 82.43±0.0382.43± 0.03 81.90±0.1381.90± 0.13 81.40±0.0981.40± 0.09 rel-f1/driver-top3 90.75±0.4590.75± 0.45 91.18±0.4091.18± 0.40 90.86±0.2690.86± 0.26 90.95±0.2790.95± 0.27 rel-hm/user-churn 62.31±0.2162.31± 0.21 62.03±0.2162.03± 0.21 61.76±0.2261.76± 0.22 63.48±0.0363.48± 0.03 rel-stack/user-badge 79.74±0.4579.74± 0.45 79.55±0.5779.55± 0.57 79.75±0.7179.75± 0.71 79.43±0.5879.43± 0.58 rel-stack/user-engagement 84.73±0.4384.73± 0.43 82.60±0.5582.60± 0.55 83.18±0.3783.18± 0.37 82.12±0.3982.12± 0.39 rel-trial/study-outcome 59.36±1.7259.36± 1.72 55.83±1.1555.83± 1.15 52.95±2.2052.95± 2.20 52.74±3.0052.74± 3.00 Mean 72.92±0.2772.92± 0.27 72.49±0.1472.49± 0.14 72.13±0.2872.13± 0.28 71.93±0.3471.93± 0.34 Table 11: nMAE (%, lower is better) of RT-J for (W,K)(W,K) on the 81928192-row test subsample at L=8192L=8192, mean ± std. over 33 context seeds; the mean row averages the tasks per seed. Bold: within one standard deviation of the best per row. Task (10K,20)(10K,20) (10K,10)(10K,10) (1K,20)(1K,20) (1K,10)(1K,10) rel-amazon/item-ltv 8.49±0.018.49± 0.01 8.25±0.098.25± 0.09 8.35±0.038.35± 0.03 8.07±0.038.07± 0.03 rel-amazon/user-ltv 30.21±0.1030.21± 0.10 30.44±0.0730.44± 0.07 30.45±0.0730.45± 0.07 30.17±0.1130.17± 0.11 rel-avito/ad-ctr 42.17±0.5542.17± 0.55 42.22±0.4542.22± 0.45 43.07±0.6443.07± 0.64 43.11±0.5943.11± 0.59 rel-event/user-attendance 43.46±1.6643.46± 1.66 43.38±1.7143.38± 1.71 43.45±1.7043.45± 1.70 43.78±1.8543.78± 1.85 rel-f1/driver-position 38.53±0.0338.53± 0.03 39.53±0.1039.53± 0.10 39.97±0.1839.97± 0.18 39.75±0.0739.75± 0.07 rel-hm/item-sales 14.07±0.2014.07± 0.20 11.61±0.0911.61± 0.09 12.56±0.0712.56± 0.07 13.38±0.0613.38± 0.06 rel-stack/post-votes 15.54±0.0415.54± 0.04 15.46±0.0415.46± 0.04 15.46±0.0215.46± 0.02 15.21±0.0415.21± 0.04 rel-trial/site-success 91.10±0.1491.10± 0.14 91.94±0.2991.94± 0.29 91.80±0.3191.80± 0.31 92.18±0.4092.18± 0.40 rel-trial/study-adverse 16.39±0.0316.39± 0.03 16.61±0.0416.61± 0.04 16.76±0.0016.76± 0.00 16.90±0.0516.90± 0.05 Mean 33.33±0.1533.33± 0.15 33.27±0.1833.27± 0.18 33.54±0.2033.54± 0.20 33.62±0.2033.62± 0.20 Appendix J Additional and Per-Task Results J.1 Baselines with extended context Figure 15 repeats the aggregate baselines comparison of Figure 3 on a fixed 81928192-row test subsample per task, extending all four baselines to a 131K131K-cell context (∼4,500 4,500 in-context examples on average, 20×20× what RT-J sees at 8K8K cells)—far past RT-J’s 8K8K training context. The extra context pays off on classification: the TabICLv2 pipelines pass RT-J’s 8K8K-context AUROC by 3.63.6–4.24.2 points at the largest context and the LightGBM pipelines by up to about 11 point. On regression, only LLM Agent + TabICLv2 passes RT-J, and only from 65K65K cells on (32.532.5 vs. 33.433.4 nMAE at 131K131K); RDBLearn + TabICLv2 improves steadily with context but stays 0.30.3–0.50.5 nMAE behind at the two largest sizes, and the LightGBM pipelines stay 1.71.7–5.25.2 nMAE behind at every extended context size (16K16K–131K131K cells). Figure 15: Few-shot relational prediction vs. in-context-learning baselines on a fixed 81928192-row-per-task subsample of the RelBench test set, with the baselines extended to 131K131K-cell contexts. As in Figure 3, the x-axis is the mean number of in-context labeled task rows and the markers on the RT-J curve are annotated with the underlying context size (RT-J caps at its 8K8K-cell training context). With 20×20× more labels than RT-J ever sees, the TabICLv2 pipelines lead on classification, and LLM Agent + TabICLv2 passes RT-J’s regression error only from 65K65K cells on. J.2 Per-task baseline curves Figures 16 and 17 disaggregate the aggregate baselines comparison of Figure 3 into one panel per RelBench task, plotting test nMAE for the 99 regression tasks and test AUROC for the 1212 classification tasks against inference-time context size. The five methods, the context grid, and the full-test evaluation protocol match Section 5; per-task y-axes are auto-scaled. On regression, RT-J’s aggregate dominance in Figure 3 is anchored by wide wins at every context size on tasks whose targets are heavy-tailed or spread across the graph (rel-stack/post-votes, rel-amazon/user-ltv, rel-event/user-attendance, rel-avito/ad-ctr), where the flat-feature pipelines stall or get worse with more context. Elsewhere the TabICLv2 pipelines keep improving with context and pass RT-J at the largest budgets (rel-amazon/item-ltv), lead it at every context size on rel-hm/item-sales and rel-f1/driver-position, as do the feature-engineering pipelines on rel-trial/site-success, while rel-trial/study-adverse ends 0.80.8 nMAE behind RDBLearn + TabICLv2. On classification, RT-J leads or ties on nearly every task at small contexts—the label-scarce regime this paper targets. At 8K8K the picture is mixed: RT-J stays ahead on rel-event/user-repeat, rel-amazon/user-churn, rel-f1/driver-dnf and rel-avito/user-visits and ties on rel-hm/user-churn, while the feature-engineering pipelines overtake it on rel-avito/user-clicks, rel-amazon/item-churn, the two rel-stack tasks, rel-trial/study-outcome, rel-event/user-ignore and rel-f1/driver-top3, mirroring the aggregate convergence at the largest label budget. Figure 16: Per-task test nMAE versus the average number of in-context examples on RelBench regression tasks; per-task counterpart of Figure 3 (left). Figure 17: Per-task test AUROC versus the average number of in-context examples on RelBench classification tasks; per-task counterpart of Figure 3 (right). J.3 Cost of test-time compute scaling Table 12 compares the cost of context tuning and context ensembling (Section 6), counted in passes of the context sampler and model over an evaluation split, and in labels used beyond the training split from which in-context examples are drawn. See Table 9 for forward pass latencies. Table 12: Passes over an evaluation split, in total and on the critical path when fully parallelized, and labels used beyond the training split. Setting Total passes Fully parallel Extra labels Default context 11 test 11 test none Tuning (120120 configurations, 44 seeds) 144144 val. ++ 11 test 11 val. ++ 11 test ≤4096≤ 4096 (val.) Ensembling (1616 seeds) 1616 test 11 test none Tuning scores 120120 configurations (L,ℓ,b,ρ)(L, ,b,ρ) with 44 context seeds each. A single pass samples one context at the largest L and scores every L≥ℓL≥ on its prefixes, so the grid takes 3636 passes per seed over the shared (ℓ,b,ρ)( ,b,ρ) settings, i.e. 144144 passes over a validation subsample of up to 40964096 rows, plus one test pass with the selected configuration. Ensembling needs no labels beyond the training split. Both procedures parallelize fully across configurations and seeds. J.4 Per-task ablation curves This appendix disaggregates each ablation in Section 7 into per-task panels, one per RelBench task, evaluated on the 81928192-row test subsample of Section 7 across context sizes; per-panel y-axes are auto-scaled so the visual ranking of methods within a task is preserved even where its absolute scale differs. These panels disaggregate the aggregate ablation figures (context retrieval, schema semantics) directly. J.4.1 Context retrieval (per task) Per-task counterpart of Figure 6. Random walk’s advantage at small L is broad: at L=256L=256 it is the best or tied retriever on most regression tasks. Beyond L=1024L=1024 RDBLearn-feature similarity leads on most of them (item-ltv, user-ltv, ad-ctr, item-sales, post-votes, study-adverse) and the two are close on user-attendance and driver-position; on site-success the structure-agnostic retrievers stay flat near 9494 nMAE, 3232-width BFS is far ahead beyond L=1024L=1024, random walk drifts back up past L=1024L=1024 and RDBLearn-feature similarity is erratic. On classification the same two retrievers lead nearly everywhere, RDBLearn-feature similarity by the wider margin on the churn and driver tasks and random walk on the rel-event tasks; the tasks with few labels (user-clicks, user-engagement, study-outcome) are noisy for every retriever. The 256256-width BFS variant is a strong outlier at small L on several tasks; it is excluded from per-panel y-limit computation so the other curves remain readable, but its line is still drawn. Figure 18: Per-task test nMAE for context-retrieval ablations on RelBench regression tasks; per-task counterpart of Figure 6 (left). Figure 19: Per-task test AUROC for context-retrieval ablations on RelBench classification tasks; per-task counterpart of Figure 6 (right). J.4.2 Schema semantics (per task) Per-task counterpart of Figure 5. Schema semantics help on most tasks. The largest classification gaps (rel-event/user-ignore, rel-event/user-repeat) are worth up to 5.25.2 AUROC points; shuffled names win at every context size on rel-avito/user-clicks, rel-f1/driver-top3 and rel-hm/item-sales, on rel-amazon/user-ltv from L=512L=512, and on rel-f1/driver-position by up to 5.15.1 nMAE at small contexts. Figure 20: Per-task test nMAE for the schema-semantics ablation on RelBench regression tasks; per-task counterpart of Figure 5 (left). Figure 21: Per-task test AUROC for the schema-semantics ablation on RelBench classification tasks; per-task counterpart of Figure 5 (right). J.4.3 Context hyperparameter selection (per task) Each task’s tuned context is chosen by a grid search over the global context length L∈512,…,8192L∈\512,…,8192\, the per-seed cell budget ℓ∈256,…,8192 ∈\256,…,8192\ (with ℓ≤L ≤ L), the BFS width b∈8,32,128b∈\8,32,128\, and the recency flag ρ∈0,1ρ∈\0,1\—120120 configurations per task—selecting per task the configuration that maximizes validation performance on a 40964096-row validation subsample (four context seeds, per-row predictions averaged before scoring; nMAE for regression, AUROC for classification). Table 13 reports the resulting test metrics on the 8192-subsampled test set, comparing this tuned context against the shared default context (L=8KL=8K, ℓ=256 =256, b=32b=32, ρ=1ρ=1) at a single context seed; columns L, ℓ , b, ρ report the selected values. Table 13: Per-task test metrics on the 8192-subsampled test set: shared default context (L=8KL=8K, ℓ=256 =256, b=32b=32, ρ=1ρ=1) vs. the per-task tuned context. Columns L, ℓ , b, ρ report the configuration selected on the validation split. Single context seed (no ensembling). All metrics shown in percent. Best per row in bold; lower is better for nMAE, higher for AUROC. DB Task L ℓ b ρ Default Tuned Regression (nMAE %, lower is better) rel-amazon item-ltv 4K4K 20482048 88 00 8.508.50 7.887.88 rel-amazon user-ltv 4K4K 20482048 88 00 30.0230.02 29.4529.45 rel-avito ad-ctr 8K8K 10241024 88 00 42.6942.69 42.4742.47 rel-event user-attendance 8K8K 256256 88 11 43.5943.59 42.5442.54 rel-f1 driver-position 8K8K 256256 88 11 38.6038.60 38.6038.60 rel-hm item-sales 512512 256256 128128 00 14.0014.00 11.7911.79 rel-stack post-votes 8K8K 20482048 88 00 15.5115.51 15.0315.03 rel-trial site-success 8K8K 40964096 3232 00 90.9890.98 88.6188.61 rel-trial study-adverse 8K8K 256256 128128 11 16.3816.38 16.3516.35 Mean 33.3633.36 32.5232.52 Classification (AUROC %, higher is better) rel-amazon item-churn 8K8K 512512 128128 11 76.7376.73 77.7977.79 rel-amazon user-churn 8K8K 256256 3232 11 66.4866.48 66.4866.48 rel-avito user-clicks 8K8K 20482048 128128 11 55.5455.54 47.6347.63 rel-avito user-visits 1K1K 512512 128128 00 61.6361.63 61.3361.33 rel-event user-ignore 4K4K 512512 88 11 78.4378.43 80.5380.53 rel-event user-repeat 2K2K 512512 88 11 76.3176.31 76.2976.29 rel-f1 driver-dnf 8K8K 256256 128128 11 82.8582.85 83.0383.03 rel-f1 driver-top3 8K8K 256256 128128 11 91.0291.02 90.7590.75 rel-hm user-churn 8K8K 512512 128128 00 63.1463.14 65.5065.50 rel-stack user-badge 8K8K 40964096 3232 11 80.2080.20 79.4979.49 rel-stack user-engagement 512512 512512 88 11 83.7683.76 79.3979.39 rel-trial study-outcome 8K8K 20482048 3232 00 58.2058.20 60.5260.52 Mean 72.8672.86 72.3972.39 J.4.4 Context ensembling (per task) Per-task counterpart of Figure 5, for the default (blue) and per-task-tuned (orange) context. Increasing the number of ensembled context seeds nensn_ens improves classification on most tasks (99 of 1212 at the default context and 1010 of 1212 at the tuned one), but degrades rel-avito/user-clicks by 3.03.0 points at the default context, while several regression tasks (rel-amazon/user-ltv, rel-stack/post-votes) are insensitive, in line with the flatter aggregate regression curve. The panels also localize the val–test mismatches behind the aggregate classification crossover in Figure 5: on rel-avito/user-clicks and rel-stack/user-engagement the tuned context is clearly worse than the default at every ensemble size, and on rel-trial/study-outcome it starts 2.32.3 AUROC ahead but ends 0.70.7 behind at nens=16n_ens=16. (On rel-amazon/user-churn, tuning selects the default configuration, so the two curves coincide.) Figure 22: Per-task test nMAE for context ensembling, default vs. per-task-tuned context, on RelBench regression tasks; per-task counterpart of Figure 5 (left). Figure 23: Per-task test AUROC for context ensembling, default vs. per-task-tuned context, on RelBench classification tasks; per-task counterpart of Figure 5 (right).