Paper deep dive
Atom-JEPA: Joint-Embedding Predictive Architecture for 3D Atomistic Systems
Kasper Helverskov Petersen, Rasmus Hannibal Tirsgaard, François R J Cornet, Mikkel Jordahn, Mikkel N. Schmidt
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large-scale self-supervised pretraining has reshaped modern machine learning, substantially advancing the ability of language and vision models to generalize across downstream tasks. While deep learning has driven considerable progress in modeling atomistic systems in recent years, self-supervised pretraining in this domain has not yet achieved comparable downstream generalization. To address this, we introduce Atom-JEPA, a self-supervised pretraining framework that learns latent representations from unlabeled 3D structures through complementary atom-level and substructure-level objectives inspired by joint-embedding predictive architectures. We pretrain Atom-JEPA on large-scale molecular and crystalline datasets and evaluate its transfer performance by fine-tuning on a diverse set of downstream property prediction tasks. Atom-JEPA achieves state-of-the-art performance on molecular ADMET and quantum-chemical property prediction tasks, and is highly competitive in predicting the physical properties of crystalline materials. These results demonstrate the potential of latent-space predictive pretraining to support broad downstream generalization from structural data alone. Code and pretrained model checkpoints are publicly available at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2610.08400v1
- Canonical: https://arxiv.org/abs/2610.08400v1
Trouble viewing inline? Open PDF directly →
Full Text
207,710 characters extracted from source content.
Expand or collapse full text
Atom-JEPA: Joint-Embedding Predictive Architecture for 3D Atomistic Systems Kasper Helverskov Petersen* Rasmus Hannibal Tirsgaard* François R J Cornet Affiliation: Mikkel Jordahn*, Mikkel N. Schmidt Affiliation: Department of Applied Mathematics and Computer Science Affiliation: Technical University of Denmark Affiliation: Corresponding authors: khepe,rhti,mikkjo@dtu.dk Abstract Large-scale self-supervised pretraining has reshaped modern machine learning, substantially advancing the ability of language and vision models to generalize across downstream tasks. While deep learning has driven considerable progress in modeling atomistic systems in recent years, self-supervised pretraining in this domain has not yet achieved comparable downstream generalization. To address this, we introduce Atom-JEPA, a self-supervised pretraining framework that learns latent representations from unlabeled 3D structures through complementary atom-level and substructure-level objectives inspired by joint-embedding predictive architectures. We pretrain Atom-JEPA on large-scale molecular and crystalline datasets and evaluate its transfer performance by fine-tuning on a diverse set of downstream property prediction tasks. Atom-JEPA achieves state-of-the-art performance on molecular ADMET and quantum-chemical property prediction tasks, and is highly competitive in predicting the physical properties of crystalline materials. These results demonstrate the potential of latent-space predictive pretraining to support broad downstream generalization from structural data alone. Code and pretrained model checkpoints are publicly available at https://github.com/khelverskovp/atom-jepa. 11footnotetext: Core contributors. 1 Introduction In recent years, machine learning has been adopted at an accelerating rate in molecular and materials discovery. Generative models enable inverse design through candidate generation (Hoogeboom et al., 2022; Cornet et al., 2024; Zeni et al., 2025), structure prediction (Abramson et al., 2024; Wohlwend et al., 2025; Chai Discovery Team et al., 2025) and property-guided optimization, complementing traditional workflows reliant on expert-guided design, computational screening, and experimental structure determination. Machine-learning interatomic-potentials (MLIPs) (Schüt et al., 2021; Batatia et al., 2022; Rhodes et al., 2025; Fu et al., 2025; Wood et al., 2026; Kavanagh et al., 2026) serve as surrogates for electronic structure methods such as DFT, while predictive models are increasingly used to screen promising candidates across a wide range of properties (Kellenberger et al., 2007; Zhuang et al., 2014; Pillai et al., 2023; Wong et al., 2024; Sun et al., 2024; Bai et al., 2025) before running costly computational or experimental evaluation. Historically, many of these approaches have relied on tailored models trained from scratch on individual datasets, often using molecular fingerprints (Rogers and Hahn, 2010) or string-based representations (Weininger, 1988). Recently, increasing computational resources and growing availability of structural data has enabled the training of deep neural networks directly on 3D atomistic structures (Schüt et al., 2017; Gilmer et al., 2017), offering the potential to learn representations that capture chemically relevant information directly from atomistic structure. However, a central question that remains unanswered is how to effectively leverage massive unlabelled datasets to improve performance on sparsely labeled downstream tasks, where less complex approaches such as simple fingerprint baselines often remain competitive. Foundation models, i.e. neural networks pretrained on large-scale datasets and subsequently adapted to downstream tasks, have been central to recent advances in deep learning. Many modern computer vision models (Siméoni et al., 2026), large-language models (Chiang et al., 2022), and tabular data models (Hollmann et al., 2022; Qu et al., 2025) increasingly rely on large-scale pretraining. In machine learning for atomistic systems, there is a similar growing interest in developing foundation models (Pyzer-Knapp et al., 2025; Yuan et al., 2026) that can exhibit strong generalization across different chemical systems and tasks. Existing approaches either pretrain via supervised regression on quantum-chemical properties such as energies, forces, stress, and electronic characteristics, or through self-supervised structural objectives such as denoising, masking, and environment prediction. Based on the success of self-supervised learning (SSL) in computer vision and language models, we posit that SSL is the way forward for strong foundation models for atomistic systems as well. In this work, we propose and demonstrate Atom-JEPA, a performant adaptation of the joint-embedding predictive architecture framework (LeCun, 2022) to 3D atomistic systems. Specifically, Atom-JEPA combines complementary atom-level and substructure-level predictive objectives in latent space, using only 3D positions and atom types. In contrast, existing approaches based on 3D atomistic structure primarily formulate self-supervision through prediction or reconstruction in the input space, combining 3D information with other modalities, or employing latent-space predictive objectives only at the substructure level. For an overview of Atom-JEPA, we refer to Fig. 1. We train Atom-JEPA models on both molecular and crystalline structures, analyse their learned embedding spaces, and finally evaluate their performance on supervised downstream tasks spanning ADMET, quantum-chemical, and crystalline material properties. To summarise: 1. We propose Atom-JEPA, to the best of our knowledge, the first JEPA-style loss that operates both on the atom- and substructure level of 3D atomistic systems alone. 2. We show that Atom-JEPA outperforms most existing open-source models and methods across ADMET, quantum-chemical and crystalline material property predictions. 3. We analyse the learned embeddings of Atom-JEPA trained models, illustrating where they do (and do not) separate atomistic systems by higher-order structural and functional groups. Figure 1: Overview of Atom-JEPA. Molecular and crystalline structures are partitioned into complementary context (red) and target (blue) subgraphs by sampling a k-hop neighborhood around an anchor atom. A 3D equivariant graph neural network encodes the context, which is then used to predict target representations at two scales: individual atoms and mean-pooled substructures. Target representations are extracted from the full structure after using a target encoder updated as an exponential moving average of the context encoder. Pretraining minimizes prediction errors in latent space, with the two subgraphs alternating as context and target. The pretrained context encoder is subsequently fine-tuned for downstream property prediction. 2 Related Work Supervised pretraining Supervised pretraining trains a neural network on a large labeled dataset, before finetuning or adapting it to downstream tasks (Shoghi et al., 2024; Ivković et al., 2024; Kong et al., 2025; Kim et al., 2026). For atomistic systems, models are typically pretrained on energies and forces computed using density functional theory (DFT), and while these labels provide rich supervision about the underlying potential-energy surface, this pretraining objective may not capture all information relevant to diverse downstream properties. Another limitation is scalability: generating high-quality labels for molecules or crystals is computationally expensive. OMol25, for example, contains labels for 150 million configurations of atomistic systems at the ω 97M-V/def2-TZVPD level of theory and required approximately 6.6 billion CPU hours to generate (Levine et al., 2026). In contrast, SSL provides a route to exploiting structural data without requiring expensive label simulations. Self-supervised pretraining Self-supervised learning as pretraining has also seen increasing attention in recent years. One of the earliest methods used VAEs, where the authors learn a latent representation of molecules by training a VAE on SMILES strings (Gómez-Bombarelli et al., 2018). Since then, the development has been wide and varied. Several works have done SSL on 2D graphs, using both masking and contrastive learning losses (Hu et al., 2020; Li et al., 2022; Wang et al., 2023; Xia et al., 2023; Xue et al., 2026) and more recently, similar losses have been proposed for 3D molecular graphs, with the expectation that features incorporating geometry should allow better downstream performance (Zhu et al., 2022; Zhou et al., 2023; Wu et al., 2025). Finally, a recent surge of papers has proposed using positional 3D denoising, sometimes in combination with masking losses, as a pretraining objective (Zaidi et al., 2023; Feng et al., 2023; Liu et al., 2025; Perez and Gomez-Bombarelli, 2026; Morehead et al., 2026). Joint-Embedding Predictive Architecture Joint-embedding predictive architectures (JEPA) (LeCun, 2022) predict directly in latent space rather than in the input domain, and rely on generating views of data which teaches the model to focus on higher, more abstract level features of datapoints by using a predictor network. Graph-JEPA (Skenderi et al., 2025) divides 2D graphs into sub-graphs, and uses these sub-graphs as context and target views, but without a node-level objective. Polymer-JEPA (Piccoli et al., 2026) extends the Graph-JEPA framework to 2D polymers by changing the target encoder to encode the full polymer, while only computing the loss over a target subgraph, by pooling the embeddings of the nodes in that subgraph that are not contained within the context subgraph, again with no node-level objective, and on 2D graphs. Crys-JEPA (Liu et al., 2026) operates on sequences (created from the 3D crystal graph) using a Transformer and generates contexts by rotating and translating original target graphs, whilst using formation energy in the loss to produce energy-aware embeddings. C-Free (Ariguib et al., 2026) is most similar to our method, but with significant differences: C-Free operates on 2D and 3D graphs jointly and generates contexts via k-hops along covalent bonds, where the targets are the complements to the context subgraph. In comparison, we sidestep the need for multiple modalities by introducing an atom-level JEPA objective, and generating contexts via k-hops on a cutoff radius based graph. Most recently, Mol-JEPA (Rottach et al., 2026) is a framework in which representations from multiple pretrained models, computed descriptors, and experimental measurements define different molecular modalities, with masked modalities predicted in latent space from the remaining modalities. To summarise, our method is to the best of our knowledge, the first JEPA model for atomistic systems to learn atomic representations solely through latent-space prediction between 3D structural views. 3 Method The core idea of Atom-JEPA is analogous to I-JEPA for images (Assran et al., 2023), but tailored to atomistic systems in 3D space: given part of a molecule or crystal (context view), predict the representation of the remaining part (target view) using context and target encoders coupled through an exponential moving average (EMA). We consider two prediction objectives: (1) an atom-level objective that predicts the representation of each atom in the target view, and (2) a substructure-level objective that predicts the mean-pooled representation of all target atoms. For the context encoder, target encoder, and predictor, we use an SE(3)-equivariant graph neural network (GNN). We restrict our training data to structures that are close to local minima of the potential energy surface, as including off-equilibrium structures without information about their relative stability (e.g. total energy), could make the learning problem ambiguous. We discuss this choice further in Section A.4. 3.1 3D Graph Representation and Partitioning We represent an atomistic system with N atoms by a 3D graph G=(V,E)G=(V,E). The node set V=(zi,i)i=1NV=\(z_i,r_i)\^N_i=1 represents atoms by their species ziz_i and coordinates i∈ℝ3r_i ^3, while the edge set E=(i,j)∣i≠j,∥i−j∥<rcE=\(i,j) i\!≠\!j,\|r_i\!-\!r_j\|\!<\!r_c\ encodes spatial proximity defined by a cutoff radius rcr_c. For crystalline systems, the edge set is constructed under periodic boundary conditions, including edges to periodic images within the cutoff radius. Given an atomistic system, we partition atom indices into a k-hop EgoNet ℐaI_a and its complement ℐbI_b: we sample an anchor atom c∼Unif1,…,Nc \1,…,N\ and a hop count k∼Unifk from a predefined set of hop counts K, and define the EgoNet centered at c and its complement as ℐa=i∈1,…,N∣d(c,i)≤k,ℐb=1,…,N∖ℐa,I_a=\i∈\1,…,N\ d(c,i)≤ k\, _b=\1,…,N\ _a, (1) where d(c,i)d(c,i) denotes the shortest-path distance between atoms c and i, measured in number of edges. The corresponding induced subgraphs are Ga=G[ℐa]G_a=G[I_a] and Gb=G[ℐb]G_b=G[I_b] where G[ℐ]=(Vℐ,Eℐ),Vℐ=(zi,i)∣i∈ℐ,Eℐ=(i,j)∈E∣i,j∈ℐ.G[I]=(V_I,E_I), V_I=\(z_i,r_i) i \, E_I=\(i,j)∈ E i,j \. (2) For each partition, we train in both prediction directions, defining the context and target graphs (Gx,Gy)(G_x,G_y) as either (Ga,Gb)(G_a,G_b) or (Gb,Ga)(G_b,G_a). 3.2 Context, Target and Predictor Models The context and target encoders, fθf_θ and fθ¯f_ θ respectively, share the same 3D equivariant GNN architecture, taking 3D graphs as inputs and producing atom-level features. The parameters of the context encoder, θ, are updated via gradients, whereas the target encoder parameters, θ¯ θ, are an exponential moving average of θ. The context encoder is only given context graph GxG_x, whilst the target encoder is given the full graph G; the JEPA loss is only computed over the target graph GyG_y. We label the context encoder output as xs_x and the target encoder output on the GyG_y graph as ys_y. Given the context representations xs_x, the predictor, gϕg_φ, predicts representations produced by the target encoder. The predictor consists of a shallow 3D equivariant GNN backbone followed by objective-specific readout heads. We consider two prediction objectives: (1) a substructure objective, ℒsubL_sub, that targets a global representation of the target subgraph, and (2) an atom-level objective, ℒatomL_atom, that targets the representation of each target atom. We write the atom features produced by the predictor as xh_x and the mean-pooled features as ¯x h_x. 3.3 Prediction Objectives & Atom-JEPA Loss Substructure Objective In the substructure objective, the target is the mean-pooled invariant scalar features produced by the target encoder indexed over the target subgraph, whereas the prediction is constructed by passing the invariant component of the pooled context through a two-layer MLP mapping into the feature space of the target encoder, ¯y=1|ℐy|∑i∈ℐy,i(0)and¯^y=Rϕsub(¯x)=MLP(¯x(0)), s_y= 1|I_y| _i _ys_y,i^(0) s_y=R^sub_φ( h_x)=MLP( h_x^(0)), (3) where (⋅)(0)(·)^(0) denotes the zero-th degree (invariant) features alone. Atom-level Objective The atom-level objective instead produces one prediction ^y,i s_y,i for every target atom i∈ℐyi _y, which is compared against the corresponding target encoder representation y,is_y,i. In addition to the context features, the atom-level readout head of the predictor is provided with an equivariant relative positional encoding iq_i of the target atom’s position relative to the context subgraph, ^y,i=Rϕatom(¯x,i)=MLP(i),i∈ℐy, s_y,i=R^atom_φ( h_x,q_i)=MLP(u_i), i _y, (4) where iu_i is constructed by computing the inner product between each feature channel in iq_i and ¯x h_x and concatenating the individual inner products. We refer to Appendix Sections A.3.2 and A.3.3 for details on the construction of iu_i and iq_i respectively. Atom-JEPA Loss We use mean squared error between predicted and target representations, ℓ(,)=1C‖−‖22, (a,b)= 1C\|a-b\|_2^2, where C is the feature dimension as the loss. The substructure and atom-level objectives are thus defined as ℒsubx→y=1B∑g=1Bℓ(¯^y(g),¯y(g))andℒatomx→y=1∑g=1B|ℐy(g)|∑g=1B∑i∈ℐy(g)ℓ(^y,i,y,i), _sub^x→ y= 1B _g=1^B ( s_y^(g), s_y^(g) ) _atom^x→ y= 1 _g=1^B|I_y^(g)| _g=1^B _i _y^(g) ( s_y,i,s_y,i ), (5) where x→yx→ y denotes the prediction direction in which GxG_x is used as the context graph and GyG_y as the target graph and B is the batch size. The subgraph loss is a mean over the B graphs in a batch, whilst the atom-level loss is a mean over all the target atoms in a batch B. The final loss is then, ℒAtom-JEPA=λatomℒatom+λsubℒsub,L_Atom-JEPA= _atomL_atom+ _subL_sub, (6) where λatom,sub _\atom,sub\ control the weight of each objective. 4 Experiments We use EquiformerV3 (Liao et al., 2026) as the SE(3) equivariant GNN architecture for the context encoder, target encoder, and predictor. For molecular pretraining, we use ∼ 19 million molecules from Uni-Mol (Zhou et al., 2023); for crystal pretraining, we use ∼ 1.7 million structures from Alexandria (Cavignac et al., 2026) with energies less than 50meV/atom50~meV/atom above the convex hull. Details about datasets and preprocessing are provided in App. C. Pretraining details are provided in App. G. After pretraining, we finetune the context encoder for downstream property prediction across both domains. For molecules, we consider ADMET prediction and quantum-chemical property prediction on QM9 (Ramakrishnan et al., 2014). For crystals, we evaluate materials-property prediction using the Matbench benchmark suite (Dunn et al., 2020). We further conduct ablation studies to assess the effects of pretraining, varying the pretraining prediction objective, and the scale and composition of the pretraining data (Section 4.2). Figure 2: Average rank across 38 endpoints from three benchmarks (Biogen 4 endpoints, ChEMBL-MT 25 endpoints, ExpansionRx 9 endpoints). Top part weights each dataset equally, whereas bottom part weighs each endpoint equally. Bars join models whose paired rank difference could not be distinguished from zero by a stratified seed/fold bootstrap over endpoints (10610^6 replicates, resampled within dataset, 95%95\% BCa interval). Tables with individual results can be found in Appendix E. 4.1 Experimental Results Results on ADMET We extensively investigate the transfer learning for ADMET tasks on three different datasets, spanning various amounts of data, label sparsity, and splitting methods: 1. ChEMBL-MT (Adrian et al., 2025): Multi-task, multiple sources of data, 25 targets, Taylor-Butina clustering on Morgan fingerprints with pre-defined splits. 2. ExpansionRx (MacDermott-Opeskin and Castellanos, 2026): Multi-task, single source for collection, 10 targets, temporal split of training and test data. 3. Biogen ADME (Fang et al., 2023): Multi-task, internal source, 6 targets, Bemis–Murcko scaffold and Taylor-Butina clustering on Morgan fingerprints. For completeness, we also include results on TDC-ADMET (Huang et al., 2021) in Appendix E, but exclude them from the statistical analysis, due to reported data acquisition and leaderboard inconsistencies (Koleiev et al., 2026). As Atom-JEPA requires 3D structures, we use the Uni-Mol pipeline to generate conformers (described in Section H.1.2). To ensure a fair comparison, we tune the hyperparameters using the training data from a cluster split on Biogen from Adrian et al. (2025). We use the average validation MAE over endpoints as our selection criteria (details in Section H.1). We follow the transfer learning setup of multi-head multi-target training strategy used by CKERMT (Xue et al., 2026) with some minor changes (see Appendix H). Table 1: ADMET results. For each endpoint, the model with the lowest mean error across folds is designated as Best, and an unpaired Tukey’s test is used to identify additional models that are statistically indistinguishable (Indist.) from the best model at the 95%95\% confidence level (see Appendix E.1). Sum denotes the total best or indistinguishable endpoints. ∗ reports only the mean error across splits; these values are treated as constant in the statistical tests. † did not report Biogen results for two targets and is therefore excluded from the statistical tests for these targets. Biogen (6) ChEMBL MT (25) ExpansionRx (9) Model Best / Indist. Sum Best / Indist. Sum Best / Indist. Sum Total (40) Atom-JEPA ×10 conf 3 / 3 6 6 / 10 16 7 / 2 9 31 Atom-JEPA 0 / 6 6 0 / 15 15 0 / 9 9 30 CKERMT (Xue et al., 2026) 0 / 6 6 11 / 8 19 1 / 3 4 29 KERMT* (Adrian et al., 2025) 1 / 1† 2 3 / 9 12 0 / 1 1 15† Chemprop (Graff et al., 2026) 1 / 3 4 0 / 12 12 0 / 1 1 17 Mol-JEPA ⋅· modalities (Rottach et al., 2026) 1 / 2 3 2 / 5 7 0 / 3 3 13 LightBGM Morgan + RDKit 0 / 2 2 3 / 9 12 1 / 2 3 17 We present the aggregated results from the three main ADMET datasets in Figs. 2 and 1. Atom-JEPA reaches on par or significantly better performance than the next best model, CKERMT, and outperforms fingerprint-based models. Notably, Atom-JEPA has no explicitly coded expert chemical knowledge in comparison to all other models, including the self-supervised models of KERMT and CKERMT. This also explains why we find that Atom-JEPA’s predictions are the least correlated with other models tested, see Appendix E.2. This is especially useful for the common ADMET practice of ensembling models. We additionally observe significantly improved prediction accuracy when averaging predictions over our 10 generated conformers (Atom-JEPA ×10× 10 conf) for each molecule, suggesting that different conformers can provide different information to the prediction. Interestingly, Atom-JEPA performs especially well on ExpansionRX, a dataset that differs notably from other benchmarks (Xue et al., 2026), including our pretraining data Uni-Mol, where only a single molecule shares the same scaffold (see Table 6). This suggests that the features learned by Atom-JEPA are more robust at transferring, compared to existing models. Results on QM9 QM9 consists of ∼ 130k small organic molecules with 12 quantum-chemical target properties. We use the random 110k/10k/11k split most commonly used in the literature; finetuning details are given in Section H.2. The same optimization protocol and hyperparameters are used for all targets, and all readout heads use sum aggregation over per-atom features. Table 2 compares Atom-JEPA with previous models. 3D-EMGP, DenoiseVAE, TriForces and JMP-L use different random data partitions; their results are shown in grey for reference and excluded from highlighting. Atom-JEPA outperforms all SSL pretraining methods on 11/12 targets and ties with SliDe on the remaining target, CνC_ν. Overall, Atom-JEPA achieves the best results on 8/12 targets with tied performance on α and CνC_ν. On G and U0U_0 it is within 0.3% of the best result, making it best or within 0.3% of the best on 10/12 targets. Relative to EquiformerV2_V2, the predecessor of our EquiformerV3_V3 backbone, Atom-JEPA reduces MAE on all 12 targets, with a mean relative reduction of 36%. GotenNetL, trained from scratch, remains strong, achieving the best performance on 4/12 targets, which suggests that this architecture is particularly well suited to QM9. As GotenNet is a 3D equivariant GNN, Atom-JEPA pretraining could in principle be applied to its backbone. Beyond using a different random split with a smaller test set and correspondingly larger training set, JMP-L is not directly comparable, as it is pretrained with supervised DFT labels aligned with several QM9 targets, on molecules that overlap with QM9. Even so, Atom-JEPA achieves lower error on α, μ, and R2R^2, three targets not aligned with JMP-L’s supervised pretraining labels, suggesting stronger transfer. Table 2: QM9 results. The evaluation metric is MAE (↓)( ). Bold: best result in each column; underline: within 10% of the best. Grey rows with † use a different random train/validation/test split from 110k/10k/11k and are shown for reference only, excluded from highlighting. “–” denotes unreported targets. Task α Δε εHOMO _HOMO εLUMO _LUMO μ CνC_ν G H R2R^2 U U0U_0 ZPVE Units ma03ma_0^3 meV meV meV mD mcalmol K mcalmol K meV meV ma02ma_0^2 meV meV meV No pretraining MACE (Batatia et al., 2022) 38 42 22 19 15 21 5.5 4.7 210 4.1 4.1 1.23 EquiformerV2_V2 (Liao et al., 2024) 47 29.0 14.4 13.3 9.9 23 7.57 6.22 186 6.49 6.17 1.47 GotenNetL (Aykent and Xia, 2025) 28 19.8 13.4 12.2 6.7 19 4.98 3.30 24 3.41 3.37 1.08 Supervised pretraining Transformer-M (Luo et al., 2023) 41 27.4 17.5 16.2 37 22 9.63 9.39 75 9.41 9.37 1.18 JMP-L† (Shoghi et al., 2024) 32 19.1 8.8 8.6 8.0 17 4.3 2.8 163 2.8 2.9 0.9 Self-supervised pretraining only 3D-EMGP† (Jiao et al., 2023) 57 37.1 21.3 18.2 20 26 9.30 8.70 92 8.60 8.60 1.38 GeoSSL-DDM (Liu et al., 2023) 46 40.2 23.5 19.4 15 24 7.65 7.09 122 6.99 6.92 1.31 Coord (Zaidi et al., 2023; Feng et al., 2023) 52 31.8 17.7 14.3 12 20 6.91 6.45 450 6.11 6.57 1.71 Frad (VRN) (Ni et al., 2024a) 42 27.7 17.9 13.8 11 21 6.03 6.01 354 5.35 5.41 1.63 Frad (RN) (Feng et al., 2023) 37 27.8 15.3 13.7 10 20 6.19 5.55 342 5.62 5.33 1.42 DenoiseVAE† (Liu et al., 2025) 65 26.0 14.2 11.9 7.9 15 5.35 4.19 62 4.03 4.31 1.03 SliDe (Ni et al., 2024b) 37 26.2 13.6 12.3 8.7 19 5.37 4.26 341 4.29 4.28 1.52 TriForces† (Ramlaoui et al., 2026) 75 38.8 20.2 20.4 18.0 31 – – – – 8.9 – Atom-JEPA 28 24.6 12.2 10.9 6.1 19 4.99 3.28 29 3.38 3.38 1.06 Results on Matbench Table 3 reports means and standard deviations across the five official folds for 8 Matbench tasks; training details are in Section H.3. Baseline results are taken from the official leaderboard for CGCNN, ALIGNN, and coGN; from the original papers for Crystal Twins (CT) and JMP-L; and from the Triforces paper for MACE, Orb, eSEN, and both TriForces variants. Table 3: Matbench results. The evaluation metric is MAE (↓)( ) for all tasks except MP Is Metal, which uses F1 (↑)( ). Values are means over five folds, with standard deviations where available. Pretrained models are ordered by increasing pretraining dataset size, shown in parentheses after each method. Bold: best result in each task. “–” denotes targets not reported by the original authors. * results from (Ramlaoui et al., 2026). Tasks Phonons Dielectric Log GVRH Log KVRH Perovskites MP Gap MP E Form MP Is Metal Units cm-1 – log10 _10(GPa) log10 _10(GPa) meV eV meV/atom F1 #Samples 1,265 4,764 10,987 10,987 18,928 106,113 132,752 106,113 No pretraining CGCNN (Xie and Grossman, 2018) 57.8±12.357.8 [rgb]0.5,0.5,0.5 ± 12.3 .599±.083.599 [rgb]0.5,0.5,0.5 ±.083 .090±.002.090 [rgb]0.5,0.5,0.5 ±.002 .071±.003.071 [rgb]0.5,0.5,0.5 ±.003 45.2±.745.2 [rgb]0.5,0.5,0.5 ±.7 .297±.004.297 [rgb]0.5,0.5,0.5 ±.004 33.7±.633.7 [rgb]0.5,0.5,0.5 ±.6 .946±.008.946 [rgb]0.5,0.5,0.5 ±.008 ALIGNN (Choudhary and DeCost, 2021) 29.5±2.129.5 [rgb]0.5,0.5,0.5 ± 2.1 .345±.087.345 [rgb]0.5,0.5,0.5 ±.087 .072±.001.072 [rgb]0.5,0.5,0.5 ±.001 .057±.003.057 [rgb]0.5,0.5,0.5 ±.003 28.8±.928.8 [rgb]0.5,0.5,0.5 ±.9 .186±.003.186 [rgb]0.5,0.5,0.5 ±.003 21.5±.521.5 [rgb]0.5,0.5,0.5 ±.5 .902±.002.902 [rgb]0.5,0.5,0.5 ±.002 coGN (Ruff et al., 2024) 29.7±2.029.7 [rgb]0.5,0.5,0.5 ± 2.0 .309±.086.309 [rgb]0.5,0.5,0.5 ±.086 .069±.001.069 [rgb]0.5,0.5,0.5 ±.001 .054±.003.054 [rgb]0.5,0.5,0.5 ±.003 26.9±.826.9 [rgb]0.5,0.5,0.5 ±.8 .156±.002.156 [rgb]0.5,0.5,0.5 ±.002 17.0±.317.0 [rgb]0.5,0.5,0.5 ±.3 .901±.003.901 [rgb]0.5,0.5,0.5 ±.003 With pretraining CTSimSiam (Magar et al., 2022) (0.43M) 48.9±7.748.9 [rgb]0.5,0.5,0.5 ± 7.7 .417±.079.417 [rgb]0.5,0.5,0.5 ±.079 .087±.003.087 [rgb]0.5,0.5,0.5 ±.003 .067±.003.067 [rgb]0.5,0.5,0.5 ±.003 42.0±1.042.0 [rgb]0.5,0.5,0.5 ± 1.0 .281±.025.281 [rgb]0.5,0.5,0.5 ±.025 37.0±.037.0 [rgb]0.5,0.5,0.5 ±.0 – Atom-JEPA (1.7M) 23.3±4.523.3 [rgb]0.5,0.5,0.5 ± 4.5 .268±.074.268 [rgb]0.5,0.5,0.5 ±.074 .067±.001.067 [rgb]0.5,0.5,0.5 ±.001 .047±.002.047 [rgb]0.5,0.5,0.5 ±.002 29.1±1.029.1 [rgb]0.5,0.5,0.5 ± 1.0 .162±.002.162 [rgb]0.5,0.5,0.5 ±.002 20.3±.220.3 [rgb]0.5,0.5,0.5 ±.2 .908±.002.908 [rgb]0.5,0.5,0.5 ±.002 Orb∗ (Neumann et al., 2024) (4M) 26.2 .251 .063 .051 30.7 .194 21.1 .895 MACE∗ (Batatia et al., 2022)(4M) 36.7 .279 .082 .055 61.4 .370 40.8 .858 eSEN∗ (Fu et al., 2025) (4M) 57.8 .205 .093 .072 40.1 .392 83.5 .811 TriForces MACE (Ramlaoui et al., 2026) (5M) 27.6 .159 .073 .050 35.1 .250 34.4 .876 TriForces eSEN (Ramlaoui et al., 2026) (5M) 19.5 .232 .058 .043 25.6 .139 20.2 .888 JMP-L (Shoghi et al., 2024) (120M) 20.6 .249 .059 .045 26.0 .091 10.1 – Compared with models trained from scratch, Atom-JEPA achieves the lowest error on the four smaller tasks, with improvements of up to 21%. On the larger MP regression tasks, coGN remains stronger, although pretraining improves our encoder by 12–15% on these tasks relative to training from scratch (Section 4.2). Among pretrained models with at most 4M pretraining structures, Atom-JEPA achieves the lowest error on 6/8 tasks, and outperforms Crystal Twins on all tasks. This holds even though MACE, Orb, and eSEN are pretrained on 4M structures with energy and force labels, while Atom-JEPA is pretrained on 1.7M structures without any labels. Comparison to TriForces (5M structures) highlights the importance of the backbone architecture: Atom-JEPA outperforms TriForces MACE on 7/8 tasks, whereas compared to TriForces eSEN it performs worse, with the exception of MP Is Metal (.908 versus .888 F1) and MP E Form, which is essentially tied (20.3 versus 20.2 meV/atom). JMP-L, pretrained with supervision on 120M structures, achieves lower MAE than Atom-JEPA on all seven regression tasks. 4.2 Ablation Studies Effect of Atom-JEPA pretraining We compare the Atom-JEPA pretrained encoder to a randomly initialized counterpart on both ADMET and Matbench, using the training procedures described in Sections H.1 and H.3. Atom-JEPA pretraining improves downstream performance for both molecular and crystalline tasks (Table 4). On the molecular ADMET benchmarks, pretraining reduces error by 9.6% on Biogen and 8.5% on ExpansionRx. ChEMBL-MT does not see an improvement, but we suspect that it is confounded by hyperparameters not generalizing well from Biogen to the much larger dataset size. On Matbench, pretraining improves all eight tasks and all 40 task–fold combinations, yielding mean foldwise relative MAE reductions of 10.1–14.5% across the seven regression tasks and increasing mean classification F1 from 0.9010 to 0.9083. These gains span datasets ranging from 1,265 to 132,752 samples and persist on the largest regression benchmarks: MP Gap and MP E Form improve by 14.5% and 12.0%, respectively. Even without pretraining, the encoder is competitive with established from-scratch models, matching or exceeding their performance on several tasks (Table 3 & Table 4) . This confirms that the gains from pretraining do not simply stem from a weak baseline. Figure 3: UMAP projections of Atom-JEPA representations for molecules annotated as belonging to one chemical family from the 32,252 molecules in the manually annotated three-star subset of ChEBI (Malik et al., 2026), colored by chemical family, and 50,000 randomly sampled crystals from the Alexandria pretraining dataset, colored by crystal system. Table 4: Effect of Atom-JEPA pretraining. Identical encoder architecture is compared with and without pretraining on identical official folds; values are means and standard deviations across the five folds or endpoints for ADMET tasks. Δ (rel.) is the mean fold-wise or endpoint relative improvement. “Impr.” counts folds or endpoints on which pretraining wins; p is a two-sided paired t-test over the five Matbench folds or mean MAE over the folds in the endpoints for ADMET tasks. Task (Units) #Samples No pretraining Atom-JEPA Δ (rel.) Impr.. p ADMET datasets Biogen 3,521 .3825.3825 .3458.3458 9.6%9.6\% 6/6 .011.011 ExpansionRX 7,608 .3673.3673 .3361.3361 8.5%8.5\% 9/9 <.001<.001 ChEMBL-MT 114,112 .4543.4543 .4567 −.5%-.5\% 12/25 .57 Matbench Phonons (cm-1) 1,265 26.9±4.026.9 [rgb]0.5,0.5,0.5 ± 4.0 23.3±4.523.3 [rgb]0.5,0.5,0.5 ± 4.5 13.5%13.5\% 5/5 .015.015 Dielectric (–) 4,764 .298±.084.298 [rgb]0.5,0.5,0.5 ±.084 .268±.074.268 [rgb]0.5,0.5,0.5 ±.074 10.1%10.1\% 5/5 .011.011 Log GVRH (log10 _10(GPa)) 10,987 .0773±.0010.0773 [rgb]0.5,0.5,0.5 ±.0010 .0670±.0010.0670 [rgb]0.5,0.5,0.5 ±.0010 13.3%13.3\% 5/5 <.001<.001 Log KVRH (log10 _10(GPa)) 10,987 .0539±.0034.0539 [rgb]0.5,0.5,0.5 ±.0034 .0473±.0021.0473 [rgb]0.5,0.5,0.5 ±.0021 12.3%12.3\% 5/5 .001.001 Perovskites (meV) 18,928 32.5±.632.5 [rgb]0.5,0.5,0.5 ±.6 29.1±1.029.1 [rgb]0.5,0.5,0.5 ± 1.0 10.5%10.5\% 5/5 <.001<.001 MP Gap (eV) 106,113 .1896±.0022.1896 [rgb]0.5,0.5,0.5 ±.0022 .1622±.0022.1622 [rgb]0.5,0.5,0.5 ±.0022 14.5%14.5\% 5/5 <.001<.001 MP E Form (meV/atom) 132,752 23.03±.2923.03 [rgb]0.5,0.5,0.5 ±.29 20.26±.1920.26 [rgb]0.5,0.5,0.5 ±.19 12.0%12.0\% 5/5 <.001<.001 MP Is Metal (F1) 106,113 .9010±.0021.9010 [rgb]0.5,0.5,0.5 ±.0021 .9083±.0024.9083 [rgb]0.5,0.5,0.5 ±.0024 .8%.8\% 5/5 <.001<.001 We hypothesize that these performance gains arise because Atom-JEPA pretraining encourages representations to organize according to high-level features that transfer well across downstream tasks in both domains. The UMAP visualizations in Fig. 3 suggest such organization may be by chemical family or crystal system, providing a qualitative indication consistent with this hypothesis. In contrast, an encoder trained from scratch starts from representations shaped by the inductive biases of the backbone architecture, which primarily organize systems according to size and composition initially. For a deeper analysis of the learned representations of Atom-JEPA in Appendix D. Pretraining objective Next, we examine how the choice of pretraining prediction objective affects downstream performance. In addition to the joint-objective encoder pretrained on the 19M molecules from Uni-Mol, we pretrain an encoder using only the atom-level objective on the same data. We also pretrain three encoders on the 130K molecules from the QM9 dataset using atom-only, substructure-only, and joint objectives. All encoders are evaluated using all-layer MLP probes on frozen features, on the 12 QM9 properties. With QM9 pretraining, atom-only achieves the lowest MAE on 10/12 properties, while substructure-only performs best on εLUMO _LUMO and the joint objective performs best on R2R^2 (Fig. 4a). With Uni-Mol pretraining, the joint objective improves over atom-only on 10/12 properties, ties on ZPVE, and performs worse only on μ (Fig. 4b). We additionally compare the two Uni-Mol-pretrained encoders on the six Biogen ADME tasks. The joint objective performs better on average, although atom-only remains competitive, performing better or similarly on 3/6 tasks (Fig. 4d). These results suggest that the benefit of combining atom-level and substructure-level objectives depends on both the pretraining dataset and the downstream task. Figure 4: Effect of pretraining objectives and dataset. Frozen-feature MLP probes across 12 QM9 targets. The horizontal axis shows relative MAE reduction to the baseline of each panel. Positive values indicate improvement. (a) Joint (Atom+substructure) and substructure-only objectives relative to atom-only pretraining on QM9. (b) The joint objective relative to atom-only pretraining on Uni-Mol. (c) Uni-Mol pretraining relative to QM9 pretraining, with the objective held fixed. (d) The joint objective relative to atom-only pretraining on Uni-Mol on Biogen ADME validation objective. Data scaling Increasing the pretraining dataset from 130k (QM9) to 19M molecules (Uni-Mol) improves both atom-only and joint-objective pretraining on 11/12 QM9 properties, with R2R^2 as the sole exception (Fig. 4c). The gains are particularly pronounced for the joint objective, including MAE reductions of 27% for Δε , 42.5% for μ, and 45.9% for H. Notably, Uni-Mol pretraining substantially improves performance even though QM9 pretraining already exactly matches the molecular domain of the downstream evaluation. While this comparison changes both dataset size and composition, the results suggest that scaling Atom-JEPA pretraining to larger and more chemically and structurally diverse datasets could further improve downstream performance. 5 Conclusion In this work, we proposed Atom-JEPA, a self-supervised pretraining framework for 3D atomic systems that combines atom-level and substructure-level prediction in latent space. Using structural data alone without expensive property labels nor molecular attributes as the pretraining signal, Atom-JEPA learns representations that transfer effectively to molecular ADMET, quantum-chemical, and crystalline materials property prediction. Ablation studies demonstrate consistent improvement over training the same encoder from scratch, and indicate potential for further increasing downstream performance by scaling to larger and more diverse pretraining data. Overall, Atom-JEPA establishes latent-space prediction from 3D structure as a promising approach to learning transferable representations for molecules and materials, reducing reliance on property-labeled pretraining data. Acknowledgements This work was supported by the Novo Nordisk Foundation under grant no NNF25OC0095542 by funding KHS, FRJC and MNS (AI-driven materials optimization for light trapping in thin-film solar cells). This work was supported by the Novo Nordisk Foundation (Grant ID: NNF25OC0105141) by funding RHT, MJ and MNS and via access to the Gefion AI Supercomputer, operated by the Danish Centre for AI Innovation (DCAI). Reproducibility statement To facilitate full reproducibility, we report all hyperparameters used in our pretraining and downstream fine-tuning experiments. The source code for pretraining and all finetuning experiments can be found at https://github.com/khelverskovp/atom-jepa together with experimental results across all folds. Pretrained model checkpoints can be found at https://huggingface.co/atom-jepa/atom-jepa. References Abramson et al. (2024) J. Abramson, J. Adler, J. Dunger, R. Evans, T. Green, A. Pritzel, O. Ronneberger, L. Willmore, A. J. Ballard, J. Bambrick, S. W. Bodenstein, D. A. Evans, C. Hung, M. O’Neill, D. Reiman, K. Tunyasuvunakool, Z. Wu, A. Žemgulytė, E. Arvaniti, C. Beattie, O. Bertolli, A. Bridgland, A. Cherepanov, M. Congreve, A. I. Cowen-Rivers, A. Cowie, M. Figurnov, F. B. Fuchs, H. Gladman, R. Jain, Y. A. Khan, C. M. R. Low, K. Perlin, A. Potapenko, P. Savy, S. Singh, A. Stecula, A. Thillaisundaram, C. Tong, S. Yakneen, E. D. Zhong, M. Zielinski, A. Žídek, V. Bapst, P. Kohli, M. Jaderberg, D. Hassabis, and J. M. Jumper Accurate structure prediction of biomolecular interactions with alphafold 3. Nature 630 (8016), p. 493–500. External Links: Document, ISBN 1476-4687, Link Cited by: §1. Adrian et al. (2025) M. Adrian, Y. Chung, K. Boyd, S. Paliwal, S. P. Veccham, and A. C. Cheng Multitask finetuning and acceleration of chemical pretrained models for small molecule drug property prediction. CoRR abs/2510.12719. External Links: Link, Document, 2510.12719 Cited by: Table 5, Table 17, §H.1, §H.1.1, item 1, §4.1, Table 1. Agrawal et al. (2022) K. K. Agrawal, A. K. Mondal, A. Ghosh, and B. Richards α-req : assessing representation quality in self-supervised learning by measuring eigenspectrum decay. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 17626–17638. External Links: Document, Link Cited by: §G.2. Ariguib et al. (2026) B. Ariguib, M. Niepert, and A. Manolache Learning the neighborhood: contrast-free multimodal self-supervised molecular graph pretraining. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §2. Assran et al. (2023) M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas Self-supervised learning from images with a joint-embedding predictive architecture. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 15619–15629. Cited by: Appendix A, §3. Aykent and Xia (2025) S. Aykent and T. Xia GotenNet: rethinking efficient 3d equivariant graph neural networks. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §H.2, Table 2. Bai et al. (2025) R. Bai, Y. Yao, Q. Lin, L. Wu, Z. Li, H. Wang, M. Ma, D. Mu, L. Hu, H. Yang, W. Li, S. Zhu, X. Wu, X. Rui, and Y. Yu Preferable single-atom catalysts enabled by natural language processing for high energy density Na-S batteries. Nature Communications 16 (1), p. 5827. External Links: Document, ISSN 2041-1723, Link Cited by: §1. Batatia et al. (2022) I. Batatia, D. P. Kovacs, G. N. C. Simm, C. Ortner, and G. Csanyi MACE: higher order equivariant message passing neural networks for fast and accurate force fields. In Advances in Neural Information Processing Systems, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), External Links: Link Cited by: §1, Table 2, Table 3. Cavignac et al. (2026) T. Cavignac, J. Schmidt, P. De Breuck, A. Loew, T. F. T. Cerqueira, H. Wang, A. Bochkarev, Y. Lysogorskiy, A. H. Romero, R. Drautz, S. Botti, and M. A. L. Marques AI-driven expansion and application of the Alexandria database. Journal of Physics: Materials 9 (2), p. 025014. External Links: Document, Link Cited by: Table 5, §4. Chai Discovery Team et al. (2025) Chai Discovery Team, J. Boitreaud, J. Dent, D. Geisz, M. McPartlon, J. Meier, Z. Qiao, A. Rogozhnikov, N. Rollins, P. Wollenhaupt, and K. Wu Zero-shot antibody design in a 24-well plate. bioRxiv. External Links: Document, Link, https://w.biorxiv.org/content/early/2025/07/06/2025.07.05.663018.full.pdf Cited by: §1. Chiang et al. (2022) C. Chiang, Y. Chuang, and H. Lee Recent advances in pre-trained language models: why do they work and how do they work. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing: Tutorial Abstracts, M. A. Alonso and Z. Wei (Eds.), Taipei, p. 8–15. External Links: Link, Document Cited by: §1. Choudhary and DeCost (2021) K. Choudhary and B. DeCost Atomistic line graph neural network for improved materials property predictions. npj Computational Materials 7 (1). External Links: ISSN 2057-3960, Link, Document Cited by: Table 3. Cornet et al. (2024) F. R. J. Cornet, G. Bartosh, M. N. Schmidt, and C. A. Naesseth Equivariant neural diffusion for molecule generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1. Dunn et al. (2020) A. Dunn, Q. Wang, A. Ganose, D. Dopp, and A. Jain Benchmarking materials property prediction methods: the matbench test set and automatminer reference algorithm. npj Computational Materials 6 (1). External Links: ISSN 2057-3960, Link, Document Cited by: Table 5, §4. Fang et al. (2023) C. Fang, Y. Wang, R. Grater, S. Kapadnis, C. Black, P. Trapa, and S. Sciabola Prospective validation of machine learning algorithms for absorption, distribution, metabolism, and excretion prediction: an industrial perspective. Journal of Chemical Information and Modeling 63 (11), p. 3263–3274. External Links: ISSN 1549-9596, Document, Link, https://pubs.acs.org/jcisd8/article-pdf/63/11/3263/8908637/ci3c00160.pdf Cited by: Table 5, §H.1.1, item 3. Feng et al. (2023) S. Feng, Y. Ni, Y. Lan, Z. Ma, and W. Ma Fractional denoising for 3D molecular pre-training. In Proceedings of the 40th International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.), Proceedings of Machine Learning Research, Vol. 202, p. 9938–9961. External Links: Link Cited by: §2, Table 2, Table 2. Fu et al. (2025) X. Fu, B. M. Wood, L. Barroso-Luque, D. S. Levine, M. Gao, M. Dzamba, and C. L. Zitnick Learning smooth and expressive interatomic potentials for physical property prediction. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §1, Table 3. Garrido et al. (2023) Q. Garrido, R. Balestriero, L. Najman, and Y. Lecun RankMe: assessing the downstream performance of pretrained self-supervised representations by their rank. External Links: 2210.02885, Link Cited by: §G.2. Gilmer et al. (2017) J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, p. 1263–1272. Cited by: §1. Gómez-Bombarelli et al. (2018) R. Gómez-Bombarelli, J. N. Wei, D. Duvenaud, J. M. Hernández-Lobato, B. Sánchez-Lengeling, D. Sheberla, J. Aguilera-Iparraguirre, T. D. Hirzel, R. P. Adams, and A. Aspuru-Guzik Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science 4 (2), p. 268–276. External Links: ISSN 2374-7951, Link, Document Cited by: §2. Graff et al. (2026) D. E. Graff, N. K. Morgan, J. W. Burns, A. C. Doner, B. Li, S. Li, J. Manu, A. Menon, H. Pang, H. Wu, et al. Chemprop v2: an efficient, modular machine learning package for chemical property prediction. Journal of Chemical Information and Modeling 66 (1), p. 28–33. Cited by: Table 1. Hollmann et al. (2022) N. Hollmann, S. Müller, K. Eggensperger, and F. Hutter TabPFN: a transformer that solves small tabular classification problems in a second. In International Conference on Learning Representations, External Links: Link Cited by: §1. Hoogeboom et al. (2022) E. Hoogeboom, V. G. Satorras, C. Vignac, and M. Welling Equivariant diffusion for molecule generation in 3d. In Thirty-ninth International Conference on Machine Learning, External Links: 2203.17003, Link Cited by: §1. Hu et al. (2020) W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. Pande, and J. Leskovec Strategies for pre-training graph neural networks. In International Conference on Learning Representations, External Links: Link Cited by: §2. Huang et al. (2021) K. Huang, T. Fu, W. Gao, Y. Zhao, Y. H. Roohani, J. Leskovec, C. W. Coley, C. Xiao, J. Sun, and M. Zitnik Therapeutics data commons: machine learning datasets and tasks for drug discovery and development. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, J. Vanschoren and S. Yeung (Eds.), External Links: Link Cited by: Table 5, §4.1. Ivković et al. (2024) Ž. Ivković, J. Jover, and J. Harvey Transfer learning based on atomic feature extraction for the prediction of experimental 13c chemical shifts. Digital Discovery 3 (11), p. 2242–2251. External Links: ISSN 2635-098X, Document, Link, https://pubs.rsc.org/d/article-pdf/3/11/2242/9678230/d4d00168k.pdf Cited by: §2. Jiao et al. (2023) R. Jiao, J. Han, W. Huang, Y. Rong, and Y. Liu Energy-motivated equivariant pretraining for 3d molecular graphs. AAAI’23/IAAI’23/EAAI’23. External Links: ISBN 978-1-57735-880-0, Link, Document Cited by: Table 2. Kavanagh et al. (2026) S. R. Kavanagh, C. W. Tan, M. Wang, M. L. Descoteaux, G. de Miranda Nascimento, U. Unneberg, L. Zichi, F. Libbi, N. Rivano, A. Glover, V. Bharadwaj, A. Johansson, W. C. Witt, A. Musaelian, and B. Kozinsky Fast and accurate equivariant foundation models for atomistic simulation. External Links: 2607.28461, Link Cited by: §1. Kellenberger et al. (2007) E. Kellenberger, J. Springael, M. Parmentier, M. Hachet-Haas, J. Galzi, and D. Rognan Identification of Nonpeptide CCR5 Receptor Agonists by Structure-based Virtual Screening. Journal of Medicinal Chemistry 50 (6), p. 1294–1303. External Links: Document, ISSN 0022-2623, Link Cited by: §1. Kendall et al. (2018) A. Kendall, Y. Gal, and R. Cipolla Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, p. 7482–7491. External Links: Link, Document Cited by: §H.1. Kim et al. (2026) S. Y. Kim, Y. J. Park, and J. Li Leveraging neural network interatomic potentials for a foundation model of chemistry. npj Computational Materials. External Links: Document, ISBN 2057-3960, Link Cited by: §2. Koleiev et al. (2026) I. Koleiev, R. Stratiichuk, N. Shevchuk, M. Melnychenko, A. Nyporko, D. Todoryshyn, V. Husak, S. Starosyla, S. Yesylevskyy, and A. Nafiiev An end-user audit of reproducibility, data leakage, and overfitting of the top-ranked admet prediction models in tdc leaderboards. Journal of Chemical Information and Modeling. Cited by: Table 20, Table 21, Table 22, Table 23, Table 24, §4.1. Kong et al. (2025) L. Kong, N. Shoghi, G. Hu, P. Li, and V. Fung MatterTune: an integrated, user-friendly platform for fine-tuning atomistic foundation models to accelerate materials simulation and discovery. Digital Discovery 4 (8), p. 2253–2262. External Links: ISSN 2635-098X, Document, Link, https://pubs.rsc.org/d/article-pdf/4/8/2253/10332376/d5d00154d.pdf Cited by: §2. LeCun (2022) Y. LeCun A path towards autonomous machine intelligence. External Links: Link Cited by: §1, §2. Levine et al. (2026) D. S. Levine, M. Shuaibi, E. W. C. Spotte-Smith, M. G. Taylor, M. R. Hasyim, K. Michel, I. Batatia, G. Csányi, M. Dzamba, P. Eastman, N. C. Frey, X. Fu, V. Gharakhanyan, A. S. Krishnapriyan, J. A. Rackers, S. Raja, A. Rizvi, A. S. Rosen, Z. Ulissi, S. Vargas, C. L. Zitnick, S. M. Blau, and B. M. Wood The open molecules 2025 (omol25) dataset, evaluations, and models. External Links: 2505.08762, Link Cited by: §2. Li et al. (2022) H. Li, D. Zhao, and J. Zeng KPGT: knowledge-guided pre-training of graph transformer for molecular property prediction. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, p. 857–867. External Links: Link, Document Cited by: §2. Liao et al. (2026) Y. Liao, A. J. Hoffman, S. C. Shen, A. Duval, S. W. Norwood, and T. Smidt EquiformerV3: scaling efficient, expressive, and general se(3)-equivariant graph attention transformers. External Links: 2604.09130, Link Cited by: §G.1, §4. Liao et al. (2024) Y. Liao, B. M. Wood, A. Das, and T. Smidt EquiformerV2: improved equivariant transformer for scaling to higher-degree representations. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: Table 2. Liu et al. (2026) N. Liu, N. Kazeev, S. G. Dale, A. Maevskiy, Y. Zeng, R. Kubo, P. Huang, T. Laurent, Y. LeCun, K. S. Novoselov, and X. Bresson Crys-jepa: accelerating crystal discovery via embedding screening and generative refinement. External Links: 2605.14759, Link Cited by: §2. Liu et al. (2023) S. Liu, H. Guo, and J. Tang Molecular geometry pretraining with SE(3)-invariant denoising distance matching. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: Table 2. Liu et al. (2025) Y. Liu, J. Chen, R. Jiao, J. Li, W. Huang, and B. Su DenoiseVAE: learning molecule-adaptive noise distributions for denoising-based 3d molecular pre-training. In International Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025, p. 22078–22101. External Links: Link Cited by: §2, Table 2. Luo et al. (2023) S. Luo, T. Chen, Y. Xu, S. Zheng, T. Liu, L. Wang, and D. He One transformer can understand both 2d & 3d molecular data. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: Table 2. MacDermott-Opeskin and Castellanos (2026) H. MacDermott-Opeskin and M. Castellanos Announcement 1: expansionrx-openadmet blind challenge. Zenodo. External Links: Document, Link Cited by: Table 5, item 2. Magar et al. (2022) R. Magar, Y. Wang, and A. Barati Farimani Crystal Twins: self-supervised learning for crystalline material property prediction. npj Computational Materials 8 (1), p. 231. External Links: Document Cited by: Table 3. Malik et al. (2026) A. Malik, M. Arsalan, C. Moreno, J. Mosquera, E. Félix, T. Kizilören, V. Muthukrishnan, B. Zdrazil, A. R. Leach, and N. M. O’Boyle ChEBI: re-engineered for a sustainable future. Nucleic Acids Research 54 (D1), p. D1768–D1778. External Links: ISSN 1362-4962, Document, Link, https://academic.oup.com/nar/article-pdf/54/D1/D1768/65596350/gkaf1271.pdf Cited by: §D.1.2, Appendix D, Figure 3. Morehead et al. (2026) A. Morehead, M. Cretu, A. Panescu, R. Anand, M. Weiler, T. Perez, S. M. Blau, S. Farrell, W. Bhimji, A. Jain, H. Sahasrabuddhe, P. Lio, T. Jaakkola, R. Gomez-Bombarelli, R. Ying, N. B. Erichson, and M. W. Mahoney Zatom-1: a multimodal flow foundation model for 3d molecules and materials. In ICLR 2026 Workshop on Foundation Models for Science: Real-World Impact and Science-First Design, External Links: Link Cited by: §2. Neumann et al. (2024) M. Neumann, J. Gin, B. Rhodes, S. Bennett, Z. Li, H. Choubisa, A. Hussey, and J. Godwin Orb: a fast, scalable neural network potential. External Links: 2410.22570, Link Cited by: Table 3. Ni et al. (2024a) Y. Ni, S. Feng, X. Hong, Y. Sun, W. Ma, Z. Ma, Q. Ye, and Y. Lan Pre-training with fractional denoising to enhance molecular property prediction. Nature Machine Intelligence 6 (10), p. 1169–1178. Cited by: Table 2. Ni et al. (2024b) Y. Ni, S. Feng, W. Ma, Z. Ma, and Y. Lan Sliced denoising: a physics-informed molecular pre-training method. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: Table 2. Perez and Gomez-Bombarelli (2026) T. Perez and R. Gomez-Bombarelli Self-conditioned denoising for atomistic representation learning. External Links: 2603.17196, Link Cited by: §2. Piccoli et al. (2026) F. Piccoli, G. Vogel, and J. M. Weber Joint embedding predictive architecture for self-supervised pretraining on polymer molecular graphs. Digital Discovery 5 (2), p. 819–834. External Links: ISSN 2635-098X, Document, Link, https://pubs.rsc.org/d/article-pdf/5/2/819/10334612/d5d00308c.pdf Cited by: §2. Pillai et al. (2023) H. S. Pillai, Y. Li, S. Wang, N. Omidvar, Q. Mu, L. E. K. Achenie, F. Abild-Pedersen, J. Yang, G. Wu, and H. Xin Interpretable design of Ir-free trimetallic electrocatalysts for ammonia oxidation with graph neural networks. Nature Communications 14 (1), p. 792. External Links: Link, Document, ISSN 2041-1723 Cited by: §1. Pyzer-Knapp et al. (2025) E. O. Pyzer-Knapp, M. Manica, P. Staar, L. Morin, P. Ruch, T. Laino, J. R. Smith, and A. Curioni Foundation models for materials discovery –current state and future directions. npj Computational Materials 11 (1), p. 61. External Links: Document, ISBN 2057-3960, Link Cited by: §1. Qu et al. (2025) J. Qu, D. Holzmüller, G. Varoquaux, and M. L. Morvan TabICL: a tabular foundation model for in-context learning on large data. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §1. Ramakrishnan et al. (2014) R. Ramakrishnan, P. O. Dral, M. Rupp, and O. A. von Lilienfeld Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data 1, p. 140022. External Links: Document Cited by: Table 5, §4. Ramlaoui et al. (2026) A. Ramlaoui, A. Duval, H. Bull, V. Schmidt, H. Talbot, F. D. Malliaros, and J. Musielewicz TriForces: augmenting atomistic GNNs for transferable representations. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: Table 2, Table 3, Table 3, Table 3. Rhodes et al. (2025) B. Rhodes, S. Vandenhaute, V. Šimkus, J. Gin, J. Godwin, T. Duignan, and M. Neumann Orb-v3: atomistic simulation at scale. External Links: 2504.06231, Link Cited by: §1. Rogers and Hahn (2010) D. Rogers and M. Hahn Extended-connectivity fingerprints. Journal of Chemical Information and Modeling 50 (5), p. 742–754. External Links: ISSN 1549-9596, Document, Link, https://pubs.acs.org/jcisd8/article-pdf/50/5/742/11822562/ci100050t.pdf Cited by: §1. Rottach et al. (2026) F. Rottach, S. Schieferdecker, W. Rudman, R. Balestriero, and C. Eickhoff Mol-jepa: a multimodal joint embedding predictive architecture for molecules. External Links: 2608.22642, Link Cited by: §2, Table 1. Ruff et al. (2024) R. Ruff, P. Reiser, J. Stühmer, and P. Friederich Connectivity optimized nested line graph networks for crystal structures. Digital Discovery 3 (3), p. 594–601. External Links: ISSN 2635-098X, Document, Link, https://pubs.rsc.org/d/article-pdf/3/3/594/9675919/d4d00018h.pdf Cited by: Table 3. Schüt et al. (2017) K. T. Schüt, P.-J. Kindermans, H. E. Sauceda, S. Chmiela, A. Tkatchenko, and K.-R. Müller SchNet: a continuous-filter convolutional neural network for modeling quantum interactions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Red Hook, NY, USA, p. 992–1002. External Links: ISBN 9781510860964 Cited by: §A.3.3, §1. Schüt et al. (2018) K. T. Schüt, H. E. Sauceda, P. Kindermans, A. Tkatchenko, and K. Müller Schnet–a deep learning architecture for molecules and materials. The Journal of chemical physics 148 (24). Cited by: §H.2. Schüt et al. (2021) K. T. Schüt, O. T. Unke, and M. Gastegger Equivariant message passing for the prediction of tensorial properties and molecular spectra. In Thirty-eighth International Conference on Machine Learning, External Links: 2102.03150, Link Cited by: §H.2, §1. Shoghi et al. (2024) N. Shoghi, A. Kolluru, J. R. Kitchin, Z. W. Ulissi, C. L. Zitnick, and B. M. Wood From molecules to materials: pre-training large generalizable models for atomic property prediction. In The Twelfth International Conference on Learning Representations, External Links: Link Cited by: §2, Table 2, Table 3. Siméoni et al. (2026) O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoa, F. Massa, D. Haziza, L. Wehrstedt, J. Wang, T. Darcet, T. Moutakanni, L. Sentana, C. Roberts, A. Vedaldi, J. Tolan, J. Brandt, C. Couprie, J. Mairal, H. Jégou, P. Labatut, and P. Bojanowski DINOv3. Transactions on Machine Learning Research. Note: Featured Certification External Links: ISSN 2835-8856, Link Cited by: §1. Skenderi et al. (2025) G. Skenderi, H. Li, J. Tang, and M. Cristani Graph-level representation learning with joint-embedding predictive architectures. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §2. Sun et al. (2024) J. Sun, R. Tu, Y. Xu, H. Yang, T. Yu, D. Zhai, X. Ci, and W. Deng Machine learning aided design of single-atom alloy catalysts for methane cracking. Nature Communications 15 (1), p. 6036. External Links: Link Cited by: §1. Thomas et al. (2018) N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and P. Riley Tensor field networks: rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219. Cited by: §A.3.3. Tompa et al. (2026) T. L. Tompa, E. Varga-Umbrich, I. Batatia, A. M. Elena, N. Bernstein, and G. Csányi Fine-tuning mlip foundation models: strategies for accuracy and transferability. arXiv preprint arXiv:2606.12704. Cited by: §H.1. Wang et al. (2023) Z. Wang, Z. Feng, Y. Li, B. Li, Y. Wang, C. Sha, M. He, and X. Li BatmanNet: bi-branch masked graph transformer autoencoder for molecular representation. Briefings in Bioinformatics 25 (1). External Links: ISSN 1477-4054, Link, Document Cited by: §2. Weininger (1988) D. Weininger SMILES, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences 28 (1), p. 31–36. External Links: ISSN 0095-2338, Document, Link, https://pubs.acs.org/jcisd8/article-pdf/28/1/31/10787984/ci00057a005.pdf Cited by: §1. Wohlwend et al. (2025) J. Wohlwend, G. Corso, S. Passaro, N. Getz, M. Reveiz, K. Leidal, W. Swiderski, L. Atkinson, T. Portnoi, I. Chinn, J. Silterra, T. Jaakkola, and R. Barzilay Boltz-1 democratizing biomolecular interaction modeling. bioRxiv. External Links: Document, Link, https://w.biorxiv.org/content/early/2025/05/06/2024.11.19.624167.full.pdf Cited by: §1. Wong et al. (2024) F. Wong, E. J. Zheng, J. A. Valeri, N. M. Donghia, M. N. Anahtar, S. Omori, A. Li, A. Cubillos-Ruiz, A. Krishnan, W. Jin, A. L. Manson, J. Friedrichs, R. Helbig, B. Hajian, D. K. Fiejtek, F. F. Wagner, H. H. Soutter, A. M. Earl, J. M. Stokes, L. D. Renner, and J. J. Collins Discovery of a structural class of antibiotics with explainable deep learning. Nat. 626 (7997), p. 177–185. External Links: Link, Document Cited by: §1. Wood et al. (2026) B. Wood, M. Dzamba, X. Fu, M. Gao, M. Shuaibi, L. Barroso-Luque, K. Abdelmaqsoud, V. Gharakhanyan, J. Kitchin, D. Levine, et al. UMA: a family of universal models for atoms. In Advances in Neural Information Processing Systems, Vol. 38, p. 129391–129427. Cited by: §H.1.3, §1. Wu et al. (2025) C. Wu, Z. Liu, W. Shu, L. Wang, Y. Luo, W. Lei, Y. Bian, J. Fang, and X. Wang 3D-GSRD: 3d molecular graph auto-encoder with selective re-mask decoding. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §2. Xia et al. (2023) J. Xia, C. Zhao, B. Hu, Z. Gao, C. Tan, Y. Liu, S. Li, and S. Z. Li Mole-BERT: rethinking pre-training graph neural networks for molecules. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §2. Xie and Grossman (2018) T. Xie and J. C. Grossman Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical Review Letters 120 (14). External Links: ISSN 1079-7114, Link, Document Cited by: Table 3. Xue et al. (2026) Y. Xue, S. P. Veccham, S. Paliwal, T. Shimko, and M. Livne Probabilistic contrastive pretraining for multi-task ADME property prediction. CoRR abs/2606.11508. External Links: Link, Document, 2606.11508 Cited by: §C.1, Table 16, Table 18, Table 19, §H.1, §H.1, §2, §4.1, §4.1, Table 1. Xuhong et al. (2018) L. Xuhong, Y. Grandvalet, and F. Davoine Explicit inductive bias for transfer learning with convolutional networks. In International conference on machine learning, p. 2825–2834. Cited by: §H.1. Yuan et al. (2026) E. C. Yuan, Y. Liu, J. Chen, P. Zhong, S. Raja, T. Kreiman, S. Vargas, W. Xu, M. Head-Gordon, C. Yang, S. M. Blau, B. Cheng, A. Krishnapriyan, and T. Head-Gordon Foundation models for atomistic simulation of chemistry and materials. Nature Reviews Chemistry 10 (3), p. 212–230. External Links: Document, Link Cited by: §1. Zaidi et al. (2023) S. Zaidi, M. Schaarschmidt, J. Martens, H. Kim, Y. W. Teh, A. Sanchez-Gonzalez, P. Battaglia, R. Pascanu, and J. Godwin Pre-training via denoising for molecular property prediction. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §2, Table 2. Zeni et al. (2025) C. Zeni, R. Pinsler, D. Zügner, A. Fowler, M. Horton, X. Fu, Z. Wang, A. Shysheya, J. Crabbé, S. Ueda, R. Sordillo, L. Sun, J. Smith, B. Nguyen, H. Schulz, S. Lewis, C. Huang, Z. Lu, Y. Zhou, H. Yang, H. Hao, J. Li, C. Yang, W. Li, R. Tomioka, and T. Xie A generative model for inorganic materials design. Nature 639 (8055), p. 624–632. External Links: Document, ISBN 1476-4687, Link Cited by: §1. Zhou et al. (2023) G. Zhou, Z. Gao, Q. Ding, H. Zheng, H. Xu, Z. Wei, L. Zhang, and G. Ke Uni-mol: a universal 3d molecular representation learning framework. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: Table 5, §H.1.3, §2, §4. Zhu et al. (2022) J. Zhu, Y. Xia, L. Wu, S. Xie, T. Qin, W. Zhou, H. Li, and T. Liu Unified 2d and 3d pre-training of molecular representations. In KDD 2022, External Links: Link Cited by: §2. Zhuang et al. (2014) C. Zhuang, S. Narayanapillai, W. Zhang, Y. Y. Sham, and C. Xing Rapid Identification of Keap1–Nrf2 Small-Molecule Inhibitors through Structure-Based Virtual Screening and Hit-Based Substructure Search. Journal of Medicinal Chemistry 57 (3), p. 1121–1126. External Links: Document, ISSN 0022-2623, Link Cited by: §1. Appendix Contents Appendix A Detailed Methodology The overall idea of Atom-JEPA is analogous to I-JEPA for images (Assran et al., 2023), but redesigned for atomic systems in 3-dimensional space: Given part of a molecule or crystal (context view), predict the representation of the remaining part (target view). We consider two prediction objectives: (1) an atom-level objective that predicts the feature representation of each atom in the target view, and (2) a substructure-level objective that predicts the mean-pooled representation of all target atoms. For the context encoder, target encoder and predictor, we use a 3D geometric graph neural network (GNN). We only use data that are either in ground-state or close to local minima of the potential energy surface. We posit that using data in off-equilibrium states (either rattled systems or from MD trajectories) without explicitly telling the model something about the stability of the system (e.g. via total energy of the system) could make the learning problem much more ambigious. A.1 Atomic 3D Graph Representation and Partitioning We represent atomic systems as 3D graphs and partition each system into complementary context and target subgraphs. An atomic system containing N atoms can be represented by a 3D graph G=(V,E)G=(V,E) with the node set V=(zi,i)i=1NV=\(z_i,r_i)\^N_i=1, where ziz_i denotes the atom type of atom i and i∈ℝ3r_i ^3 its 3D position. The edge set E encodes spatial proximity between atoms within a cutoff radius rcr_c. Given a partition of the atomic indices into a context set ℐxI_x and a target set ℐyI_y, we define the corresponding induced subgraphs as Gx=G[ℐx]=(Vx,Ex),Gy=G[ℐy]=(Vy,Ey),G_x=G[I_x]=(V_x,E_x), G_y=G[I_y]=(V_y,E_y), where Vx=(zi,i):i∈ℐxV_x=\(z_i,r_i):i _x\ and Vy=(zi,i):i∈ℐy.V_y=\(z_i,r_i):i _y\. The two index sets form a partition of the atoms in the atomic system, such that ℐx∩ℐy=∅I_x _y= and ℐx∪ℐy=1,…,NI_x _y=\1,…,N\. Molecular Graphs. For a molecule containing N atoms, the neighborhood of atom i is defined by all atoms within the cutoff radius rcr_c, (i)=j∈1,…,N∖i∣‖j−i‖<rc.N(i)=\j∈\1,…,N\ \i\ \|r_j-r_i\|<r_c\. The edge set E is obtained from these neighborhoods. For each neighboring pair (i,j)(i,j), we define the relative displacement vector ij=j−ir_ij=r_j-r_i, together with the interatomic distance ‖ij‖\|r_ij\| and unit direction ^ij=ij‖ij‖. r_ij= r_ij\|r_ij\|. Crystal Graphs. Crystalline systems are represented analogously, with the additional requirement that periodic images of atoms in neighboring unit cells are taken into account. Let the lattice be specified by three lattice vectors 1,2,3l_1,l_2,l_3, collected in the lattice matrix L. A periodic image of atom i, indexed by the integer translation vector ∈ℤ3n ^3, has position i+.r_i+nL. The periodic neighborhood of atom i is therefore (i)=(j,)∈1,…,N×ℤ3| 0<‖j+−i‖<rc.N(i)=\(j,n)∈\1,…,N\×Z^3\; |\;0<\|r_j+nl-r_i\|<r_c\. Although the underlying crystal is infinite, the graph contains only the N atoms of the reference unit cell as nodes. Periodicity is represented through edges connecting these atoms to neighboring periodic images within the cutoff radius rcr_c. Similarly to molecules, for each such edge, we define the periodic displacement vector ij=j+−i,r_ijn=r_j+nL-r_i, along with the distance ‖ij‖\|r_ijn\| and unit direction ^ij r_ijn. Context and Target Partitioning. Given an atomic system, we partition it into the context and target index sets ℐxI_x and ℐyI_y introduced above by partitioning the atoms using a k-EgoNet subgraph. We first sample an anchor atom c∼Unif(1,…,N)c (\1,…,N\) and a hop count k∼Unit(K)k (K) from a predefined set of hop counts K. The k-hop EgoNet centered at c is then defined by the index set k(c)=i∈1,…,N∣d(c,i)≤k,N_k(c)=\i∈\1,…,N\ d(c,i)≤ k\, where d(c,i)d(c,i) denotes the shortest-path distance between atoms c and i, measured in number of edges. We take the atoms of the sampled EgoNet k(c)N_k(c) as the context graph GxG_x, and the atoms in the complementary part as the target graph GyG_y. If the sampled value of k produces an EgoNet containing all atoms in the system, we instead use the largest value in K for which the target graph remains non-empty. During training, the two subgraphs are used symmetrically: each subgraph acts once as the context GxG_x and once as the target GyG_y, and the loss is evaluated in both directions. The anchor atom c and hop count k are resampled at each training epoch, yielding different partitions of the atomic system throughout training. To encourage spatially local partitions, the graph used to determine the k-hop EgoNet is constructed using a smaller cutoff radius rk≪rc.r_k r_c. This auxillary graph is used only to determine the partition (ℐx,ℐy)(I_x,I_y). Once the partition has been obtained, the full graph G, context graph GxG_x, and target graph GyG_y are constructed using the larger interaction cutoff rcr_c. Thus, rkr_k controls the locality of the sampled partition, whereas rcr_c determines the atomic interactions represented in the graphs provided to the models. A.2 Context and Target Encoders The context encoder fθf_θ and target encoder fθ¯f_ θ are both graph neural networks that take a 3D graph as input and produce atom-level feature representations. We use an equivariant architecture and write i=i(l)l=0Lmaxs_i=\s_i^(l)\^L_max_l=0 to denote atomic features, where i(0)s^(0)_i denote the invariant scalar features and i(l),l>0s^(l)_i,l>0 denote the higher-order equivariant features from l=1,…,Lmaxl=1,…,L_max. The parameters of the context encoder θ are updated via gradient based optimization during training, while the parameters of the target encoder θ¯ θ are an exponential moving average of the context encoder parameters: θ¯←τθ¯+(1−τ)θ θ←τ θ+(1-τ)θ, where τ∈[0,1]τ∈[0,1] controls the rate of the exponential moving average. The context encoder is given only the context graph GxG_x and produces a representation for each context atom, x=fθ(Gx)=xii∈ℐx.s_x=f_θ(G_x)=\s_x_i\_i _x. In contrast, the target encoder operates on the full atomic graph G. It therefore produces representations for all N atoms, ~y=fθ¯(G)=~yii=1N. s_y=f_ θ(G)=\ s_y_i\_i=1^N. After encoding the full graph, only the representations corresponding to atoms in the target subgraph are retained as prediction targets, y=~yii∈ℐy.s_y=\ s_y_i\_i _y. Thus, although the learning objective is defined only over atoms belonging to the target graph GyG_y, their target representations are computed from the full atomic system. Each target representation can therefore incorporate information from the surrounding atomic environment through the message-passing operations of the target encoder, rather than being restricted to information contained within GyG_y alone. We argue that this provides richer target representations that encode the target atoms in the context of the complete atomic system. A.3 Prediction Given the context representations xs_x, a shallow graph neural network gϕg_φ predicts representations produced by the target encoder. The predictor consists of a GNN body followed by objective-specific readout heads. Since the GNN body of the predictor operates on the atom features produced by the context encoder it does not contain an embedding layer. We consider two prediction objectives: a substructure objective that predicts a global representation of the target subgraph and an atom-level objective that predicts the representation of each individual target atom. First, the GNN body of the predictor produces further refined context-atom features xh_x, which are then mean-pooled and passed to the respective readout heads: x=gϕGNN(Gx,x),¯x=1|ℐx|∑i∈ℐxxi.h_x=g_φ^GNN(G_x,s_x), h_x= 1|I_x| _i _xh_x_i. A.3.1 Substructure Target Prediction In the substructure objective, the target is the mean-pooled invariant scalar features produced by the target encoder indexed over the target subgraph ¯y=1|ℐy|∑i∈ℐyyi(0), s_y= 1|I_y| _i _ys_y_i^(0), and the substructure readout head is a two-layer MLP with SiLU activation applied to the invariant component of the pooled context ^y=Rϕsub(¯x)=MLP(¯x(0)), s_y=R^sub_φ( h_x)=MLP( h_x^(0)), mapping into the feature space of the target encoder. A.3.2 Atom-level Target Prediction The atom-level objective instead produces one prediction ^yi s_y_i for every atom i∈ℐyi _y, which is compared against the corresponding target encoder representation yis_y_i. In addition to the context features, the atom-level readout head is provided with a positional encoding iq_i of the target atom’s position relative to the context subgraph: ^yi=Rϕatom(x,i),i∈ℐy. s_y_i=R^atom_φ(h_x,q_i), i _y. We construct the positional encoding so that it has the same degree structure as the atom features, with each i(l)q^(l)_i transforming under rotation exactly as x(l)h_x^(l) does. For each degree l≥1l≥ 1 we then take the inner product of the two over the 2l+12l+1 components seperately in each feature channel and concatenate them with the invariant parts to form i=[¯x(0)‖i(0)‖⟨¯x(1),i(1)⟩‖⋯‖⟨¯x(Lmax),i(Lmax)⟩]∈ℝC(Lmax+2),u_i= [ h_x^(0)\|q_i^(0)\| h^(1)_x,q^(1)_i \|·s\| h^(L_max)_x,q^(L_max)_i ] ^C(L_max+2), where ⋅|⋅·\|· denotes concatenation and ⟨¯x(l),i(l)⟩∈ℝC h_x^(l),q_i^(l) ^C denote the channel-wise inner product, with c-th entry ⟨¯x(l),i(l)⟩c=∑m=−llh¯x,mc(l)qi,mc(l). h_x^(l),q^(l)_i _c= _m=-l^l h^(l)_x,mcq_i,mc^(l). The invariant term ¯i(0) h^(0)_i carries only the content of the context, the term i(0)q^(0)_i depends only on the distance to the target atom, and the remaining terms couple the directional components of the context to the direction of the query. A two-layer MLP maps this feature into the target encoder’s feature space ^yi=MLP(i). s_y_i=MLP(u_i). A.3.3 Equivariant Relative Positional Encoding We encode target positions relative to the context using a radial-spherical-harmonic construction following Tensor Field Networks (Thomas et al., 2018), with distances expanded in a Gaussian radial basis as in SchNet (Schüt et al., 2017). For molecules, we define the position of each target atom i relative to the context graph. First, the centroid of the context atoms is computed as x=1|ℐx|∑j∈ℐxj.r_x= 1|I_x| _j _xr_j. The relative displacement of target atom i from the context centroid is then xi=i−xr_xi=r_i-r_x. We expand the distance ‖xi‖\|r_xi\| in a Gaussian radial basis φ(‖xi‖)∈ℝK (\|r_xi\|) ^K, with φk(‖xi‖)=exp(−(‖xi‖−μk)22σ2), _k(\|r_xi\|)= ( -(\|r_xi\|- _k)^22σ^2 ), where the centers μk _k are spaced over [0,rq][0,r_q]. The direction ^xi=xi/‖xi‖ r_xi=r_xi/\|r_xi\| is encoded using spherical harmonics, giving qi,mc(l)=Ym(l)(^xi)((l)φ(‖xi‖)+(l))c,q^(l)_i,mc=Y^(l)_m( r_xi)(W^(l) (\|r_xi\|)+b^(l))_c, where l=0,…,Lmax,|m|≤l,c=1,…Cl=0,…,L_max,|m|≤ l,c=1,… C. The parameters (l)∈ℝC×K,(l)∈ℝCW^(l) ^C× K,b^(l) ^C are a separate learned linear layer for each degree. For crystals, the relative displacement xir_xi is defined differently to account for periodic boundary conditions. The centroid of the context atoms is not well defined from the coordinates of a unit cell, since atoms that are spatially close may lie on opposite sides of a periodic boundary. We therefore define a reference atom cx∈ℐxc_x _x for the context graph. When the context graph is the sampled k-EgoNet cx=c_x=c , while when the complementary subgraph forms the context, cxc_x is sampled uniformly from ℐxI_x. The relative displacement of target atom i is then defined using its closest periodic image with respect to cxc_x xi=i+∗−cx,∗=argminn∈ℤ3‖i+−cx‖.r_xi=r_i+n^*L-r_c_x, ^*= _n ^3\|r_i+nL-r_c_x\|. Thus, for crystals xir_xi is the minimum-image displacement from the context reference atom cxc_x to target atom i. The resulting displacement is then encoded in the same way as in the molecular case. A.3.4 Loss We use mean squared error between predicted and target representations, ℓ(,)=1C‖−‖22, (a,b)= 1C\|a-b\|_2^2, where C is the feature dimension. For the substructure objective, the loss is averaged over the B graphs in the batch ℒsubx→y=1B∑g=1Bℓ(^y(g),¯y(g)),L_sub^x→ y= 1B _g=1^B ( s_y^(g), s_y^(g) ), where x→yx→ y denotes the prediction direction in which GxG_x is used as the context graph and GyG_y as the target graph. For the atom-level objective, the loss is averaged over all target atoms across all graphs in the batch ℒatomx→y=1∑g=1B|ℐy(g)|∑g=1B∑i∈ℐy(g)ℓ(^yi,yi).L_atom^x→ y= 1 _g=1^B|I_y^(g)| _g=1^B _i _y^(g) ( s_y_i,s_y_i ). Both objectives are evaluated symmetrically in both directions, with the two complementary subgraphs alternating as context and target. Thus, for o∈sub,atomo∈\sub,atom\, ℒo=12(ℒox→y+ℒoy→x),L_o= 12 (L_o^x→ y+L_o^y→ x ), and the complete loss is ℒ=λsubℒsub+λatomℒatomL= _subL_sub+ _atomL_atom. We train models with each objective individually and jointly, setting λpool=λatom=1 _pool= _atom=1. A.4 Data Curation We use structures at or near local energy minima because we expect their geometry to help constrain the identity of the target sub-structure and atom. The intuition is that, in a relaxed structure, the atomic arrangement must be compatible with the interactions between the constituent atom types. Strong distortions of the system can weaken this connection, making the target embedding of a specific atom significantly harder to predict from the context ℐxI_x. Restricting the data to near-minimum structures should reduce such ambiguity, even though several atom types could still be compatible with the same context. An alternative to this approach would be to include off-equilibrium structures and then condition the model on the system’s potential energy, as this would likely also help constrain the learning problem. Appendix B Limitations Throughout our experiments, we have used the Equiformer V3 architecture for all tasks. While we do observe a strong effect from using our SSL objective, we do not have a direct comparison to other SSL objectives with the same architecture. For ADMET datasets, other SSL objectives are not directly translatable due to their reliance on chemical bond encoding, whereas our SSL objective relies on the 3D structure of molecules. An additional nuance to this, is that Atom-JEPA is the only ADMET method tested that explicitly uses a 3D structure as input from conformers. We hypothesise that in addition to the Atom-JEPA loss, the ADMET performance could be driven by the strong implicit biases present in Equiformer V3 such as rotational equivariance or the data augmentation benefit of generating multiple conformers. We found a strong influence from the conformer choices as highlighted in Appendix H.1.3. Notably, the number of conformers used during training and averaging the model prediction over multiple conformers improve prediction accuracy. At the same time the quality of the conformer (as judged by the energy), seems to not have an impact on the performance. This suggest that it is rather the diversity of the conformers that improve the model not better conformers. This could be a problem when using weak conformer generation methods such as ours (see Appendix H.1.2), as their strong biases restrict the true conformer space. We have not investigated more accurate conformer generation strategies through higher order theory here. ADMET tasks are notoriously difficult to benchmark correctly, due to their heavy generalization requirements from non-uniformly random splits, small test samples, and noisy labels. To asses optimal performance for each model, their hyperparameters would have been tuned through 2-fold cross-validation. It would have been computationally infeasible to do this for finetuning of Atom-JEPA and costly to do for all the MLP baselines. While it would be possible to do 2-fold cross-validation for the LGBT models, we opted to use our shared hyperparameter setup consistent across all models to make cross-model comparisons more fair. Appendix C Datasets Table 5 summarizes the datasets used for large-scale pretraining and downstream evaluation. In addition to its use as a downstream benchmark, we use QM9 for a separate set of pretraining ablations of the loss objectives. We subsample the Alexandria PBE dataset by retaining only near-stable relaxed crystal structures with an energy above the convex hull of at most 0.05 eV/atom and containing between 2 and 100 atoms per unit cell. These filtering criteria reduce the dataset from 5,952,515 to 1,696,344 structures. The Uni-Mol dataset contains 10 3D conformers per molecule. We use only a single conformer per molecule (the first), resulting in 18,837,320 training samples. Table 5: Summary of datasets. Uni-Mol and Alexandria are used for large-scale pretraining. The remaining datasets are used for downstream evaluation. QM9 is additionally used for pretraining in the separate ablation experiments. Dataset Samples Tasks Description Large-scale pretraining datasets Uni-Mol (Zhou et al., 2023) 18,837,320 – 3D molecular conformers Alexandria PBE (Cavignac et al., 2026) 1,696,344 – DFT-relaxed inorganic crystals Downstream evaluation datasets ChEMBL-MT (Adrian et al., 2025) 114,000 25 Multi-task ADMET ExpansionRx (MacDermott-Opeskin and Castellanos, 2026) 7,608 10 Multi-task ADMET Biogen ADME (Fang et al., 2023) 3,521 6 Multi-task ADME TDC-ADMET (Huang et al., 2021) 475–13,130 22 Individual-task ADMET QM9 (Ramakrishnan et al., 2014) 130,831 12 Quantum-chemical properties Matbench (Dunn et al., 2020) 1,265–132,752 8 Materials-property prediction C.1 Structural Overlap Due to the size of Unimol and Alexandria, some of the structures used for pretraining of Atom-JEPA are also present in datasets of ADMET, QM9, and Matbench datasets. While self-supervised training has not been observed to bias accuracy noticeably (Xue et al., 2026), we still report the overlaps for transparency of the test sets for ADMET tasks in Table 6, full datasets of QM9 in Table 7, and full dataset of Matbench in Table 8. For most ADMET endpoints, Unimol contains somewhere around 35%−75%35\%-75\% of the structures of molecules seen in the endpoints. Due to the temporal split of ExpansionRX, only 1 test point is found in Unimol and at most 2%2\% of the molecules have the same scaffold. We found that Atom-JEPA’s improvement over other models is the largest on specifically ExpansionRX, but that the accuracy cannot be attributed to structural overlap between the pretraining data and ExpansionRX. For QM9 the overlap is also quite small at 4%4\%. For Matbench the overlap to Alexandria varies for the endpoints. Table 6: Uni-Mol pretraining-set overlap in downstream evaluation data. Dataset rows are unions over their evaluation test membership. "Unique molecules" and "Found in Uni-Mol" use the Exact definition and with an observed label for that endpoint. Exact denotes identical RDKit canonical isomeric molecular graphs and retains stereochemistry, isotopes, formal charge, and disconnected fragments. Parent compares cleaned fragment parents, ignoring salts and solvents while retaining stereochemistry. Broad parent also normalizes charge and isotopes, removes stereochemistry, and uses a heteroatom-tautomer hash. Murcko scaffold is the percentage of unique Bemis-Murcko scaffolds observed in Uni-Mol and measures structural relatedness. Percentage rounded to nearest percentage. Evaluation dataset Unique molecules Found in Uni-Mol Overlap (%) Exact Parent Broad parent Murcko scaffold ChEMBL-MT 23,213 11,964 52 52 73 90 CL Microsome Human 968 632 65 67 73 88 CL Microsome Mouse 136 84 62 66 69 92 CL Microsome Rat 341 226 66 66 69 87 CL Total Dog 90 54 60 71 94 97 CL Total Human 241 150 62 71 94 97 CL Total Monkey 42 21 50 55 95 97 CL Total Rat 124 72 58 65 94 95 CYP2C8 Inhibition 53 28 53 55 60 77 CYP2C9 Inhibition 515 264 51 53 60 81 CYP2D6 Inhibition 449 252 56 60 67 88 CYP3A4 Inhibition 1,084 615 57 58 64 79 Fraction unbound plasma (Dog) 50 23 46 62 96 95 Fraction unbound plasma (Human) 603 274 45 49 83 74 Fraction unbound plasma (Monkey) 24 10 42 50 96 95 Fraction unbound plasma (Rat) 70 40 57 67 96 95 Papp Caco2 1,216 495 41 42 74 90 Pgp (Human) 419 296 71 73 77 93 hERG binding 757 322 43 44 75 90 LogD pH 7.4 763 516 68 72 79 94 LogSaq Kinetic 11,350 6,528 58 58 82 94 LogSaq Thermo 6,750 2,835 42 43 59 86 VDss Dog 87 53 61 71 94 97 VDss Human 245 152 62 71 94 97 VDss Monkey 41 20 49 54 95 97 VDss Rat 111 61 55 63 95 95 ExpansionRx 2,282 1 0 0 0 2 LogD 2,270 1 0 0 0 2 KSOL 2,170 1 0 0 0 1 HLM CLint 782 0 0 0 0 1 MLM CLint 1,170 0 0 0 0 1 Caco-2 Permeability Papp A>B 1,616 0 0 0 0 2 Caco-2 Permeability Efflux 1,616 0 0 0 0 2 MPPB 454 0 0 0 0 1 MBPB 451 0 0 0 0 1 MGMB 209 0 0 0 0 0 Biogen ADME 1,830 907 50 50 63 83 Log HLM CLint (mL/min/kg) 1,620 809 50 50 63 83 Log MDR1-MDCK 1,384 703 51 51 63 83 Log SOLUBILITY PH 6.8 (ug/mL) 1,097 508 46 47 60 83 Log Plasma Protein Binding Human 120 53 44 44 64 75 Log Plasma Protein Binding Rat 93 42 45 45 67 81 Log RLM CLint (mL/min/kg) 1,596 804 50 51 64 84 TDC-ADMET 13,816 7,406 54 56 71 89 AMES 1,453 513 35 36 51 73 B Martins 394 248 63 72 84 93 Bioavailability Ma 128 89 70 83 94 99 Caco2 Wang 181 67 37 39 66 86 Clearance Hepatocyte AZ 203 142 70 73 82 94 Clearance Microsome AZ 221 155 70 72 76 94 CYP2C9 Substrate Carbon-Mangels 135 89 66 69 88 96 CYP2C9 Veith 2,419 1,541 64 66 82 93 CYP2D6 Substrate Carbon-Mangels 134 93 69 77 92 100 CYP2D6 Veith 2,626 1,707 65 67 84 94 CYP3A4 Substrate Carbon-Mangels 134 90 67 75 92 97 CYP3A4 Veith 2,467 1,603 65 67 82 93 DILI 96 59 61 70 96 99 Half-life Obach 134 84 63 78 93 96 hERG 128 35 27 32 73 74 HIA Hou 117 60 51 55 84 97 LD50 Zhu 1,466 538 37 38 53 84 Lipophilicity AstraZeneca 840 601 72 74 78 90 Pgb Broccatelli 245 137 56 60 82 90 PPBR AZ 343 241 70 71 79 96 Solubility AqSolDB 1,995 762 38 41 57 84 VDss Lombardo 221 45 20 25 89 67 Table 7: Full QM9 overlap with Uni-Mol pretraining data. Exact denotes an identical canonical isomeric molecular graph. Parent removes salts and solvents while retaining stereochemistry. Broad parent additionally normalizes charge, isotopes, stereochemistry, and common heteroatom tautomers. Murcko scaffold measures structural relatedness rather than molecule identity and excludes acyclic molecules from its denominator. All QM9 properties share this molecular inventory. The graph-based profiles exclude 1,819 PyG QM9 entries whose explicit-hydrogen SMILES cannot be sanitized by RDKit; Unique molecules therefore denotes unique valid graph identities. Evaluation dataset Unique molecules Found in Uni-Mol Overlap (%) Exact Parent Broad parent Murcko scaffold QM9 128,822 4,742 4 4 5 9 Table 8: Matbench overlap with Alexandria PBE pretraining data. Strict requires the same elemental composition and a tight primitive-cell match without lattice rescaling. Relaxed uses lattice rescaling and moderate geometric tolerances to accommodate relaxation differences. Prototype permits element substitution but requires the same canonical anonymous formula, Pearson symbol, space group, and Wyckoff sequence. Composition requires only the same reduced elemental composition. The latter two columns measure structural or chemical relatedness, not identical materials. Evaluation dataset Unique structures Found in Alexandria Overlap (%) Strict Relaxed Prototype Composition Matbench 151,731 59,032 39 44 76 59 Phonons 1,265 1,190 94 96 100 99 Dielectric 4,764 3,399 71 80 93 92 Log GVRH 10,987 8,697 79 83 99 93 Log KVRH 10,987 8,697 79 83 99 93 Perovskites 18,928 26 0 0 64 5 MP Gap 106,113 58,659 55 63 84 78 MP E Form 132,752 58,959 44 51 78 67 MP Is Metal 106,113 58,659 55 63 84 78 Appendix D Analysis of Learned Representations Here we examine the learned representation space of Atom-JEPA using visualizations, nearest-neighbor analysis and probes on frozen features. First, for each of the two main models trained for molecules and crystals respectively, we randomly sample 50,000 structures from the corresponding pretraining dataset and mean-pool over the final-layer scalar atom features. For all embedding visualizations, we standardize the features and apply principal component analysis (PCA), retaining components that explain at least 90% of the variance, followed by UMAP with 60 neighbors and a minimum distance of 0.4 to obtain a two-dimensional projection. We color crystals by crystallographic symmetry and molecules by physicochemical properties and structural features. We further assess structural similarity within embedding neighborhoods and use probes to quantify the structural, geometric and physical-property information captured beyond what can be predicted from atomic composition, system size and geometry descriptor baselines. For the molecular model, we extend the analysis to 32,252 molecules from the manually annotated three-star subset of ChEBI (Chemical Entities of Biological Interest) (Malik et al., 2026), coloring their embeddings by chemical family. Finally, on QM9 molecules, we compare the pretrained molecular model with a randomly initialized counterpart through visualizations and probes of each QM9 target using final-layer and all-layer features, assessing Atom-JEPA pretraining’s contribution beyond the 3D equivariant GNN backbone’s inductive biases. D.1 Structure and Property Organization in the Embedding Space D.1.1 Crystal Representations Figures 5 and 6 visualize the Atom-JEPA features of 50,000 random samples from the Alexandria pretraining dataset. Figure 5 colors each crystal by its corresponding crystal system and Figure 6 colors by the six most frequent space groups in the sample. (a) Combined crystal systems. (b) Each crystal system highlighted against the others in gray. Figure 5: UMAP visualization of crystal representations by crystal system. (a) Combined space groups. (b) Each of the six most frequent space groups highlighted against the others in gray. Figure 6: UMAP visualization of crystal representations by space group. To quantify structural organization in the embedding space, we evaluate the 10 nearest neighbors of 1,000 randomly selected samples using Euclidean distance in Table 9. We compare these neighborhoods with nearest neighbors measured by composition (118-dimensional vector of elemental fractions), random crystals with matching atom count, and unrestricted random crystals. Embedding neighborhoods exhibit substantially greater space-group agreement than all three controls. Only 0.05% of embedding-neighbor pairs in the sample share the same reduced composition (elemental counts divided by their greatest common divisor), indicating that this symmetry organization extends across different chemical compositions. Table 9: Structural neighborhood agreement. Neighbor selection Same space group Embedding neighbors 69.7% Composition-only neighbors 14.4% Random crystals with matching atom count 12.4% Unmatched random crystals 4.9% Next, we assess the information accessible through linear probes on the frozen features by predicting geometric quantities and physical properties. We partition the 50,000 sample into training, validation, and test sets containing 70%, 15%, and 15% of the structures, respectively, with reduced compositions kept disjoint across partitions. We then fit ridge-regression probes with regularization selected by validation mean squared error. Features are standardized using training-set statistics, and performance is reported as R2R^2, the coefficient of determination on the test set. Table 10 compares embeddings with a composition-and-size baseline for predicting log-volume per atom ln(V/N) (V/N), where V is the unit-cell volume and N is the number of atoms in the cell as well as predicting mean coordination within 3 Å. The baseline represents each crystal using 122 features: 118 elemental fractions and four size features given by N, lnN N, N2N^2 and N3N^3 allowing the regression to model nonlinear relationships between cell size and the target. Coordination is calculated using periodic neighbours, including qualifying self-images. Combining embeddings with composition and size improves prediction of both quantities, indicating that the learned features include complementary geometric information that can be accessed by linear probes. While the embeddings alone slightly outperform the baseline for coordination, they perform worse for log-volume per atom. Table 10: Test-set coefficient of determination R2R^2 for geometric quantities. Predictor Log-volume-per-atom Coordination Composition + atom count .942 .794 Embeddings .854 .814 Both together .973 .894 Table 11 reports linear-probe performance for energy above hull, formation energy, and indirect band gap with targets taken from the corresponding Alexandria record. We compare embeddings with composition-and-size baselines, additionally incorporating nine geometric descriptors that summarize cell geometry and local atomic environments. Embeddings alone outperform these baselines for energy above hull but underperform for formation energy and band gap. However, combining embeddings with composition, size, and geometry yields the highest test R2R^2 for all three targets. This suggests that the embeddings provide complementary predictive features, rather than consistently replacing explicit descriptors in the setting of linear probing. Formation energy is predicted most strongly, with the combined model improving R2R^2 from 0.861 to 0.908 over the baseline including geometry. Energy-above-hull prediction remains weak, despite achieving the largest relative improvement over that baseline: the combined model reaches R2=0.136R^2=0.136, which corresponds to an MAE of 11.7 meV/atom, compared with 13.2 meV/atom for a training-mean predictor. These results concern the near-stable subset used for pretraining, restricted to relaxed structures with energy above hull at most 0.05 eV/atom. The probe therefore assesses fine differences within the narrow 0–50 meV/atom interval, rather than discrimination across a broad range of crystal stabilities. The nine geometric descriptors that we use are log-volume per atom; the ratio of the longest to shortest lattice-vector length; the root-mean-square cosine of the cell angles; the mean, standard deviation, minimum, and maximum coordination within 3 Å; and the mean and standard deviation of nearest-neighbor distances. Table 11: Test-set coefficient of determination R2R^2 for physical properties. Predictor Energy above hull Formation energy Indirect band gap Composition + size .072 .821 .493 Composition + size + geometry .079 .861 .507 Embeddings .109 .712 .446 Embeddings + composition + size + geometry .136 .908 .542 D.1.2 Molecular Representations Next, we examine molecular representations using 50,000 randomly sampled molecules from the Uni-mol pretraining dataset. Figures 7, 8, and 9 shows visualizations colored by molecular weight, net formal charge, ring count, aromatic heavy-atom fraction, the six most frequent functional groups, and molecular scaffolds. At the low level, we observe limited separation according to molecular properties and no clear clustering by functional group. A small exception is negatively charged molecules, which are concentrated around two localized islands, whereas neutral and positively charged molecules are distributed broadly. At the mid-level, molecular scaffolds show clearer separation, with some distinguishable local clusters, although many scaffolds remain scattered. Thus, scaffold identity is more apparent in the organization of the representations than individual low-level features, but does not fully determine it. Figure 7: Uni-Mol representations colored by molecular weight, formal charge, ring count, and aromatic heavy-atom fraction. Figure 8: Uni-Mol representations with six functional groups highlighted individually. (a) 6 most frequent scaffolds combined (b) 6 most frequent scaffolds highlighted individually Figure 9: Uni-Mol representations with the 6 most frequent molecular scaffolds highlighted, shown combined and individually (remaining points in grey). To assess whether low-level information remains accessible despite this limited visual separation, table 12 evaluates linear ridge-regression probes for ring count, aromatic heavy atom fraction and radius of gyration, and compare to a composition-and-size baseline. We use a 70/15/15% train/val/test split of scaffold groups defined by Murcko scaffold, with acyclic molecules grouped by canonical SMILES. Again features are standardized using training-set statistics, and regularization is selected for each target by validation mean squared error, and we report the test-set coefficient of determination. Embeddings outperform composition and size for all three targets, with the largest improvement for radius of gyration. Combined features achieve the highest scores, although gains over embeddings alone are small for aromatic fraction and radius of gyration. Thus, despite limited visual separation in the UMAP projections, the representations make molecular structural and geometric properties accessible to linear prediction across scaffold-disjoint partitions. Table 12: Uni-Mol structural probe results. Input Ring count Aromatic fraction Radius of gyration Composition + size .706 .642 .615 Embeddings .909 .848 .964 Combined .958 .856 .969 Next, we encode and plot approximately 32,252 molecules from the manually annotated three-star subset of ChEBI Malik et al. (2026) and color them by chemical family in figures 10 and 11. Here, the projections exhibit clearer clustering than the low-level property and mid-level scaffold colorings of the Uni-Mol sample. Table 13 quantifies this using linear ridge scores fitted to binary membership labels. We use the same scaffold-group splitting procedure and training-set standardization, and regularization as above, selecting regularization separately for each family by validation average precision. Evaluation includes families with at least 50 annotated positives in training, 20 in validation, and 20 in testing, together with at least 20 unannotated examples in each of validation and testing. We report average precision and AUROC averaged equally across eligible families. The outputs are ranking scores, and unannotated membership is treated as the negative class for evaluation. Embeddings substantially outperform composition and size in recovering annotated chemical-family membership, while their combination yields only small further gains. Together, these results suggest a hierarchy in the organization of the learned representations: low-level properties and functional groups show no or very limited visual separation, mid-level scaffolds exhibit partial clustering, and high-level chemical families show the clearest organization. This pattern is consistent with the hypothesized effect of Atom-JEPA pretraining, with latent-space prediction encouraging representations to prioritize higher-level chemical features while reducing the influence of low-level details. The strong structural probe results further suggest that this emphasis does not require discarding low-level information entirely; such information remains recoverable. Figure 10: UMAP visualization of ChEBI representations, with exclusive chemical families highlighted. Figure 11: Visualization of ChEBI representations, with chemical families highlighted individually and the rest in grey. Table 13: ChEBI family probe results. Input Mean average precision Mean AUROC Composition + size .186 .874 Embeddings .547 .964 Combined .554 .966 D.2 Effect of Pretraining on Molecular Representations Finally, to assess how Atom-JEPA pretraining shapes molecular representations beyond the inductive biases of the GNN backbone, we compare pretrained and randomly initialized encoders with identical architectures on the QM9 dataset. For each, we train frozen-backbone probes using the dataset split specified in H and visualize representations of the test set molecules to relate their organization to predictive performance. Figure 12 highlights the three most frequent chemical formulas in the test set. The randomly initialized encoder separates these formulas into 3 distinct clusters, suggesting that without pretraining representations are organized entirely by molecular size and elemental composition. In contrast, as found above Atom-JEPA representations are not separated by low-level features such as size and composition. Coloring the embeddings by the twelve QM9 properties (Figure 13) also reveals clear patterns for several targets in the random representations. These patterns are consistent with a simple organization by size and composition, which already accounts for substantial variation in several properties. Again, the Atom-JEPA representations do not exhibit the same apparent patterns. To assess how these visual differences translate to downstream performance, we train all-layer MLP probes on frozen per-atom features from both encoders and report results in Table 14. All probes outperform the training-set mean predictor on every target, confirming that the backbone’s inductive biases alone provide useful predictive features. Nevertheless, Atom-JEPA achieves substantially lower MAE across all twelve targets with both probe types. With all-layer features, pretraining reduces MAE by approximately three- to fourfold for polarizability, the HOMO–LUMO gap, HOMO and LUMO energies, and heat capacity, and by seven- to eightfold for the atom-referenced energies G, H, U, and U0U_0. Gains are smaller for dipole moment, where random features already perform comparatively well. An additional finding is that all-layer probes outperform last-layer probes on every target for Atom-JEPA and on eleven of twelve targets for the random encoder, with particularly pronounced gains for Atom-JEPA energy predictions. These results show that clear visual separation by target value does not indicate greater predictive utility. The patterns in the random representations reflect a simple organization by molecular size and elemental composition, whereas Atom-JEPA supports substantially more accurate prediction without comparable formula-based clustering. Together with the chemical-family analysis, these findings are consistent with latent-space predictive pretraining shifting representations beyond these basic attributes toward higher-level, property-relevant molecular information that is not readily apparent in two-dimensional projections, but nevertheless improve downstream performance. Figure 12: The three most frequent size/composition profiles highlighted in the randomly initialized and the Atom-JEPA pretrained QM9 representation, respectively Figure 13: Pretrained and randomly initialized QM9 representations colored by within-property percentile rank for all twelve targets. Table 14: QM9 frozen-backbone MLP probe test MAE results using all-layer or last-layer features for a pretrained encoder vs random initialization, with a training-set mean predictor as a baseline (atom-referenced for U, U0U_0, H, G). Model α Δε εHOMO _HOMO εLUMO _LUMO μ CvC_v G H ⟨R2⟩ R^2 U U0U_0 ZPVE a03a_0^3 eV eV eV D calmol K calmol K eV eV a02a_0^2 eV eV eV Training-set mean 6.343 1.077 .434 1.055 1.165 3.195 7.511 8.304 203.364 8.244 8.170 .720 All layers Atom-JEPA .091 .095 .061 .064 .032 .031 .013 .012 .419 .012 .012 .001 Random initialization .371 .279 .171 .221 .039 .108 .091 .089 1.850 .089 .091 .004 Last layer Atom-JEPA .217 .141 .094 .099 .034 .071 .061 .070 1.579 .068 .072 .004 Random initialization .479 .327 .192 .246 .038 .135 .114 .110 2.444 .114 .110 .005 Appendix E ADMET Full Results E.1 Statistical Tests Setup To compute statistical indistinguishable models in Table 1, we compute an ANOVA test over each endpoint. For Biogen and ChEMBL the design is blocked across folds, whereas ExpansionRX is unpaired due to using the same test set each run. The best model that is compared against is the model with the lowest mean error over the respective endpoint folds. For runs where we have the individual fold errors, we check if each models computed 95%95\% confidence intervals do not overlap using a pairwise Tukey test. Because we only have the mean run error for KERMT, we instead compare using a single-sided t-test with Holm–Bonferroni correction. E.2 Model Prediction Correlation We present the full analysis of the differences between the ADMET models. We quantify difference between the prediction of models by their spearman correlation (r)(r). We use residuals instead of direct predictions to reduce model strength as a confounder. In the interest of ensembling model predictions, we also control against the case where one model happens to be weakly correlated with other models, but strongly correlated with the sum of the other models. For completeness we calculate the uniqueness score as follows: For endpoint t and run s, let em∈ℝne_m ^n denote model m’s residual vector y^−y y-y over the n test molecules with an observed label for t, and let (m)P(m) be one representative of each of the four model families other than m’s own. We regress eme_m on ekk∈(m)\e_k\_k (m) with an intercept by ordinary least squares and define Um(t,s)=1−Radj2=(1−R2)n−1n−p−1,U_m(t,s)=1-R^2_adj= (1-R^2 )\, n-1n-p-1, (7) where p=|(m)|=4p= (m) =4 for our for model comparisons, R2R^2 is the coefficient of determination of that fit. UmU_m is the share of model m’s squared error that no linear combination of the other families reproduces,. We do not include endpoints with less than 30 test datapoints for robustness. We only compare other models to Atom-JEPA averaged over 10 conformers to not bias the results by also including the the non-conformer averaged Atom-JEPA model. The single and 10×10× conformer predictions correlate with over 0.990.99. The results can be seen in Table 15. Table 15: Model distinctiveness. How reproducible each model’s errors are from the other four model families, measured two ways on the same per-molecule test residuals. Residual correlation is the averaged pairwise correlation between other models, averaged over test data-points, Uniqueness is the 1−Radj21-R^2_adj from regressing a model’s residuals on all four peers’ residuals at once. Per-benchmark columns are means over each benchmark’s endpoints. Mean rank ranks the families within each benchmark and endpoint cell and averages with equal weight per benchmark, with rank 1 = most distinct. The brackets are a stratified cluster bootstrap over endpoints. Group shows the group of models that cannot separated by the paired 95% bootstrap interval on their mean-rank difference. Model Biogen ChEMBL-MT ExpansionRx Mean rank across endpoints Group Residual correlation r¯ r to peers (lower is more distinct) Atom-JEPA ×10 conf .723.723 .791.791 .730.730 1.981.98 [1.66, 2.34] a Chemprop .719.719 .803.803 .681.681 2.532.53 [2.02, 2.88] ab Mol-JEPA ⋅· modalities .738.738 .801.801 .749.749 2.782.78 [2.42, 3.14] b Morgan + RDKit ⋅· LGBM .724.724 .814.814 .761.761 2.982.98 [2.59, 3.31] b CKERMT .779.779 .829.829 .787.787 4.734.73 [4.48, 4.86] c Uniqueness 1−Radj21-R^2_adj (higher is more distinct) Atom-JEPA ×10 conf .327.327 .253.253 .283.283 1.971.97 [1.65, 2.32] a Chemprop .325.325 .221.221 .392.392 2.532.53 [2.02, 2.89] ab Mol-JEPA ⋅· modalities .285.285 .221.221 .246.246 2.892.89 [2.52, 3.23] b Morgan + RDKit ⋅· LGBM .312.312 .192.192 .228.228 2.992.99 [2.57, 3.39] b CKERMT .214.214 .171.171 .187.187 4.634.63 [4.39, 4.78] c E.3 Biogen ADME Table 16: Test MAE mean ± standard deviation across scaffold splits across tasks for Biogen dataset. † : Data from Figure 13 in Xue et al. (2026). Model HLM CLint RLM CLint MDCK logS PPB Human PPB Rat Mean GBT models RDKit .3773±.0190.3773 [rgb]0.5,0.5,0.5 ±.0190 .4347±.0268.4347 [rgb]0.5,0.5,0.5 ±.0268 .3493±.0116.3493 [rgb]0.5,0.5,0.5 ±.0116 .3725±.0204.3725 [rgb]0.5,0.5,0.5 ±.0204 .4713±.1293.4713 [rgb]0.5,0.5,0.5 ±.1293 .4872±.0594.4872 [rgb]0.5,0.5,0.5 ±.0594 .4154±.0298.4154 [rgb]0.5,0.5,0.5 ±.0298 Morgan .4115±.0242.4115 [rgb]0.5,0.5,0.5 ±.0242 .4616±.0182.4616 [rgb]0.5,0.5,0.5 ±.0182 .3682±.0131.3682 [rgb]0.5,0.5,0.5 ±.0131 .3838±.0230.3838 [rgb]0.5,0.5,0.5 ±.0230 .5085±.1062.5085 [rgb]0.5,0.5,0.5 ±.1062 .6293±.0828.6293 [rgb]0.5,0.5,0.5 ±.0828 .4605±.0260.4605 [rgb]0.5,0.5,0.5 ±.0260 Morgan + RDKit .3712±.0140.3712 [rgb]0.5,0.5,0.5 ±.0140 .4136±.0239.4136 [rgb]0.5,0.5,0.5 ±.0239 .3362±.0103.3362 [rgb]0.5,0.5,0.5 ±.0103 .3663±.0143.3663 [rgb]0.5,0.5,0.5 ±.0143 .4462±.1109.4462 [rgb]0.5,0.5,0.5 ±.1109 .5125±.0984.5125 [rgb]0.5,0.5,0.5 ±.0984 .4077±.0339.4077 [rgb]0.5,0.5,0.5 ±.0339 Morgan + RDKit + Avalon + ErG .3863±.0186.3863 [rgb]0.5,0.5,0.5 ±.0186 .4160±.0196.4160 [rgb]0.5,0.5,0.5 ±.0196 .3497±.0154.3497 [rgb]0.5,0.5,0.5 ±.0154 .3882±.0252.3882 [rgb]0.5,0.5,0.5 ±.0252 .5454±.1507.5454 [rgb]0.5,0.5,0.5 ±.1507 .5686±.0660.5686 [rgb]0.5,0.5,0.5 ±.0660 .4424±.0247.4424 [rgb]0.5,0.5,0.5 ±.0247 Mol-JEPA CLS .4051±.0218.4051 [rgb]0.5,0.5,0.5 ±.0218 .4523±.0187.4523 [rgb]0.5,0.5,0.5 ±.0187 .3689±.0096.3689 [rgb]0.5,0.5,0.5 ±.0096 .3887±.0262.3887 [rgb]0.5,0.5,0.5 ±.0262 .5302±.1025.5302 [rgb]0.5,0.5,0.5 ±.1025 .5264±.0835.5264 [rgb]0.5,0.5,0.5 ±.0835 .4453±.0317.4453 [rgb]0.5,0.5,0.5 ±.0317 Atom-JEPA Last Layer .4300±.0221.4300 [rgb]0.5,0.5,0.5 ±.0221 .4869±.0206.4869 [rgb]0.5,0.5,0.5 ±.0206 .3791±.0127.3791 [rgb]0.5,0.5,0.5 ±.0127 .3813±.0201.3813 [rgb]0.5,0.5,0.5 ±.0201 .5182±.0828.5182 [rgb]0.5,0.5,0.5 ±.0828 .5090±.0991.5090 [rgb]0.5,0.5,0.5 ±.0991 .4507±.0213.4507 [rgb]0.5,0.5,0.5 ±.0213 Atom-JEPA All Layer .4021±.0134.4021 [rgb]0.5,0.5,0.5 ±.0134 .4613±.0198.4613 [rgb]0.5,0.5,0.5 ±.0198 .3563±.0039.3563 [rgb]0.5,0.5,0.5 ±.0039 .3691±.0229.3691 [rgb]0.5,0.5,0.5 ±.0229 .4798±.1242.4798 [rgb]0.5,0.5,0.5 ±.1242 .4753±.0763.4753 [rgb]0.5,0.5,0.5 ±.0763 .4240±.0266.4240 [rgb]0.5,0.5,0.5 ±.0266 MLP models RDKit .3772±.0249.3772 [rgb]0.5,0.5,0.5 ±.0249 .4400±.0239.4400 [rgb]0.5,0.5,0.5 ±.0239 .3383±.0192.3383 [rgb]0.5,0.5,0.5 ±.0192 .3798±.0272.3798 [rgb]0.5,0.5,0.5 ±.0272 .4112±.0802.4112 [rgb]0.5,0.5,0.5 ±.0802 .4227±.0932.4227 [rgb]0.5,0.5,0.5 ±.0932 .3949±.0292.3949 [rgb]0.5,0.5,0.5 ±.0292 Morgan .4323±.0271.4323 [rgb]0.5,0.5,0.5 ±.0271 .5055±.0343.5055 [rgb]0.5,0.5,0.5 ±.0343 .3917±.0215.3917 [rgb]0.5,0.5,0.5 ±.0215 .4159±.0221.4159 [rgb]0.5,0.5,0.5 ±.0221 .4793±.0778.4793 [rgb]0.5,0.5,0.5 ±.0778 .4794±.0793.4794 [rgb]0.5,0.5,0.5 ±.0793 .4507±.0218.4507 [rgb]0.5,0.5,0.5 ±.0218 Morgan + RDKit .3469±.0197.3469 [rgb]0.5,0.5,0.5 ±.0197 .3998±.0202.3998 [rgb]0.5,0.5,0.5 ±.0202 .3344±.0238.3344 [rgb]0.5,0.5,0.5 ±.0238 .3772±.0133.3772 [rgb]0.5,0.5,0.5 ±.0133 .4034±.0778.4034 [rgb]0.5,0.5,0.5 ±.0778 .4412±.0778.4412 [rgb]0.5,0.5,0.5 ±.0778 .3838±.0194.3838 [rgb]0.5,0.5,0.5 ±.0194 Morgan + RDKit + Avalon + ErG .3336±.0161.3336 [rgb]0.5,0.5,0.5 ±.0161 .3892±.0139.3892 [rgb]0.5,0.5,0.5 ±.0139 .3197±.0193.3197 [rgb]0.5,0.5,0.5 ±.0193 .3679±.0241.3679 [rgb]0.5,0.5,0.5 ±.0241 .4538±.0758.4538 [rgb]0.5,0.5,0.5 ±.0758 .4651±.0205.4651 [rgb]0.5,0.5,0.5 ±.0205 .3882±.0137.3882 [rgb]0.5,0.5,0.5 ±.0137 Mol-JEPA CLS .3710±.0209.3710 [rgb]0.5,0.5,0.5 ±.0209 .4148±.0205.4148 [rgb]0.5,0.5,0.5 ±.0205 .3357±.0172.3357 [rgb]0.5,0.5,0.5 ±.0172 .3660±.0176.3660 [rgb]0.5,0.5,0.5 ±.0176 .4000±.0441.4000 [rgb]0.5,0.5,0.5 ±.0441 .3810±.0375.3810 [rgb]0.5,0.5,0.5 ±.0375 .3781±.0137.3781 [rgb]0.5,0.5,0.5 ±.0137 Mol-JEPA modalities .3423±.0115.3423 [rgb]0.5,0.5,0.5 ±.0115 .3932±.0221.3932 [rgb]0.5,0.5,0.5 ±.0221 .3237±.0143.3237 [rgb]0.5,0.5,0.5 ±.0143 .3293±.0174.3293 [rgb]0.5,0.5,0.5 ±.0174 .4048±.0585.4048 [rgb]0.5,0.5,0.5 ±.0585 .3782±.0409.3782 [rgb]0.5,0.5,0.5 ±.0409 .3619±.0141.3619 [rgb]0.5,0.5,0.5 ±.0141 Atom-JEPA Last Layer .3754±.0244.3754 [rgb]0.5,0.5,0.5 ±.0244 .4304±.0144.4304 [rgb]0.5,0.5,0.5 ±.0144 .3383±.0086.3383 [rgb]0.5,0.5,0.5 ±.0086 .3594±.0238.3594 [rgb]0.5,0.5,0.5 ±.0238 .4297±.0637.4297 [rgb]0.5,0.5,0.5 ±.0637 .4516±.0503.4516 [rgb]0.5,0.5,0.5 ±.0503 .3975±.0181.3975 [rgb]0.5,0.5,0.5 ±.0181 Atom-JEPA All Layer .3757±.0230.3757 [rgb]0.5,0.5,0.5 ±.0230 .4322±.0257.4322 [rgb]0.5,0.5,0.5 ±.0257 .3153±.0115.3153 [rgb]0.5,0.5,0.5 ±.0115 .3518±.0197.3518 [rgb]0.5,0.5,0.5 ±.0197 .4282±.0883.4282 [rgb]0.5,0.5,0.5 ±.0883 .4320±.0772.4320 [rgb]0.5,0.5,0.5 ±.0772 .3892±.0270.3892 [rgb]0.5,0.5,0.5 ±.0270 Finetuning No Pretrain .3232±.0170.3232 [rgb]0.5,0.5,0.5 ±.0170 .3781±.0082.3781 [rgb]0.5,0.5,0.5 ±.0082 .3322±.0114.3322 [rgb]0.5,0.5,0.5 ±.0114 .3274±.0091.3274 [rgb]0.5,0.5,0.5 ±.0091 .4696±.1024.4696 [rgb]0.5,0.5,0.5 ±.1024 .4623±.0711.4623 [rgb]0.5,0.5,0.5 ±.0711 .3821±.0281.3821 [rgb]0.5,0.5,0.5 ±.0281 Atom-JEPA .2950±.0146.2950 [rgb]0.5,0.5,0.5 ±.0146 .3548±.0185.3548 [rgb]0.5,0.5,0.5 ±.0185 .2816±.0107.2816 [rgb]0.5,0.5,0.5 ±.0107 .3246±.0110.3246 [rgb]0.5,0.5,0.5 ±.0110 .4221±.0922.4221 [rgb]0.5,0.5,0.5 ±.0922 .3970±.0747.3970 [rgb]0.5,0.5,0.5 ±.0747 .3458±.0260.3458 [rgb]0.5,0.5,0.5 ±.0260 Atom-JEPA (10-conf. ensemble) .2931±.0147.2931 [rgb]0.5,0.5,0.5 ±.0147 .3523±.0186.3523 [rgb]0.5,0.5,0.5 ±.0186 .2796±.0110.2796 [rgb]0.5,0.5,0.5 ±.0110 .3231±.0112.3231 [rgb]0.5,0.5,0.5 ±.0112 .4199±.0940.4199 [rgb]0.5,0.5,0.5 ±.0940 .3932±.0759.3932 [rgb]0.5,0.5,0.5 ±.0759 .3435±.0267.3435 [rgb]0.5,0.5,0.5 ±.0267 Literature models Chemprop .3133±.0058.3133 [rgb]0.5,0.5,0.5 ±.0058 .3683±.0141.3683 [rgb]0.5,0.5,0.5 ±.0141 .2954±.0070.2954 [rgb]0.5,0.5,0.5 ±.0070 .3305±.0143.3305 [rgb]0.5,0.5,0.5 ±.0143 .3968±.1017.3968 [rgb]0.5,0.5,0.5 ±.1017 .4062±.1136.4062 [rgb]0.5,0.5,0.5 ±.1136 .3518±.0317.3518 [rgb]0.5,0.5,0.5 ±.0317 CKERMT Our runs .2996±.0095.2996 [rgb]0.5,0.5,0.5 ±.0095 .3646±.0194.3646 [rgb]0.5,0.5,0.5 ±.0194 .2892±.0136.2892 [rgb]0.5,0.5,0.5 ±.0136 .3265±.0104.3265 [rgb]0.5,0.5,0.5 ±.0104 .4093±.0820.4093 [rgb]0.5,0.5,0.5 ±.0820 .4451±.0953.4451 [rgb]0.5,0.5,0.5 ±.0953 .3557±.0290.3557 [rgb]0.5,0.5,0.5 ±.0290 KERMT base† .320.320 .402.402 .289.289 .344.344 – – – KERMT task-specific† .309.309 .402.402 .278.278 .339.339 – – – CKERMT base† .302.302 .389.389 .275.275 .329.329 – – – CKERMT task-specific† .294.294 .389.389 .268.268 .333.333 – – – Table 17: Test Pearson r2r^2 ± standard deviation across cluster splits across tasks for Biogen dataset. All literature model data from Adrian et al. (2025). Model HLM CLint RLM CLint MDCK logS PPB Human PPB Rat Mean GBT models RDKit .3846±.0170.3846 [rgb]0.5,0.5,0.5 ±.0170 .4337±.0041.4337 [rgb]0.5,0.5,0.5 ±.0041 .3921±.0071.3921 [rgb]0.5,0.5,0.5 ±.0071 .2143±.0069.2143 [rgb]0.5,0.5,0.5 ±.0069 .3666±.0005.3666 [rgb]0.5,0.5,0.5 ±.0005 .3733±.0807.3733 [rgb]0.5,0.5,0.5 ±.0807 .3608±.0134.3608 [rgb]0.5,0.5,0.5 ±.0134 Morgan .3256±.0244.3256 [rgb]0.5,0.5,0.5 ±.0244 .3722±.0079.3722 [rgb]0.5,0.5,0.5 ±.0079 .3635±.0093.3635 [rgb]0.5,0.5,0.5 ±.0093 .1569±.0032.1569 [rgb]0.5,0.5,0.5 ±.0032 .3720±.0551.3720 [rgb]0.5,0.5,0.5 ±.0551 .0372±.0293.0372 [rgb]0.5,0.5,0.5 ±.0293 .2712±.0103.2712 [rgb]0.5,0.5,0.5 ±.0103 Morgan + RDKit .4411±.0020.4411 [rgb]0.5,0.5,0.5 ±.0020 .4950±.0055.4950 [rgb]0.5,0.5,0.5 ±.0055 .4397±.0073.4397 [rgb]0.5,0.5,0.5 ±.0073 .2346±.0007.2346 [rgb]0.5,0.5,0.5 ±.0007 .4152±.0089.4152 [rgb]0.5,0.5,0.5 ±.0089 .3111±.0075.3111 [rgb]0.5,0.5,0.5 ±.0075 .3895±.0008.3895 [rgb]0.5,0.5,0.5 ±.0008 Morgan + RDKit + Avalon + ErG .4469±.0229.4469 [rgb]0.5,0.5,0.5 ±.0229 .4800±.0083.4800 [rgb]0.5,0.5,0.5 ±.0083 .3881±.0075.3881 [rgb]0.5,0.5,0.5 ±.0075 .2428±.0125.2428 [rgb]0.5,0.5,0.5 ±.0125 .3454±.0115.3454 [rgb]0.5,0.5,0.5 ±.0115 .2428±.0730.2428 [rgb]0.5,0.5,0.5 ±.0730 .3577±.0146.3577 [rgb]0.5,0.5,0.5 ±.0146 Atom-JEPA Last Layer .2940±.0205.2940 [rgb]0.5,0.5,0.5 ±.0205 .3840±.0111.3840 [rgb]0.5,0.5,0.5 ±.0111 .2995±.0118.2995 [rgb]0.5,0.5,0.5 ±.0118 .2293±.0027.2293 [rgb]0.5,0.5,0.5 ±.0027 .3661±.0463.3661 [rgb]0.5,0.5,0.5 ±.0463 .2613±.0799.2613 [rgb]0.5,0.5,0.5 ±.0799 .3057±.0057.3057 [rgb]0.5,0.5,0.5 ±.0057 Atom-JEPA All Layer .3598±.0168.3598 [rgb]0.5,0.5,0.5 ±.0168 .4226±.0057.4226 [rgb]0.5,0.5,0.5 ±.0057 .4018±.0167.4018 [rgb]0.5,0.5,0.5 ±.0167 .2429±.0378.2429 [rgb]0.5,0.5,0.5 ±.0378 .4674±.0436.4674 [rgb]0.5,0.5,0.5 ±.0436 .4564±.0267.4564 [rgb]0.5,0.5,0.5 ±.0267 .3918±.0008.3918 [rgb]0.5,0.5,0.5 ±.0008 MLP models RDKit .4821±.0022.4821 [rgb]0.5,0.5,0.5 ±.0022 .5182±.0073.5182 [rgb]0.5,0.5,0.5 ±.0073 .4707±.0073.4707 [rgb]0.5,0.5,0.5 ±.0073 .2141±.0092.2141 [rgb]0.5,0.5,0.5 ±.0092 .4479±.0370.4479 [rgb]0.5,0.5,0.5 ±.0370 .4347±.0364.4347 [rgb]0.5,0.5,0.5 ±.0364 .4279±.0141.4279 [rgb]0.5,0.5,0.5 ±.0141 Morgan .2489±.0071.2489 [rgb]0.5,0.5,0.5 ±.0071 .3001±.0031.3001 [rgb]0.5,0.5,0.5 ±.0031 .2284±.0060.2284 [rgb]0.5,0.5,0.5 ±.0060 .2256±.0144.2256 [rgb]0.5,0.5,0.5 ±.0144 .3418±.0352.3418 [rgb]0.5,0.5,0.5 ±.0352 .2555±.0316.2555 [rgb]0.5,0.5,0.5 ±.0316 .2667±.0119.2667 [rgb]0.5,0.5,0.5 ±.0119 Morgan + RDKit .4223±.0139.4223 [rgb]0.5,0.5,0.5 ±.0139 .4911±.0116.4911 [rgb]0.5,0.5,0.5 ±.0116 .4131±.0040.4131 [rgb]0.5,0.5,0.5 ±.0040 .1898±.0021.1898 [rgb]0.5,0.5,0.5 ±.0021 .4922±.0395.4922 [rgb]0.5,0.5,0.5 ±.0395 .4088±.0663.4088 [rgb]0.5,0.5,0.5 ±.0663 .4029±.0123.4029 [rgb]0.5,0.5,0.5 ±.0123 Morgan + RDKit + Avalon + ErG .4639±.0092.4639 [rgb]0.5,0.5,0.5 ±.0092 .5312±.0098.5312 [rgb]0.5,0.5,0.5 ±.0098 .5027±.0168.5027 [rgb]0.5,0.5,0.5 ±.0168 .2164±.0070.2164 [rgb]0.5,0.5,0.5 ±.0070 .4828±.0407.4828 [rgb]0.5,0.5,0.5 ±.0407 .3682±.0311.3682 [rgb]0.5,0.5,0.5 ±.0311 .4275±.0135.4275 [rgb]0.5,0.5,0.5 ±.0135 Atom-JEPA Last Layer .4515±.0043.4515 [rgb]0.5,0.5,0.5 ±.0043 .4706±.0036.4706 [rgb]0.5,0.5,0.5 ±.0036 .4180±.0013.4180 [rgb]0.5,0.5,0.5 ±.0013 .3240±.0147.3240 [rgb]0.5,0.5,0.5 ±.0147 .4224±.0350.4224 [rgb]0.5,0.5,0.5 ±.0350 .3639±.0450.3639 [rgb]0.5,0.5,0.5 ±.0450 .4084±.0159.4084 [rgb]0.5,0.5,0.5 ±.0159 Atom-JEPA All Layer .4566±.0053.4566 [rgb]0.5,0.5,0.5 ±.0053 .4836±.0078.4836 [rgb]0.5,0.5,0.5 ±.0078 .5208±.0054.5208 [rgb]0.5,0.5,0.5 ±.0054 .3518±.0005.3518 [rgb]0.5,0.5,0.5 ±.0005 .5498±.0067.5498 [rgb]0.5,0.5,0.5 ±.0067 .5042±.0059.5042 [rgb]0.5,0.5,0.5 ±.0059 .4778±.0011.4778 [rgb]0.5,0.5,0.5 ±.0011 Finetuning Atom-JEPA .6384±.0048.6384 [rgb]0.5,0.5,0.5 ±.0048 .6417±.0053.6417 [rgb]0.5,0.5,0.5 ±.0053 .6021±.0104.6021 [rgb]0.5,0.5,0.5 ±.0104 .3728±.0154.3728 [rgb]0.5,0.5,0.5 ±.0154 .5808±.0187.5808 [rgb]0.5,0.5,0.5 ±.0187 .4332±.0532.4332 [rgb]0.5,0.5,0.5 ±.0532 .5448±.0146.5448 [rgb]0.5,0.5,0.5 ±.0146 Atom-JEPA (10-conf. ensemble) .6444±.0048.6444 [rgb]0.5,0.5,0.5 ±.0048 .6465±.0052.6465 [rgb]0.5,0.5,0.5 ±.0052 .6070±.0105.6070 [rgb]0.5,0.5,0.5 ±.0105 .3777±.0153.3777 [rgb]0.5,0.5,0.5 ±.0153 .5881±.0190.5881 [rgb]0.5,0.5,0.5 ±.0190 .4380±.0534.4380 [rgb]0.5,0.5,0.5 ±.0534 .5503±.0146.5503 [rgb]0.5,0.5,0.5 ±.0146 Literature models MolFormer ST .510±.004.510 [rgb]0.5,0.5,0.5 ±.004 .570±.015.570 [rgb]0.5,0.5,0.5 ±.015 .415±.007.415 [rgb]0.5,0.5,0.5 ±.007 .181±.010.181 [rgb]0.5,0.5,0.5 ±.010 .269±.035.269 [rgb]0.5,0.5,0.5 ±.035 .119±.023.119 [rgb]0.5,0.5,0.5 ±.023 .344.344 Chemprop ST .566±.022.566 [rgb]0.5,0.5,0.5 ±.022 .596±.014.596 [rgb]0.5,0.5,0.5 ±.014 .475±.009.475 [rgb]0.5,0.5,0.5 ±.009 .296±.027.296 [rgb]0.5,0.5,0.5 ±.027 .190±.012.190 [rgb]0.5,0.5,0.5 ±.012 .121±.085.121 [rgb]0.5,0.5,0.5 ±.085 .374.374 KPGT STcomplex .505±.004.505 [rgb]0.5,0.5,0.5 ±.004 .548±.013.548 [rgb]0.5,0.5,0.5 ±.013 .520±.007.520 [rgb]0.5,0.5,0.5 ±.007 .337±.003.337 [rgb]0.5,0.5,0.5 ±.003 .291±.031.291 [rgb]0.5,0.5,0.5 ±.031 .269±.043.269 [rgb]0.5,0.5,0.5 ±.043 .412.412 KPGT ST .580±.009.580 [rgb]0.5,0.5,0.5 ±.009 .597±.013.597 [rgb]0.5,0.5,0.5 ±.013 .538±.012.538 [rgb]0.5,0.5,0.5 ±.012 .369±.012.369 [rgb]0.5,0.5,0.5 ±.012 .358±.062.358 [rgb]0.5,0.5,0.5 ±.062 .284±.064.284 [rgb]0.5,0.5,0.5 ±.064 .454.454 KERMT ST .600±.019.600 [rgb]0.5,0.5,0.5 ±.019 .577±.017.577 [rgb]0.5,0.5,0.5 ±.017 .525±.009.525 [rgb]0.5,0.5,0.5 ±.009 .343±.009.343 [rgb]0.5,0.5,0.5 ±.009 .381±.018.381 [rgb]0.5,0.5,0.5 ±.018 .314±.064.314 [rgb]0.5,0.5,0.5 ±.064 .457.457 Chemprop MT .558±.019.558 [rgb]0.5,0.5,0.5 ±.019 .602±.011.602 [rgb]0.5,0.5,0.5 ±.011 .519±.012.519 [rgb]0.5,0.5,0.5 ±.012 .380±.012.380 [rgb]0.5,0.5,0.5 ±.012 .572±.059.572 [rgb]0.5,0.5,0.5 ±.059 .501±.046.501 [rgb]0.5,0.5,0.5 ±.046 .522.522 KPGT MT .562±.007.562 [rgb]0.5,0.5,0.5 ±.007 .580±.020.580 [rgb]0.5,0.5,0.5 ±.020 .529±.007.529 [rgb]0.5,0.5,0.5 ±.007 .318±.013.318 [rgb]0.5,0.5,0.5 ±.013 .362±.019.362 [rgb]0.5,0.5,0.5 ±.019 .315±.020.315 [rgb]0.5,0.5,0.5 ±.020 .444.444 KERMT MT .634±.010.634 [rgb]0.5,0.5,0.5 ±.010 .602±.011.602 [rgb]0.5,0.5,0.5 ±.011 .535±.009.535 [rgb]0.5,0.5,0.5 ±.009 .365±.008.365 [rgb]0.5,0.5,0.5 ±.008 .481±.040.481 [rgb]0.5,0.5,0.5 ±.040 .427±.038.427 [rgb]0.5,0.5,0.5 ±.038 .507.507 E.4 ExpansionRX and ChEMBL-MT Table 18: Test MAE mean ± standard deviation across cluster splits across tasks for ExpansionRX dataset. † : Data from Xue et al. (2026) . Model D KSOL HLM CLint MLM CLint Caco-2 Papp A→ →B Caco-2 Efflux MPPB MBPB MGMB Macro test GBDT models Morgan † .517±.011.517 [rgb]0.5,0.5,0.5 ±.011 .490±.028.490 [rgb]0.5,0.5,0.5 ±.028 .382±.007.382 [rgb]0.5,0.5,0.5 ±.007 .406±.011.406 [rgb]0.5,0.5,0.5 ±.011 .511±.013.511 [rgb]0.5,0.5,0.5 ±.013 .518±.032.518 [rgb]0.5,0.5,0.5 ±.032 .313±.020.313 [rgb]0.5,0.5,0.5 ±.020 .317±.007.317 [rgb]0.5,0.5,0.5 ±.007 .396±.012.396 [rgb]0.5,0.5,0.5 ±.012 .428±.004.428 [rgb]0.5,0.5,0.5 ±.004 RDKit .567±.009.567 [rgb]0.5,0.5,0.5 ±.009 .535±.007.535 [rgb]0.5,0.5,0.5 ±.007 .363±.004.363 [rgb]0.5,0.5,0.5 ±.004 .401±.008.401 [rgb]0.5,0.5,0.5 ±.008 .468±.005.468 [rgb]0.5,0.5,0.5 ±.005 .494±.010.494 [rgb]0.5,0.5,0.5 ±.010 .346±.011.346 [rgb]0.5,0.5,0.5 ±.011 .322±.005.322 [rgb]0.5,0.5,0.5 ±.005 .353±.014.353 [rgb]0.5,0.5,0.5 ±.014 .428±.002.428 [rgb]0.5,0.5,0.5 ±.002 Morgan + RDKit .523±.009.523 [rgb]0.5,0.5,0.5 ±.009 .540±.010.540 [rgb]0.5,0.5,0.5 ±.010 .355±.009.355 [rgb]0.5,0.5,0.5 ±.009 .409±.008.409 [rgb]0.5,0.5,0.5 ±.008 .458±.009.458 [rgb]0.5,0.5,0.5 ±.009 .514±.012.514 [rgb]0.5,0.5,0.5 ±.012 .332±.006.332 [rgb]0.5,0.5,0.5 ±.006 .277±.008.277 [rgb]0.5,0.5,0.5 ±.008 .357±.013.357 [rgb]0.5,0.5,0.5 ±.013 .418±.003.418 [rgb]0.5,0.5,0.5 ±.003 Morgan + RDKit + Avalon + ErG .511±.019.511 [rgb]0.5,0.5,0.5 ±.019 .477±.010.477 [rgb]0.5,0.5,0.5 ±.010 .371±.009.371 [rgb]0.5,0.5,0.5 ±.009 .445±.017.445 [rgb]0.5,0.5,0.5 ±.017 .464±.022.464 [rgb]0.5,0.5,0.5 ±.022 .482±.032.482 [rgb]0.5,0.5,0.5 ±.032 .333±.012.333 [rgb]0.5,0.5,0.5 ±.012 .311±.021.311 [rgb]0.5,0.5,0.5 ±.021 .328±.022.328 [rgb]0.5,0.5,0.5 ±.022 .414±.005.414 [rgb]0.5,0.5,0.5 ±.005 Mol-JEPA cls .586±.005.586 [rgb]0.5,0.5,0.5 ±.005 .521±.007.521 [rgb]0.5,0.5,0.5 ±.007 .381±.006.381 [rgb]0.5,0.5,0.5 ±.006 .435±.007.435 [rgb]0.5,0.5,0.5 ±.007 .530±.005.530 [rgb]0.5,0.5,0.5 ±.005 .524±.004.524 [rgb]0.5,0.5,0.5 ±.004 .337±.015.337 [rgb]0.5,0.5,0.5 ±.015 .376±.018.376 [rgb]0.5,0.5,0.5 ±.018 .331±.007.331 [rgb]0.5,0.5,0.5 ±.007 .447±.005.447 [rgb]0.5,0.5,0.5 ±.005 Atom-JEPA Last Layer .613±.017.613 [rgb]0.5,0.5,0.5 ±.017 .495±.004.495 [rgb]0.5,0.5,0.5 ±.004 .389±.004.389 [rgb]0.5,0.5,0.5 ±.004 .426±.003.426 [rgb]0.5,0.5,0.5 ±.003 .586±.020.586 [rgb]0.5,0.5,0.5 ±.020 .556±.008.556 [rgb]0.5,0.5,0.5 ±.008 .328±.018.328 [rgb]0.5,0.5,0.5 ±.018 .327±.014.327 [rgb]0.5,0.5,0.5 ±.014 .374±.010.374 [rgb]0.5,0.5,0.5 ±.010 .455±.004.455 [rgb]0.5,0.5,0.5 ±.004 Atom-JEPA All Layers .577±.015.577 [rgb]0.5,0.5,0.5 ±.015 .509±.004.509 [rgb]0.5,0.5,0.5 ±.004 .357±.008.357 [rgb]0.5,0.5,0.5 ±.008 .431±.008.431 [rgb]0.5,0.5,0.5 ±.008 .533±.017.533 [rgb]0.5,0.5,0.5 ±.017 .522±.009.522 [rgb]0.5,0.5,0.5 ±.009 .354±.007.354 [rgb]0.5,0.5,0.5 ±.007 .331±.009.331 [rgb]0.5,0.5,0.5 ±.009 .309±.009.309 [rgb]0.5,0.5,0.5 ±.009 .436±.003.436 [rgb]0.5,0.5,0.5 ±.003 MLP models Morgan .635±.014.635 [rgb]0.5,0.5,0.5 ±.014 .464±.032.464 [rgb]0.5,0.5,0.5 ±.032 .417±.013.417 [rgb]0.5,0.5,0.5 ±.013 .448±.006.448 [rgb]0.5,0.5,0.5 ±.006 .630±.021.630 [rgb]0.5,0.5,0.5 ±.021 .579±.019.579 [rgb]0.5,0.5,0.5 ±.019 .261±.003.261 [rgb]0.5,0.5,0.5 ±.003 .306±.014.306 [rgb]0.5,0.5,0.5 ±.014 .292±.011.292 [rgb]0.5,0.5,0.5 ±.011 .448±.006.448 [rgb]0.5,0.5,0.5 ±.006 RDKit .647±.010.647 [rgb]0.5,0.5,0.5 ±.010 .598±.008.598 [rgb]0.5,0.5,0.5 ±.008 .406±.006.406 [rgb]0.5,0.5,0.5 ±.006 .440±.006.440 [rgb]0.5,0.5,0.5 ±.006 .516±.004.516 [rgb]0.5,0.5,0.5 ±.004 .548±.004.548 [rgb]0.5,0.5,0.5 ±.004 .437±.016.437 [rgb]0.5,0.5,0.5 ±.016 .391±.011.391 [rgb]0.5,0.5,0.5 ±.011 .414±.013.414 [rgb]0.5,0.5,0.5 ±.013 .489±.006.489 [rgb]0.5,0.5,0.5 ±.006 Morgan + RDKit .494±.004.494 [rgb]0.5,0.5,0.5 ±.004 .418±.014.418 [rgb]0.5,0.5,0.5 ±.014 .373±.011.373 [rgb]0.5,0.5,0.5 ±.011 .454±.020.454 [rgb]0.5,0.5,0.5 ±.020 .558±.012.558 [rgb]0.5,0.5,0.5 ±.012 .522±.010.522 [rgb]0.5,0.5,0.5 ±.010 .259±.009.259 [rgb]0.5,0.5,0.5 ±.009 .263±.009.263 [rgb]0.5,0.5,0.5 ±.009 .300±.006.300 [rgb]0.5,0.5,0.5 ±.006 .405±.006.405 [rgb]0.5,0.5,0.5 ±.006 Morgan + RDKit + Avalon + ErG .476±.010.476 [rgb]0.5,0.5,0.5 ±.010 .389±.004.389 [rgb]0.5,0.5,0.5 ±.004 .396±.004.396 [rgb]0.5,0.5,0.5 ±.004 .419±.019.419 [rgb]0.5,0.5,0.5 ±.019 .512±.011.512 [rgb]0.5,0.5,0.5 ±.011 .454±.007.454 [rgb]0.5,0.5,0.5 ±.007 .229±.008.229 [rgb]0.5,0.5,0.5 ±.008 .244±.004.244 [rgb]0.5,0.5,0.5 ±.004 .282±.003.282 [rgb]0.5,0.5,0.5 ±.003 .378±.004.378 [rgb]0.5,0.5,0.5 ±.004 Mol-JEPA CLS .573±.013.573 [rgb]0.5,0.5,0.5 ±.013 .509±.020.509 [rgb]0.5,0.5,0.5 ±.020 .374±.004.374 [rgb]0.5,0.5,0.5 ±.004 .456±.007.456 [rgb]0.5,0.5,0.5 ±.007 .457±.005.457 [rgb]0.5,0.5,0.5 ±.005 .483±.010.483 [rgb]0.5,0.5,0.5 ±.010 .338±.017.338 [rgb]0.5,0.5,0.5 ±.017 .372±.021.372 [rgb]0.5,0.5,0.5 ±.021 .394±.014.394 [rgb]0.5,0.5,0.5 ±.014 .440±.009.440 [rgb]0.5,0.5,0.5 ±.009 Mol-JEPA Modalities .545±.014.545 [rgb]0.5,0.5,0.5 ±.014 .486±.010.486 [rgb]0.5,0.5,0.5 ±.010 .396±.011.396 [rgb]0.5,0.5,0.5 ±.011 .434±.018.434 [rgb]0.5,0.5,0.5 ±.018 .477±.010.477 [rgb]0.5,0.5,0.5 ±.010 .457±.012.457 [rgb]0.5,0.5,0.5 ±.012 .356±.007.356 [rgb]0.5,0.5,0.5 ±.007 .369±.007.369 [rgb]0.5,0.5,0.5 ±.007 .337±.003.337 [rgb]0.5,0.5,0.5 ±.003 .428±.003.428 [rgb]0.5,0.5,0.5 ±.003 Atom-JEPA All Layers .624±.015.624 [rgb]0.5,0.5,0.5 ±.015 .522±.012.522 [rgb]0.5,0.5,0.5 ±.012 .361±.009.361 [rgb]0.5,0.5,0.5 ±.009 .445±.007.445 [rgb]0.5,0.5,0.5 ±.007 .520±.010.520 [rgb]0.5,0.5,0.5 ±.010 .551±.012.551 [rgb]0.5,0.5,0.5 ±.012 .337±.011.337 [rgb]0.5,0.5,0.5 ±.011 .265±.005.265 [rgb]0.5,0.5,0.5 ±.005 .283±.004.283 [rgb]0.5,0.5,0.5 ±.004 .434±.006.434 [rgb]0.5,0.5,0.5 ±.006 Finetuning No Pretrain .390±.007.390 [rgb]0.5,0.5,0.5 ±.007 .376±.005.376 [rgb]0.5,0.5,0.5 ±.005 .347±.007.347 [rgb]0.5,0.5,0.5 ±.007 .411±.005.411 [rgb]0.5,0.5,0.5 ±.005 .529±.005.529 [rgb]0.5,0.5,0.5 ±.005 .486±.007.486 [rgb]0.5,0.5,0.5 ±.007 .258±.007.258 [rgb]0.5,0.5,0.5 ±.007 .227±.007.227 [rgb]0.5,0.5,0.5 ±.007 .282±.004.282 [rgb]0.5,0.5,0.5 ±.004 .367±.004.367 [rgb]0.5,0.5,0.5 ±.004 Atom-JEPA .363±.006.363 [rgb]0.5,0.5,0.5 ±.006 .346±.003.346 [rgb]0.5,0.5,0.5 ±.003 .316±.006.316 [rgb]0.5,0.5,0.5 ±.006 .403±.006.403 [rgb]0.5,0.5,0.5 ±.006 .475±.006.475 [rgb]0.5,0.5,0.5 ±.006 .439±.006.439 [rgb]0.5,0.5,0.5 ±.006 .218±.006.218 [rgb]0.5,0.5,0.5 ±.006 .188±.003.188 [rgb]0.5,0.5,0.5 ±.003 .259±.004.259 [rgb]0.5,0.5,0.5 ±.004 .334±.002.334 [rgb]0.5,0.5,0.5 ±.002 Atom-JEPA (10-conf. ensemble) .363±.006.363 [rgb]0.5,0.5,0.5 ±.006 .344±.003.344 [rgb]0.5,0.5,0.5 ±.003 .316±.006.316 [rgb]0.5,0.5,0.5 ±.006 .403±.006.403 [rgb]0.5,0.5,0.5 ±.006 .475±.006.475 [rgb]0.5,0.5,0.5 ±.006 .439±.006.439 [rgb]0.5,0.5,0.5 ±.006 .218±.005.218 [rgb]0.5,0.5,0.5 ±.005 .188±.003.188 [rgb]0.5,0.5,0.5 ±.003 .259±.004.259 [rgb]0.5,0.5,0.5 ±.004 .334±.002.334 [rgb]0.5,0.5,0.5 ±.002 Literature models Chemprop .449±.009.449 [rgb]0.5,0.5,0.5 ±.009 .404±.006.404 [rgb]0.5,0.5,0.5 ±.006 .552±.105.552 [rgb]0.5,0.5,0.5 ±.105 .487±.025.487 [rgb]0.5,0.5,0.5 ±.025 .528±.032.528 [rgb]0.5,0.5,0.5 ±.032 .456±.024.456 [rgb]0.5,0.5,0.5 ±.024 .251±.016.251 [rgb]0.5,0.5,0.5 ±.016 .339±.019.339 [rgb]0.5,0.5,0.5 ±.019 .371±.031.371 [rgb]0.5,0.5,0.5 ±.031 .426±.012.426 [rgb]0.5,0.5,0.5 ±.012 CKERMT Our runs .407±.009.407 [rgb]0.5,0.5,0.5 ±.009 .406±.008.406 [rgb]0.5,0.5,0.5 ±.008 .350±.010.350 [rgb]0.5,0.5,0.5 ±.010 .385±.009.385 [rgb]0.5,0.5,0.5 ±.009 .472±.009.472 [rgb]0.5,0.5,0.5 ±.009 .466±.007.466 [rgb]0.5,0.5,0.5 ±.007 .250±.003.250 [rgb]0.5,0.5,0.5 ±.003 .273±.011.273 [rgb]0.5,0.5,0.5 ±.011 .333±.023.333 [rgb]0.5,0.5,0.5 ±.023 .372±.003.372 [rgb]0.5,0.5,0.5 ±.003 KERMT base‡ .393.393 .417.417 .429.429 .458.458 .472.472 .489.489 .247.247 .225.225 .287.287 .380.380 KERMT task-specific‡ .396.396 .396.396 .400.400 .423.423 .469.469 .479.479 .263.263 .251.251 .299.299 .375.375 CMIM only task-specific‡ .461.461 .477.477 .418.418 .420.420 .468.468 .479.479 .284.284 .247.247 .293.293 .394.394 CKERMT base – – – – – – – – – – CKERMT task-specific‡ .390.390 .387.387 .405.405 .396.396 .470.470 .471.471 .252.252 .232.232 .286.286 .365.365 CKERMT upscale task-specific† .382.382 .385.385 .401.401 .394.394 .463.463 .455.455 .259.259 .233.233 .288.288 .362.362 CKERMT Biogen task-specific† .388.388 .395.395 .393.393 .406.406 .457.457 .460.460 .257.257 .234.234 .289.289 .364.364 CKERMT Biogen sampled-30k task-specific† .393.393 .373.373 .413.413 .410.410 .453.453 .447.447 .228.228 .220.220 .290.290 .359.359 CKERMT Biogen+OA+CM task-specific† .377.377 .375.375 .397.397 .395.395 .460.460 .459.459 .255.255 .226.226 .300.300 .360.360 CKERMT upscale+Biogen task-specific† .388.388 .375.375 .411.411 .403.403 .452.452 .458.458 .249.249 .225.225 .288.288 .361.361 Table 19: Test MAE on the ChEMBL-MT benchmark. Values marked with ‡ are taken from Figure 15 in Xue et al. (2026). Mic-H, Mic-M, and Mic-R denote microsomal clearance in human, mouse, and rat, respectively. Tot-D, Tot-H, Tot-M, and Tot-R denote total clearance in dog, human, monkey, and rat. fu-D, fu-H, fu-M, and fu-R denote the fraction unbound in plasma for the corresponding species. VD-D, VD-H, VD-M, and VD-R denote steady-state volume of distribution. Kin-S and Therm-S denote kinetic and thermodynamic logSaq S_aq. Mean is the Overall value reported in Figure 15. The figure reports means to three decimal places without numerical standard deviations. Lower is better. Model Mic-H Mic-M Mic-R Tot-D Tot-H Tot-M Tot-R 2C8 2C9 2D6 3A4 fu-D fu-H D fu-M Papp P-gp fu-R VD-D VD-H VD-M VD-R hERG Kin-S Therm-S Mean LBGM Morgan .421.421 .452.452 .502.502 .463.463 .512.512 .347.347 .470.470 .506.506 .570.570 .617.617 .649.649 .504.504 .458.458 .766.766 .537.537 .476.476 .456.456 .504.504 .430.430 .466.466 .362.362 .488.488 .514.514 .379.379 1.1351.135 .519.519 RDKit .400.400 .406.406 .506.506 .421.421 .472.472 .313.313 .459.459 .513.513 .554.554 .589.589 .660.660 .434.434 .392.392 .630.630 .539.539 .467.467 .436.436 .512.512 .389.389 .403.403 .356.356 .444.444 .530.530 .369.369 .813.813 .480.480 RDKit + Morgan .399.399 .385.385 .513.513 .411.411 .464.464 .335.335 .456.456 .496.496 .556.556 .576.576 .633.633 .443.443 .390.390 .614.614 .549.549 .457.457 .433.433 .504.504 .377.377 .409.409 .425.425 .443.443 .509.509 .352.352 .809.809 .478.478 Morgan + RDKit + Avalon + ErG .401.401 .442.442 .498.498 .453.453 .508.508 .317.317 .454.454 .523.523 .551.551 .608.608 .634.634 .433.433 .402.402 .616.616 .507.507 .480.480 .431.431 .516.516 .421.421 .424.424 .416.416 .468.468 .506.506 .361.361 .840.840 .488.488 Mol-JEPA CLS .431.431 .415.415 .542.542 .417.417 .506.506 .304.304 .449.449 .515.515 .567.567 .605.605 .665.665 .483.483 .449.449 .772.772 .573.573 .534.534 .438.438 .468.468 .442.442 .445.445 .392.392 .477.477 .545.545 .389.389 1.1491.149 .519.519 Atom-JEPA Last Layers .455.455 .436.436 .531.531 .413.413 .506.506 .333.333 .445.445 .509.509 .602.602 .635.635 .729.729 .479.479 .436.436 .868.868 .563.563 .536.536 .467.467 .518.518 .419.419 .463.463 .459.459 .465.465 .581.581 .407.407 .932.932 .527.527 Atom-JEPA All Layers .425.425 .431.431 .502.502 .413.413 .496.496 .339.339 .450.450 .519.519 .574.574 .606.606 .673.673 .454.454 .399.399 .751.751 .600.600 .508.508 .445.445 .491.491 .428.428 .436.436 .420.420 .457.457 .541.541 .382.382 .895.895 .505.505 MLP Morgan .443.443 .503.503 .467.467 .451.451 .548.548 .385.385 .476.476 .480.480 .597.597 .672.672 .669.669 .460.460 .478.478 .842.842 .475.475 .545.545 .495.495 .468.468 .465.465 .459.459 .488.488 .471.471 .598.598 .442.442 1.3651.365 .550.550 RDKit .413.413 .556.556 .566.566 .437.437 .481.481 .320.320 .421.421 .585.585 .591.591 .651.651 .688.688 .460.460 .408.408 .666.666 .391.391 .502.502 .430.430 .477.477 .365.365 .430.430 .336.336 .450.450 .559.559 .398.398 .893.893 .499.499 RDKit + Morgan .513.513 .485.485 .560.560 .467.467 .553.553 .346.346 .479.479 .528.528 .599.599 .677.677 .730.730 .598.598 .681.681 1.0401.040 .581.581 .613.613 .507.507 .558.558 .542.542 .514.514 .526.526 .556.556 .672.672 .554.554 1.8181.818 .628.628 Morgan + RDKit + Avalon + ErG .510.510 .480.480 .559.559 .459.459 .548.548 .347.347 .476.476 .521.521 .599.599 .675.675 .729.729 .554.554 .623.623 .981.981 .598.598 .606.606 .508.508 .512.512 .517.517 .491.491 .506.506 .521.521 .672.672 .485.485 1.3911.391 .595.595 Mol-JEPA cls .428.428 .460.460 .553.553 .388.388 .513.513 .394.394 .446.446 .564.564 .560.560 .619.619 .652.652 .410.410 .382.382 .682.682 .363.363 .543.543 .451.451 .389.389 .383.383 .427.427 .414.414 .461.461 .550.550 .378.378 1.0191.019 .497.497 Mol-JEPA Modalities .405.405 .444.444 .545.545 .395.395 .510.510 .358.358 .428.428 .536.536 .533.533 .584.584 .647.647 .477.477 .388.388 .619.619 .445.445 .505.505 .444.444 .434.434 .379.379 .411.411 .381.381 .460.460 .521.521 .369.369 .979.979 .488.488 Atom-JEPA Last Layers .443.443 .482.482 .559.559 .390.390 .478.478 .337.337 .435.435 .492.492 .597.597 .614.614 .716.716 .437.437 .415.415 .737.737 .523.523 .544.544 .457.457 .495.495 .435.435 .476.476 .416.416 .454.454 .572.572 .395.395 .861.861 .510.510 Atom-JEPA All Layers .451.451 .459.459 .577.577 .402.402 .505.505 .359.359 .447.447 .502.502 .597.597 .617.617 .711.711 .437.437 .412.412 .746.746 .471.471 .556.556 .446.446 .462.462 .490.490 .482.482 .411.411 .468.468 .578.578 .405.405 .948.948 .518.518 Finetuning No Pretrain .376.376 .417.417 .473.473 .414.414 .488.488 .301.301 .443.443 .472.472 .565.565 .575.575 .585.585 .442.442 .382.382 .442.442 .384.384 .457.457 .443.443 .461.461 .561.561 .281.281 .755.755 .406.406 .407.407 .400.400 .427.427 .454.454 Atom-JEPA .365.365 .447.447 .472.472 .402.402 .472.472 .319.319 .436.436 .451.451 .534.534 .607.607 .592.592 .420.420 .378.378 .409.409 .390.390 .471.471 .459.459 .484.484 .560.560 .279.279 .781.781 .422.422 .416.416 .423.423 .427.427 .457.457 Atom-JEPA (10-conf. ensemble) .360.360 .443.443 .468.468 .401.401 .470.470 .316.316 .433.433 .447.447 .527.527 .599.599 .585.585 .419.419 .374.374 .408.408 .388.388 .466.466 .453.453 .477.477 .552.552 .274.274 .774.774 .420.420 .414.414 .420.420 .426.426 .453.453 Literature models Chemprop .368.368 .490.490 .476.476 .427.427 .483.483 .377.377 .465.465 .494.494 .534.534 .621.621 .626.626 .429.429 .368.368 .536.536 .438.438 .409.409 .449.449 .431.431 .498.498 .401.401 .452.452 .472.472 .484.484 .271.271 .920.920 .477.477 CKERMT Our runs .361.361 .464.464 .459.459 .420.420 .484.484 .317.317 .463.463 .492.492 .509.509 .586.586 .582.582 .395.395 .350.350 .401.401 .443.443 .408.408 .414.414 .470.470 .516.516 .266.266 .869.869 .422.422 .398.398 .354.354 .463.463 .452.452 KERMT base† .357.357 .441.441 .439.439 .454.454 .482.482 .414.414 .482.482 .486.486 .509.509 .576.576 .604.604 .409.409 .376.376 .514.514 .442.442 .426.426 .430.430 .449.449 .417.417 .408.408 .369.369 .454.454 .480.480 .287.287 .959.959 .466.466 KERMT task-specific† .360.360 .435.435 .455.455 .460.460 .473.473 .353.353 .488.488 .495.495 .512.512 .573.573 .598.598 .413.413 .361.361 .525.525 .436.436 .428.428 .424.424 .418.418 .405.405 .387.387 .371.371 .460.460 .476.476 .276.276 .910.910 .460.460 CMIM only task-specific† .369.369 .433.433 .467.467 .439.439 .456.456 .324.324 .423.423 .514.514 .512.512 .593.593 .606.606 .388.388 .376.376 .576.576 .420.420 .448.448 .432.432 .432.432 .402.402 .385.385 .354.354 .447.447 .491.491 .293.293 .856.856 .458.458 CKERMT task-specific† .362.362 .442.442 .432.432 .423.423 .471.471 .326.326 .453.453 .499.499 .506.506 .592.592 .586.586 .414.414 .348.348 .515.515 .394.394 .424.424 .427.427 .415.415 .409.409 .383.383 .360.360 .455.455 .475.475 .267.267 .878.878 .450.450 CKERMT upscale task-specific† .354.354 .433.433 .439.439 .425.425 .479.479 .338.338 .442.442 .490.490 .495.495 .568.568 .582.582 .402.402 .348.348 .507.507 .401.401 .412.412 .420.420 .410.410 .406.406 .393.393 .370.370 .448.448 .469.469 .265.265 .834.834 .445.445 CKERMT Biogen task-specific† .360.360 .446.446 .436.436 .424.424 .466.466 .330.330 .445.445 .499.499 .508.508 .578.578 .580.580 .414.414 .348.348 .516.516 .388.388 .413.413 .425.425 .417.417 .404.404 .388.388 .366.366 .442.442 .474.474 .269.269 .885.885 .449.449 CKERMT Biogen sampled-30k task-specific † .361.361 .471.471 .436.436 .421.421 .474.474 .332.332 .447.447 .494.494 .500.500 .576.576 .571.571 .427.427 .343.343 .510.510 .399.399 .407.407 .429.429 .397.397 .404.404 .387.387 .363.363 .444.444 .459.459 .263.263 .816.816 .445.445 CKERMT Biogen+OA+CM task-specific † .355.355 .443.443 .432.432 .420.420 .473.473 .345.345 .446.446 .480.480 .499.499 .572.572 .569.569 .409.409 .349.349 .506.506 .370.370 .410.410 .420.420 .402.402 .400.400 .386.386 .356.356 .452.452 .469.469 .262.262 .814.814 .442.442 CKERMT upscale+Biogen task-specific † .365.365 .443.443 .436.436 .413.413 .467.467 .345.345 .456.456 .490.490 .496.496 .582.582 .588.588 .417.417 .353.353 .506.506 .394.394 .422.422 .426.426 .412.412 .392.392 .383.383 .347.347 .448.448 .467.467 .265.265 .841.841 .446.446 E.5 TDC ADMET Table 20: Results on the TDC-ADMET benchmark: absorption. Columns correspond, from left to right, to caco2_wang, hia_hou, pgp_broccatelli, bioavailability_ma, lipophilicity_astrazeneca, and solubility_aqsoldb. Numbers from Koleiev et al. (2026). Model Caco2 HIA P-gp Bioav. Lipo. Solub. Unit MAE↓ AUROC↑ AUROC↑ AUROC↑ MAE↓ MAE↓ GBDT models Morgan .3530±.0370.3530 [rgb]0.5,0.5,0.5 ±.0370 .8170±.0300.8170 [rgb]0.5,0.5,0.5 ±.0300 .8650±.0230.8650 [rgb]0.5,0.5,0.5 ±.0230 .5690±.0730.5690 [rgb]0.5,0.5,0.5 ±.0730 .6060±.0170.6060 [rgb]0.5,0.5,0.5 ±.0170 .9700±.0060.9700 [rgb]0.5,0.5,0.5 ±.0060 RDKit .3020±.0110.3020 [rgb]0.5,0.5,0.5 ±.0110 .8900±.0500.8900 [rgb]0.5,0.5,0.5 ±.0500 .8690±.0190.8690 [rgb]0.5,0.5,0.5 ±.0190 .5800±.0430.5800 [rgb]0.5,0.5,0.5 ±.0430 .5690±.0110.5690 [rgb]0.5,0.5,0.5 ±.0110 .7630±.0110.7630 [rgb]0.5,0.5,0.5 ±.0110 Morgan + RDKit .2950±.0060.2950 [rgb]0.5,0.5,0.5 ±.0060 .8940±.0400.8940 [rgb]0.5,0.5,0.5 ±.0400 .8590±.0080.8590 [rgb]0.5,0.5,0.5 ±.0080 .5960±.0550.5960 [rgb]0.5,0.5,0.5 ±.0550 .5420±.0050.5420 [rgb]0.5,0.5,0.5 ±.0050 .7760±.0280.7760 [rgb]0.5,0.5,0.5 ±.0280 Morgan + RDKit + Avalon + ErG .3290±.0150.3290 [rgb]0.5,0.5,0.5 ±.0150 .9410±.0170.9410 [rgb]0.5,0.5,0.5 ±.0170 .8690±.0160.8690 [rgb]0.5,0.5,0.5 ±.0160 .5790±.0770.5790 [rgb]0.5,0.5,0.5 ±.0770 .5400±.0110.5400 [rgb]0.5,0.5,0.5 ±.0110 .8030±.0050.8030 [rgb]0.5,0.5,0.5 ±.0050 Atom-JEPA Last Layer .3390±.0040.3390 [rgb]0.5,0.5,0.5 ±.0040 .9380±.0330.9380 [rgb]0.5,0.5,0.5 ±.0330 .8760±.0110.8760 [rgb]0.5,0.5,0.5 ±.0110 .6100±.0560.6100 [rgb]0.5,0.5,0.5 ±.0560 .7640±.0190.7640 [rgb]0.5,0.5,0.5 ±.0190 .9270±.0150.9270 [rgb]0.5,0.5,0.5 ±.0150 Atom-JEPA All Layers .3090±.0150.3090 [rgb]0.5,0.5,0.5 ±.0150 .9410±.0280.9410 [rgb]0.5,0.5,0.5 ±.0280 .8670±.0210.8670 [rgb]0.5,0.5,0.5 ±.0210 .5850±.0550.5850 [rgb]0.5,0.5,0.5 ±.0550 .6490±.0050.6490 [rgb]0.5,0.5,0.5 ±.0050 .8620±.0130.8620 [rgb]0.5,0.5,0.5 ±.0130 MLP models Morgan .4410±.0050.4410 [rgb]0.5,0.5,0.5 ±.0050 .8940±.0050.8940 [rgb]0.5,0.5,0.5 ±.0050 .8900±.0240.8900 [rgb]0.5,0.5,0.5 ±.0240 .5990±.0540.5990 [rgb]0.5,0.5,0.5 ±.0540 .6720±.0080.6720 [rgb]0.5,0.5,0.5 ±.0080 1.1700±.00801.1700 [rgb]0.5,0.5,0.5 ±.0080 RDKit .3590±.0100.3590 [rgb]0.5,0.5,0.5 ±.0100 .9790±.0050.9790 [rgb]0.5,0.5,0.5 ±.0050 .8990±.0050.8990 [rgb]0.5,0.5,0.5 ±.0050 .6890±.0080.6890 [rgb]0.5,0.5,0.5 ±.0080 .5640±.0080.5640 [rgb]0.5,0.5,0.5 ±.0080 .7930±.0380.7930 [rgb]0.5,0.5,0.5 ±.0380 RDKit + Morgan .3160±.0100.3160 [rgb]0.5,0.5,0.5 ±.0100 .9400±.0180.9400 [rgb]0.5,0.5,0.5 ±.0180 .9120±.0130.9120 [rgb]0.5,0.5,0.5 ±.0130 .6730±.0240.6730 [rgb]0.5,0.5,0.5 ±.0240 .5310±.0020.5310 [rgb]0.5,0.5,0.5 ±.0020 .8320±.0060.8320 [rgb]0.5,0.5,0.5 ±.0060 RDKit + Morgan + Avalon + ErG .3620±.1170.3620 [rgb]0.5,0.5,0.5 ±.1170 .9490±.0040.9490 [rgb]0.5,0.5,0.5 ±.0040 .9280±.0030.9280 [rgb]0.5,0.5,0.5 ±.0030 .6060±.0540.6060 [rgb]0.5,0.5,0.5 ±.0540 .6800±.2390.6800 [rgb]0.5,0.5,0.5 ±.2390 .9390±.2730.9390 [rgb]0.5,0.5,0.5 ±.2730 Atom-JEPA Last Layer .2970±.0050.2970 [rgb]0.5,0.5,0.5 ±.0050 .9440±.0140.9440 [rgb]0.5,0.5,0.5 ±.0140 .8850±.0090.8850 [rgb]0.5,0.5,0.5 ±.0090 .6370±.0330.6370 [rgb]0.5,0.5,0.5 ±.0330 .6310±.0100.6310 [rgb]0.5,0.5,0.5 ±.0100 .8000±.0130.8000 [rgb]0.5,0.5,0.5 ±.0130 Atom-JEPA All Layers .2970±.0010.2970 [rgb]0.5,0.5,0.5 ±.0010 .9600±.0190.9600 [rgb]0.5,0.5,0.5 ±.0190 .8810±.0070.8810 [rgb]0.5,0.5,0.5 ±.0070 .6040±.0150.6040 [rgb]0.5,0.5,0.5 ±.0150 .5430±.0230.5430 [rgb]0.5,0.5,0.5 ±.0230 .7720±.0190.7720 [rgb]0.5,0.5,0.5 ±.0190 Other models MapLight .276±.005.276 [rgb]0.5,0.5,0.5 ±.005 .983±.001.983 [rgb]0.5,0.5,0.5 ±.001 .932±.001.932 [rgb]0.5,0.5,0.5 ±.001 .741±.007.741 [rgb]0.5,0.5,0.5 ±.007 .542±.003.542 [rgb]0.5,0.5,0.5 ±.003 .787±.006.787 [rgb]0.5,0.5,0.5 ±.006 MapLight+GNN .290±.008.290 [rgb]0.5,0.5,0.5 ±.008 .989±.001.989 [rgb]0.5,0.5,0.5 ±.001 .938±.0.938 [rgb]0.5,0.5,0.5 ±.0 .740±.01.740 [rgb]0.5,0.5,0.5 ±.01 .525±.003.525 [rgb]0.5,0.5,0.5 ±.003 .795±.004.795 [rgb]0.5,0.5,0.5 ±.004 CaliciBoost .256±.006.256 [rgb]0.5,0.5,0.5 ±.006 – – – – – Atom-JEPA .2910±.0100.2910 [rgb]0.5,0.5,0.5 ±.0100 .9800±.0100.9800 [rgb]0.5,0.5,0.5 ±.0100 .9180±.0100.9180 [rgb]0.5,0.5,0.5 ±.0100 .6900±.0370.6900 [rgb]0.5,0.5,0.5 ±.0370 .3970±.0030.3970 [rgb]0.5,0.5,0.5 ±.0030 .7800±.0180.7800 [rgb]0.5,0.5,0.5 ±.0180 Table 21: Results on the TDC-ADMET benchmark: distribution. Columns correspond, from left to right, to b_martins, ppbr_az, and vdss_lombardo. Numbers from Koleiev et al. (2026). RDKit + Morgan and RDKit + Morgan + Avalon + ErG MLP diverged during training on VDSS Lombardo Model B PPBR VDss Unit AUROC↑ MAE↓ Spear↑ GBDT models Morgan .8300±.0110.8300 [rgb]0.5,0.5,0.5 ±.0110 8.6780±.13808.6780 [rgb]0.5,0.5,0.5 ±.1380 .6070±.0230.6070 [rgb]0.5,0.5,0.5 ±.0230 RDKit .8770±.0150.8770 [rgb]0.5,0.5,0.5 ±.0150 7.5460±.11107.5460 [rgb]0.5,0.5,0.5 ±.1110 .6220±.0090.6220 [rgb]0.5,0.5,0.5 ±.0090 Morgan + RDKit .8600±.0140.8600 [rgb]0.5,0.5,0.5 ±.0140 7.5100±.09807.5100 [rgb]0.5,0.5,0.5 ±.0980 .6250±.0260.6250 [rgb]0.5,0.5,0.5 ±.0260 Morgan + RDKit + Avalon + ErG .8600±.0260.8600 [rgb]0.5,0.5,0.5 ±.0260 8.1450±.30108.1450 [rgb]0.5,0.5,0.5 ±.3010 .5750±.0230.5750 [rgb]0.5,0.5,0.5 ±.0230 Atom-JEPA Last Layer .8540±.0080.8540 [rgb]0.5,0.5,0.5 ±.0080 8.8830±.14408.8830 [rgb]0.5,0.5,0.5 ±.1440 .5800±.0260.5800 [rgb]0.5,0.5,0.5 ±.0260 Atom-JEPA All Layers .8500±.0210.8500 [rgb]0.5,0.5,0.5 ±.0210 8.1380±.06908.1380 [rgb]0.5,0.5,0.5 ±.0690 .5840±.0400.5840 [rgb]0.5,0.5,0.5 ±.0400 MLP models Morgan .8380±.0110.8380 [rgb]0.5,0.5,0.5 ±.0110 8.5710±.05508.5710 [rgb]0.5,0.5,0.5 ±.0550 .5070±.0460.5070 [rgb]0.5,0.5,0.5 ±.0460 RDKit .8910±.0140.8910 [rgb]0.5,0.5,0.5 ±.0140 7.6240±.07307.6240 [rgb]0.5,0.5,0.5 ±.0730 .5550±.0110.5550 [rgb]0.5,0.5,0.5 ±.0110 RDKit + Morgan .9040±.0060.9040 [rgb]0.5,0.5,0.5 ±.0060 8.1030±.08808.1030 [rgb]0.5,0.5,0.5 ±.0880 nan±nannan [rgb]0.5,0.5,0.5 ± nan RDKit + Morgan + Avalon + ErG .9020±.0090.9020 [rgb]0.5,0.5,0.5 ±.0090 9.9660±1.42809.9660 [rgb]0.5,0.5,0.5 ± 1.4280 nan±nannan [rgb]0.5,0.5,0.5 ± nan Atom-JEPA Last Layer .8690±.0150.8690 [rgb]0.5,0.5,0.5 ±.0150 8.1870±.12508.1870 [rgb]0.5,0.5,0.5 ±.1250 .5370±.0430.5370 [rgb]0.5,0.5,0.5 ±.0430 Atom-JEPA All Layers .8920±.0230.8920 [rgb]0.5,0.5,0.5 ±.0230 7.9370±.03907.9370 [rgb]0.5,0.5,0.5 ±.0390 .5770±.0460.5770 [rgb]0.5,0.5,0.5 ±.0460 Other models MapLight .920±.002.920 [rgb]0.5,0.5,0.5 ±.002 7.622±.0657.622 [rgb]0.5,0.5,0.5 ±.065 .716±.005.716 [rgb]0.5,0.5,0.5 ±.005 MapLight+GNN .916±.002.916 [rgb]0.5,0.5,0.5 ±.002 7.573±.1637.573 [rgb]0.5,0.5,0.5 ±.163 .716±.006.716 [rgb]0.5,0.5,0.5 ±.006 Atom-JEPA .8960±.0150.8960 [rgb]0.5,0.5,0.5 ±.0150 7.6690±.25307.6690 [rgb]0.5,0.5,0.5 ±.2530 .6800±.0360.6800 [rgb]0.5,0.5,0.5 ±.0360 Table 22: Results on the TDC-ADMET benchmark: metabolism. Columns correspond, from left to right, to cyp2c9_veith, cyp2d6_veith, cyp3a4_veith, cyp2c9_substrate_carbonmangels, cyp2d6_substrate_carbonmangels, and cyp3a4_substrate_carbonmangels. Numbers from Koleiev et al. (2026). Model 2C9-I 2D6-I 3A4-I 2C9-S 2D6-S 3A4-S Unit AUPRC↑ AUPRC↑ AUPRC↑ AUPRC↑ AUPRC↑ AUROC↑ GBDT models Morgan .6730±.0590.6730 [rgb]0.5,0.5,0.5 ±.0590 .5300±.0100.5300 [rgb]0.5,0.5,0.5 ±.0100 .7800±.0050.7800 [rgb]0.5,0.5,0.5 ±.0050 .2810±.0000.2810 [rgb]0.5,0.5,0.5 ±.0000 .4660±.0960.4660 [rgb]0.5,0.5,0.5 ±.0960 .5990±.0190.5990 [rgb]0.5,0.5,0.5 ±.0190 RDKit .7110±.0090.7110 [rgb]0.5,0.5,0.5 ±.0090 .5880±.0150.5880 [rgb]0.5,0.5,0.5 ±.0150 .7840±.0100.7840 [rgb]0.5,0.5,0.5 ±.0100 .2970±.0250.2970 [rgb]0.5,0.5,0.5 ±.0250 .5420±.1280.5420 [rgb]0.5,0.5,0.5 ±.1280 .6170±.0280.6170 [rgb]0.5,0.5,0.5 ±.0280 Morgan + RDKit .7120±.0100.7120 [rgb]0.5,0.5,0.5 ±.0100 .6000±.0110.6000 [rgb]0.5,0.5,0.5 ±.0110 .8090±.0060.8090 [rgb]0.5,0.5,0.5 ±.0060 .2810±.0090.2810 [rgb]0.5,0.5,0.5 ±.0090 .5560±.0500.5560 [rgb]0.5,0.5,0.5 ±.0500 .6310±.0090.6310 [rgb]0.5,0.5,0.5 ±.0090 Morgan + RDKit + Avalon + ErG .6930±.0120.6930 [rgb]0.5,0.5,0.5 ±.0120 .5630±.0290.5630 [rgb]0.5,0.5,0.5 ±.0290 .8100±.0110.8100 [rgb]0.5,0.5,0.5 ±.0110 .2930±.0150.2930 [rgb]0.5,0.5,0.5 ±.0150 .5140±.1230.5140 [rgb]0.5,0.5,0.5 ±.1230 .6210±.0300.6210 [rgb]0.5,0.5,0.5 ±.0300 Atom-JEPA Last Layer .6450±.0750.6450 [rgb]0.5,0.5,0.5 ±.0750 .3360±.0790.3360 [rgb]0.5,0.5,0.5 ±.0790 .7740±.0100.7740 [rgb]0.5,0.5,0.5 ±.0100 .3090±.0370.3090 [rgb]0.5,0.5,0.5 ±.0370 .5330±.0780.5330 [rgb]0.5,0.5,0.5 ±.0780 .5880±.0390.5880 [rgb]0.5,0.5,0.5 ±.0390 Atom-JEPA All Layers .7000±.0180.7000 [rgb]0.5,0.5,0.5 ±.0180 .5100±.0050.5100 [rgb]0.5,0.5,0.5 ±.0050 .7950±.0040.7950 [rgb]0.5,0.5,0.5 ±.0040 .2940±.0170.2940 [rgb]0.5,0.5,0.5 ±.0170 .5170±.1090.5170 [rgb]0.5,0.5,0.5 ±.1090 .6130±.0340.6130 [rgb]0.5,0.5,0.5 ±.0340 MLP models Morgan .7110±.0050.7110 [rgb]0.5,0.5,0.5 ±.0050 .6040±.0050.6040 [rgb]0.5,0.5,0.5 ±.0050 .8320±.0050.8320 [rgb]0.5,0.5,0.5 ±.0050 .4320±.0330.4320 [rgb]0.5,0.5,0.5 ±.0330 .6420±.0370.6420 [rgb]0.5,0.5,0.5 ±.0370 .6240±.0130.6240 [rgb]0.5,0.5,0.5 ±.0130 RDKit .7390±.0130.7390 [rgb]0.5,0.5,0.5 ±.0130 .6260±.0030.6260 [rgb]0.5,0.5,0.5 ±.0030 .8350±.0050.8350 [rgb]0.5,0.5,0.5 ±.0050 .3900±.0140.3900 [rgb]0.5,0.5,0.5 ±.0140 .6680±.0290.6680 [rgb]0.5,0.5,0.5 ±.0290 .5990±.0160.5990 [rgb]0.5,0.5,0.5 ±.0160 RDKit + Morgan .7510±.0040.7510 [rgb]0.5,0.5,0.5 ±.0040 .6520±.0050.6520 [rgb]0.5,0.5,0.5 ±.0050 .8700±.0020.8700 [rgb]0.5,0.5,0.5 ±.0020 .4310±.0280.4310 [rgb]0.5,0.5,0.5 ±.0280 .7200±.0210.7200 [rgb]0.5,0.5,0.5 ±.0210 .6290±.0470.6290 [rgb]0.5,0.5,0.5 ±.0470 RDKit + Morgan + Avalon + ErG .7460±.0200.7460 [rgb]0.5,0.5,0.5 ±.0200 .5660±.2060.5660 [rgb]0.5,0.5,0.5 ±.2060 .8450±.0640.8450 [rgb]0.5,0.5,0.5 ±.0640 .4220±.0560.4220 [rgb]0.5,0.5,0.5 ±.0560 .6280±.1530.6280 [rgb]0.5,0.5,0.5 ±.1530 .5840±.0500.5840 [rgb]0.5,0.5,0.5 ±.0500 Atom-JEPA Last Layer .7560±.0060.7560 [rgb]0.5,0.5,0.5 ±.0060 .6480±.0060.6480 [rgb]0.5,0.5,0.5 ±.0060 .8400±.0050.8400 [rgb]0.5,0.5,0.5 ±.0050 .3680±.0190.3680 [rgb]0.5,0.5,0.5 ±.0190 .5870±.0530.5870 [rgb]0.5,0.5,0.5 ±.0530 .5950±.0230.5950 [rgb]0.5,0.5,0.5 ±.0230 Atom-JEPA All Layers .7620±.0040.7620 [rgb]0.5,0.5,0.5 ±.0040 .6470±.0050.6470 [rgb]0.5,0.5,0.5 ±.0050 .8480±.0020.8480 [rgb]0.5,0.5,0.5 ±.0020 .4150±.0710.4150 [rgb]0.5,0.5,0.5 ±.0710 .6420±.0300.6420 [rgb]0.5,0.5,0.5 ±.0300 .5900±.0120.5900 [rgb]0.5,0.5,0.5 ±.0120 Other models MapLight .786±.004.786 [rgb]0.5,0.5,0.5 ±.004 .720±.002.720 [rgb]0.5,0.5,0.5 ±.002 .881±.001.881 [rgb]0.5,0.5,0.5 ±.001 .411±.005.411 [rgb]0.5,0.5,0.5 ±.005 .718±.008.718 [rgb]0.5,0.5,0.5 ±.008 .649±.006.649 [rgb]0.5,0.5,0.5 ±.006 MapLight+GNN .859±.001.859 [rgb]0.5,0.5,0.5 ±.001 .790±.001.790 [rgb]0.5,0.5,0.5 ±.001 .916±.00.916 [rgb]0.5,0.5,0.5 ±.00 .446±.015.446 [rgb]0.5,0.5,0.5 ±.015 .716±.007.716 [rgb]0.5,0.5,0.5 ±.007 .642±.004.642 [rgb]0.5,0.5,0.5 ±.004 Atom-JEPA .7720±.0060.7720 [rgb]0.5,0.5,0.5 ±.0060 .6610±.0180.6610 [rgb]0.5,0.5,0.5 ±.0180 .8680±.0050.8680 [rgb]0.5,0.5,0.5 ±.0050 .4560±.0270.4560 [rgb]0.5,0.5,0.5 ±.0270 .6840±.0550.6840 [rgb]0.5,0.5,0.5 ±.0550 .5790±.0230.5790 [rgb]0.5,0.5,0.5 ±.0230 Table 23: Results on the TDC-ADMET benchmark: excretion. Columns correspond, from left to right, to half_life_obach, clearance_microsome_az, and clearance_hepatocyte_az. Numbers from Koleiev et al. (2026). Model Half-life Cl-Mic Cl-Hep Unit Spear↑ Spear↑ Spear↑ GBDT models Morgan .4130±.0430.4130 [rgb]0.5,0.5,0.5 ±.0430 .4670±.0290.4670 [rgb]0.5,0.5,0.5 ±.0290 .3690±.0250.3690 [rgb]0.5,0.5,0.5 ±.0250 RDKit .3520±.0350.3520 [rgb]0.5,0.5,0.5 ±.0350 .5710±.0310.5710 [rgb]0.5,0.5,0.5 ±.0310 .4030±.0480.4030 [rgb]0.5,0.5,0.5 ±.0480 Morgan + RDKit .4150±.0370.4150 [rgb]0.5,0.5,0.5 ±.0370 .5510±.0190.5510 [rgb]0.5,0.5,0.5 ±.0190 .4270±.0330.4270 [rgb]0.5,0.5,0.5 ±.0330 Morgan + RDKit + Avalon + ErG .3810±.0440.3810 [rgb]0.5,0.5,0.5 ±.0440 .4620±.0680.4620 [rgb]0.5,0.5,0.5 ±.0680 .3750±.0190.3750 [rgb]0.5,0.5,0.5 ±.0190 Atom-JEPA Last Layer .4120±.1510.4120 [rgb]0.5,0.5,0.5 ±.1510 .4560±.0470.4560 [rgb]0.5,0.5,0.5 ±.0470 .2500±.0390.2500 [rgb]0.5,0.5,0.5 ±.0390 Atom-JEPA All Layers .4310±.0740.4310 [rgb]0.5,0.5,0.5 ±.0740 .4980±.0280.4980 [rgb]0.5,0.5,0.5 ±.0280 .2480±.0760.2480 [rgb]0.5,0.5,0.5 ±.0760 MLP models Morgan .3440±.0360.3440 [rgb]0.5,0.5,0.5 ±.0360 .4990±.0160.4990 [rgb]0.5,0.5,0.5 ±.0160 .2910±.0520.2910 [rgb]0.5,0.5,0.5 ±.0520 RDKit .3350±.0700.3350 [rgb]0.5,0.5,0.5 ±.0700 .5890±.0330.5890 [rgb]0.5,0.5,0.5 ±.0330 .3670±.0180.3670 [rgb]0.5,0.5,0.5 ±.0180 RDKit + Morgan .2550±.1480.2550 [rgb]0.5,0.5,0.5 ±.1480 .5570±.0150.5570 [rgb]0.5,0.5,0.5 ±.0150 .3840±.0090.3840 [rgb]0.5,0.5,0.5 ±.0090 RDKit + Morgan + Avalon + ErG .2400±.1120.2400 [rgb]0.5,0.5,0.5 ±.1120 .4090±.2030.4090 [rgb]0.5,0.5,0.5 ±.2030 .2290±.2250.2290 [rgb]0.5,0.5,0.5 ±.2250 Atom-JEPA Last Layer .4560±.0480.4560 [rgb]0.5,0.5,0.5 ±.0480 .5650±.0310.5650 [rgb]0.5,0.5,0.5 ±.0310 .3460±.0250.3460 [rgb]0.5,0.5,0.5 ±.0250 Atom-JEPA All Layers .3830±.1100.3830 [rgb]0.5,0.5,0.5 ±.1100 .5450±.0270.5450 [rgb]0.5,0.5,0.5 ±.0270 .3020±.0320.3020 [rgb]0.5,0.5,0.5 ±.0320 Other models MapLight .559±.011.559 [rgb]0.5,0.5,0.5 ±.011 .625±.009.625 [rgb]0.5,0.5,0.5 ±.009 .469±.016.469 [rgb]0.5,0.5,0.5 ±.016 MapLight+GNN .563±.009.563 [rgb]0.5,0.5,0.5 ±.009 .635±.007.635 [rgb]0.5,0.5,0.5 ±.007 .500±.012.500 [rgb]0.5,0.5,0.5 ±.012 CaliciBoost – – – Atom-JEPA .4910±.0900.4910 [rgb]0.5,0.5,0.5 ±.0900 .6020±.0210.6020 [rgb]0.5,0.5,0.5 ±.0210 .4990±.0230.4990 [rgb]0.5,0.5,0.5 ±.0230 Table 24: Results on the TDC-ADMET benchmark: toxicity. Columns correspond, from left to right, to herg, ames, dili, and ld50_zhu. Numbers from Koleiev et al. (2026). Model hERG AMES DILI LD50 Unit AUROC↑ AUROC↑ AUROC↑ MAE↓ GBDT models Morgan .8100±.0180.8100 [rgb]0.5,0.5,0.5 ±.0180 .7690±.0170.7690 [rgb]0.5,0.5,0.5 ±.0170 .8640±.0170.8640 [rgb]0.5,0.5,0.5 ±.0170 .6750±.0170.6750 [rgb]0.5,0.5,0.5 ±.0170 RDKit .7910±.0220.7910 [rgb]0.5,0.5,0.5 ±.0220 .8050±.0050.8050 [rgb]0.5,0.5,0.5 ±.0050 .8920±.0230.8920 [rgb]0.5,0.5,0.5 ±.0230 .6370±.0120.6370 [rgb]0.5,0.5,0.5 ±.0120 Morgan + RDKit .8120±.0170.8120 [rgb]0.5,0.5,0.5 ±.0170 .8160±.0100.8160 [rgb]0.5,0.5,0.5 ±.0100 .8870±.0180.8870 [rgb]0.5,0.5,0.5 ±.0180 .6500±.0090.6500 [rgb]0.5,0.5,0.5 ±.0090 Morgan + RDKit + Avalon + ErG .7680±.0530.7680 [rgb]0.5,0.5,0.5 ±.0530 .8050±.0130.8050 [rgb]0.5,0.5,0.5 ±.0130 .8300±.0470.8300 [rgb]0.5,0.5,0.5 ±.0470 .6400±.0130.6400 [rgb]0.5,0.5,0.5 ±.0130 Atom-JEPA Last Layer .7600±.0210.7600 [rgb]0.5,0.5,0.5 ±.0210 .7660±.0110.7660 [rgb]0.5,0.5,0.5 ±.0110 .8830±.0480.8830 [rgb]0.5,0.5,0.5 ±.0480 .6970±.0090.6970 [rgb]0.5,0.5,0.5 ±.0090 Atom-JEPA All Layers .8080±.0160.8080 [rgb]0.5,0.5,0.5 ±.0160 .8050±.0050.8050 [rgb]0.5,0.5,0.5 ±.0050 .9130±.0390.9130 [rgb]0.5,0.5,0.5 ±.0390 .6540±.0100.6540 [rgb]0.5,0.5,0.5 ±.0100 MLP models Morgan .7770±.0150.7770 [rgb]0.5,0.5,0.5 ±.0150 .7920±.0140.7920 [rgb]0.5,0.5,0.5 ±.0140 .8520±.0180.8520 [rgb]0.5,0.5,0.5 ±.0180 .6670±.0120.6670 [rgb]0.5,0.5,0.5 ±.0120 RDKit .8360±.0040.8360 [rgb]0.5,0.5,0.5 ±.0040 .8160±.0070.8160 [rgb]0.5,0.5,0.5 ±.0070 .8480±.0310.8480 [rgb]0.5,0.5,0.5 ±.0310 .6750±.0130.6750 [rgb]0.5,0.5,0.5 ±.0130 RDKit + Morgan .8170±.0300.8170 [rgb]0.5,0.5,0.5 ±.0300 .8380±.0190.8380 [rgb]0.5,0.5,0.5 ±.0190 .8590±.0100.8590 [rgb]0.5,0.5,0.5 ±.0100 .6030±.0150.6030 [rgb]0.5,0.5,0.5 ±.0150 RDKit + Morgan + Avalon + ErG .6460±.2500.6460 [rgb]0.5,0.5,0.5 ±.2500 .8330±.0080.8330 [rgb]0.5,0.5,0.5 ±.0080 .8290±.0210.8290 [rgb]0.5,0.5,0.5 ±.0210 .5530±.0210.5530 [rgb]0.5,0.5,0.5 ±.0210 Atom-JEPA Last Layer .7930±.0080.7930 [rgb]0.5,0.5,0.5 ±.0080 .8190±.0070.8190 [rgb]0.5,0.5,0.5 ±.0070 .9270±.0090.9270 [rgb]0.5,0.5,0.5 ±.0090 .6520±.0110.6520 [rgb]0.5,0.5,0.5 ±.0110 Atom-JEPA All Layers .8250±.0250.8250 [rgb]0.5,0.5,0.5 ±.0250 .8320±.0020.8320 [rgb]0.5,0.5,0.5 ±.0020 .9070±.0020.9070 [rgb]0.5,0.5,0.5 ±.0020 .6230±.0210.6230 [rgb]0.5,0.5,0.5 ±.0210 Other models MapLight .873±.004.873 [rgb]0.5,0.5,0.5 ±.004 .865±.001.865 [rgb]0.5,0.5,0.5 ±.001 .883±.003.883 [rgb]0.5,0.5,0.5 ±.003 .618±.005.618 [rgb]0.5,0.5,0.5 ±.005 MapLight+GNN .880±.003.880 [rgb]0.5,0.5,0.5 ±.003 .872±.001.872 [rgb]0.5,0.5,0.5 ±.001 .918±.005.918 [rgb]0.5,0.5,0.5 ±.005 .563±.009.563 [rgb]0.5,0.5,0.5 ±.009 Atom-JEPA .8450±.0190.8450 [rgb]0.5,0.5,0.5 ±.0190 .8400±.0120.8400 [rgb]0.5,0.5,0.5 ±.0120 .8980±.0180.8980 [rgb]0.5,0.5,0.5 ±.0180 .5590±.0100.5590 [rgb]0.5,0.5,0.5 ±.0100 Appendix F Effect of Pretraining Objectives and Data Scale Full Results Table 25 shows the full results for the MLP probe evaluation of each pretrained encoder as described in Section 4.2. The pretraining configurations differ to some degree between QM9 and Uni-Mol to accommodate differences in dataset size and molecular composition, as detailed in Table 26. These differences include architectural, graph-partitioning, and optimization hyperparameters. Within QM9, the three objective variants share the exact same pretraining hyperparameters. Within Uni-Mol, the atom-only encoder was pretrained using multiple nodes to speed up training time, with the global batch size increased from 128 to 384 and the maximum learning rate increased from 1.4×10−41.4× 10^-4 to 2.42×10−42.42× 10^-4, approximately following square-root scaling with batch size. This adjustment accommodates the distributed training configuration. All encoders are evaluated using the same all-layer MLP probing protocol on frozen features. The results therefore support the observed performance trends across the evaluated configurations, including the stronger relative performance of the joint objective with Uni-Mol pretraining and the broad improvements from pretraining on the larger dataset. Table 25: All-layer MLP probes on frozen encoder features pretrained on the QM9 dataset and Unimol dataset respectively across combinations of the pretraining objectives. The evaluation metric is MAE (↓)( ). Bold: best result within each pretraining group; bold and underline: best overall. “atom” and “sub” denote atom-level and substructure-level objectives, respectively. Task α Δε εHOMO _HOMO εLUMO _LUMO μ CνC_ν G H R2R^2 U U0U_0 ZPVE Units ma03ma_0^3 meV meV meV mD mcalmol K mcalmol K meV meV ma02ma_0^2 meV meV meV Pretrained on QM9 (130k) atom+sub 115 130.6 86.1 88.0 56.0 40.8 22.3 22.9 396 22.6 22.1 1.8 atom-only 97 114.4 69.6 81.4 49.4 32.4 18.4 18.4 398 18.4 18.6 1.5 sub-only 104 131.6 90.0 80.2 82.3 42.7 23.0 22.7 420 22.5 22.4 1.9 Pretrained on Uni-Mol (19M) atom+sub 91 94.9 60.5 64.0 32.2 30.9 13.2 12.4 419 12.4 12.4 1.4 atom-only 96 95.7 63.3 64.9 29.2 31.4 13.4 13.0 464 13.3 13.6 1.4 Appendix G Pretraining Implementation Details G.1 Hyperparameters Table 26 contains the hyperparameters used for the pretraining on Uni-Mol and Alexandria as well as the hyperparameters used for pretraining the three encoders on QM9, which we use for the ablation studies on prediction objectives and data scale. We use EquiformerV3 (Liao et al., 2026) for the context encoder, target encoder, and predictor throughout, following the notation of Equiformer V3 for the architectural hyperparameters. The context and target encoders use identical architectures. The predictor uses the same configuration as the encoders except for the hyperparameters explicitly listed in the predictor section of the table. In particular, the predictor GNN body is more shallow with 3 transformer blocks instead of 8 and a feature dimension of 128 rather than 256. Pretraining was performed on 8 NVIDIA H100 80GB HBM3 GPUs per run. The Uni-Mol run took approximately 309 hours ( 2,475 GPU-hours) for 1,364,000 training steps, and the Alexandria run took approximately 268 hours ( 2,147 GPU-hours) for 1,321,000 training steps. Table 26: Hyperparameters used for pretraining on Uni-Mol, Alexandria, and QM9. We use EquiformerV3 for the context encoder, target encoder, and predictor. The context and target encoders use identical architectures. Predictor hyperparameters not explicitly listed are identical to those of the corresponding encoder. Hyperparameter Uni-Mol Alexandria QM9 Encoder Maximum degree LmaxL_ 2 2 6 Maximum order MmaxM_ 2 2 2 Number of Transformer blocks 8 8 8 Embedding dimension dembedd_embed (2,256)(2,256) (2,256)(2,256) (6,256)(6,256) Attention hidden dimension dattn-hiddend_attn -hidden (2,64)(2,64) (2,64)(2,64) (6,64)(6,64) Number of attention heads h 8 8 8 Attention alpha dimension dattn-αd_attn -α (0,64)(0,64) (0,64)(0,64) (0,64)(0,64) Attention value dimension dattn-valued_attn -value (2,16)(2,16) (2,16)(2,16) (6,16)(6,16) FFN hidden dimension dffnd_ffn (2,512)(2,512) (2,512)(2,512) (6,512)(6,512) Attention grid resolution (Rϕ,Rθ)(R_φ,R_θ) (8,14)(8,14) (8,14)(8,14) (8,20)(8,20) FFN grid resolution (Rϕ,Rθ)(R_φ,R_θ) (14,14)(14,14) (14,14)(14,14) (20,20)(20,20) Radial basis function Gaussian Gaussian Gaussian Number of radial bases 128 128 128 Radial hidden scalar dimension dedged_edge (0,128)(0,128) (0,128)(0,128) (0,128)(0,128) Predictor Number of Transformer blocks 3 3 3 Embedding dimension dembedd_embed (2,128)(2,128) (2,128)(2,128) (6,128)(6,128) Encoder-to-predictor projection (2,256)→(2,128)(2,256)\!→\!(2,128) (2,256)→(2,128)(2,256)\!→\!(2,128) (6,256)→(6,128)(6,256)\!→\!(6,128) Pooling readout hidden dimension 128 128 128 Pooling readout output dimension 256 256 256 Number of positional radial bases 128 128 128 Positional radial range (Å) 12 12 12 Positional embedding dimension (2,128)(2,128) (2,128)(2,128) (6,128)(6,128) Atom-head input dimension 512 512 1024 Atom-head hidden dimension 128 128 128 Atom-head output dimension 256 256 256 Graph Construction and Partitioning Radius-graph cutoff rcr_c (Å) 6 6 6 k-EgoNet cutoff rkr_k (Å) 1.8 3.5 1.8 k-EgoNet hops 1,3,5,10\1,3,5,10\ 1,2,3,5\1,2,3,5\ 1,2\1,2\ Optimization Optimizer AdamW AdamW AdamW Learning-rate schedule Cosine + linear warmup Cosine + linear warmup Cosine + linear warmup Warmup fraction .15 .15 .15 Maximum learning rate 1.4×10−41.4× 10^-4 (atom+pool) 2.42×10−42.42× 10^-4 (atom-only) 1.4×10−41.4× 10^-4 1.0×10−41.0× 10^-4 Batch size 128 (atom+pool) 384 (atom-only) 128 64 Number of epochs 10 100 200 Weight decay .04 .04 .04 Dropout rate .0 .0 .0 Stochastic depth .05 .05 .05 Gradient clipping norm 100 100 100 Target EMA decay τ .995→1.0.995→ 1.0 (linear) .995→1.0.995→ 1.0 (linear) .995→1.0.995→ 1.0 (linear) G.2 Representation Collapse Metrics To monitor representation collapse during pretraining, we track RankMe (Garrido et al., 2023) and α-reQ (Agrawal et al., 2022) computed on the pooled representations of the target encoder. Both metrics operate on an embedding matrix D∈ℝB×dembedD ^B× d_embed formed by stacking the pooled representations across the batch, including both evaluation directions of the symmetric loss (i.e., each partition acting once as GyG_y against the other as context, and vice versa), gathered across all 8 GPUs used for pretraining. With a per-GPU batch size of 16, this gives B=256B=256, matching the target encoder’s feature dimension dembed=256d_embed=256. RankMe is an estimate of the effective rank of D via the Shannon entropy of its singular value distribution, bounded by min(B,dembed)=256 (B,d_embed)=256 in our setting. A RankMe value collapsing towards 1 indicates that the target encoder is producing near-constant or low-dimensional outputs, which would mean representation collapse. α-ReQ instead fits a power-law exponent α to the decay of D’s covariance eigen spectrum, such that a large α is an indication of a small number of directions dominating the representation, and small α indicates an overly flat, noise-dominated spectrum. Figure 14 shows both metrics over the course of pretraining on the Uni-Mol and Alexandria datasets, for the joint-objective encoder. In both cases, we observe that the RankMe metric rises sharply from the initial value and then plateaus at around (∼ 80 for Uni-Mol, ∼ 130–140 for Alexandria). On Uni-Mol α-ReQ falls from its initial value and then stabilizes in a moderate range. For Alexandria it also falls from its initial value, and then slowly increases throughout training, remaining still within a moderate range. Overall, both metrics remain stable throughout training, indicating that the context and target encoders do not undergo representation collapse over the course of pretraining. Figure 15 shows the total loss for both runs during training. In both cases we observe a pattern of a sharp initial decline as the predictor and encoders leave initialization, followed by a period of gradual increase as the EMA target encoder drifts away form the context encoder, and finally a convergence phase towards the end of training as the target and context encoders are brought back into closer alignment, allowing the loss to decrease again. Figure 14: Evolution of RankMe (top) and α-reQ (bottom) during pretraining for the Uni-Mol (left) and Alexandria (right) encoders. Figure 15: Total training loss during pretraining for the Uni-Mol (left) and Alexandria (right) encoders. Appendix H Finetuning Implementation Details H.1 ADMET Finetuning Atom-JEPA Due to the label imbalances and objective differences between targets present in the multi-task datasets, we follow CKERMT (Xue et al., 2026) and employ two tricks; an individual MLP head for each of the individual tasks, and homoscedastic uncertainty task weighting of the loss (Kendall et al., 2018). We follow the learning from Tompa et al. (2026), relying on freezing the pretrained model for better generalisation, and otherwise only tune the learning rate. We deviate from their recommendations to avoid weight decay on the pre-trained model by instead using L2-SP (Xuhong et al., 2018) to avoid the model from drifting too far from the pretrained model. LGBT and MLP Baselines For baselines, we implement a gradient boosted tree (LGBT) and a 3-layer MLP model using different combination of features: Morgan fingerprints, RDKit features, ErG, and Avalon features. In addition, we also experiment with adding frozen Atom-JEPA features. CKERMT As Xue et al. (2026) does not provide test MAE of individual folds, we reran their experiments to get the uncertainty for statistical tests. We use their model with the largest self-supervised dataset called "CKERMT Biogen+OA+CM task-specific". Our results agree closely with the published mean MAEs across datasets. Mol-JEPA We also use Mol-JEPA with the same MLP and LGBT setup, either using their pooled molecule level embedding (CLS) or their full 12 modality output (modalities). Chemprop We use the Chemprop v2 version, with Biogen tuned hyperparameters from Adrian et al. (2025). We also tried default parameters but found them to perform worse. For all models except Atom-JEPA and Chemprop, we run a hyper-parameter tuning setup with Optuna using a budget of 500 iterations. H.1.1 Hyperparameter Tuning We take the training split of one of the first split in Adrian et al. (2025) on Biogen split by clusters, which we part into a training and validation split. We select the validation macro MAE as our optimisation target. over all targets using Optuna for hyperparameter tuning. For all models except Atom-JEPA and CKERMT, we use a budget of 500 iterations with the TPE optimizer and hyperband pruning. All with a warm start of 15 iterations. For each feature extract(s) + feature processor, we run the full hyperparameter pipeline. If multiple feature extractors are used in the same model, all their hyperparameters are tuned at the same time. For the LGBM models we fixed the number of estimators high at 2000, as the complexity of the model is already protected by early stopping. The ranges for the LGBM was based on the values from Fang et al. (2023) and Optuna’s LightGBMTuner, but by design left slightly larger. We use CDF-normalized RDKit features. Table 27: Intervals and priors for the hyperparameters of the specific feature processor tuned over using Optuna. Processor Hyperparameter Interval Prior MLP Learning rate (10−4,10−2)(10^-4,10^-2) Log-uniform Epochs (5,500)(5,500) Log-uniform Weight decay (10−4,10−1)(10^-4,10^-1) Log-uniform Head dropout (0,0.5)(0,0.5) Uniform Head hidden units (32,256)(32,256) Log-uniform Trunk hidden units 128,256,512,1024\128,256,512,1024\ Categorical Trunk layers 2,3,4,5\2,3,4,5\ Categorical Trunk dropout (0,0.5)(0,0.5) Uniform Trunk activations ReLU, GELU. SiLU\ReLU, GELU. SiLU\ Categorical LGBM Learning rate (5⋅10−3,3⋅10−1)(5· 10^-3,3· 10^-1) Log-uniform Number of leaves (8,128)(8,128) Log-uniform Minimum child sample (5,100)(5,100) Log-uniform Subsample fraction (0.5,1.)(0.5,1.) Uniform Colsample bytree (0.4,1)(0.4,1) Uniform Reguralizer α (10−8,101)(10^-8,10^1) Log-uniform Reguralizer λ (10−8,101)(10^-8,10^1) Log-uniform Table 28: Intervals and priors for the hyperparameters of the specific feature extractor tuned over using Optuna. Feature Hyperparameter Interval Prior Morgan Radius 2,3\2,3\ Categorical Number of bits 1024,2048\1024,2048\ Categorical Use Counts (False, True)CLOSE(False, True) Boolean RDKit – – – ErG Fuzz Increase (0.1,0.5)(0.1,0.5) Uniform Avelon Use Counts (False, True)CLOSE(False, True) Boolean Atom-JEPA Node Aggregation (Mean, Sum)CLOSE(Mean, Sum) Categorical Higher Order Features (False, True)CLOSE(False, True) Boolean Feature Scaling (None, Standard, Quantile)CLOSE(None, Standard, Quantile) Categorical Mol-JEPA Feature Scaling (None, Standard, Quantile)CLOSE(None, Standard, Quantile) Categorical We allow a higher amount of optimisation iterations for the baseline models based on LightBGM and MLP heads compared to Atom-JEPA due to the difference in computational overhead. Instead we reuse the hyperparameters found by CKERMT for the architectures. Instead, we perform tuning in two sweeps: first the L2SP weight decay of the encoder 0,10−5,10−4,10−3,10−2,10−1,10−0\0,10^-5,10^-4,10^-3,10^-2,10^-1,10^-0\ versus L2L_2 weight decay of the other parameters 0,10−5,10−4,10−3,10−2,10−1,10−0\0,10^-5,10^-4,10^-3,10^-2,10^-1,10^-0\, for a total of 49 iterations. Then the learning rate (1⋅10−5,2.5⋅10−5,5⋅10−5,1⋅10−4,2.5⋅10−4,5⋅10−4,1⋅10−3\1· 10^-5,2.5· 10^-5,5· 10^-5,1· 10^-4,2.5· 10^-4,5· 10^-4,1· 10^-3\), the number of epochs 100,50,25,10\100,50,25,10\ and number of epochs the Atom-JEPA encoder is frozen for 20,10,5,0\20,10,5,0\ for a total of 98 iterations excluding impossible combinations. We accommodate the much larger data size of ChEMBL-MT by setting the encoder freezing and encoder L2SP to 0. For the baselines and Atom-JEPA we do not use early stopping on a validation set, instead we train on the full training set as we have already selected the number of epochs.. For the literature models Chemprop, CKERMT, and Mol-JEPA we follow their respective described setups. Using a single set of hyperparameters for all datasets reduces model performance, but significantly reduces the computational burden. H.1.2 Conformer Generation Details For datasets where the 3D structure of the compound is not given, we generate conformers following the same setup as Unimol. For completeness we detail the setup here. To convert a SMILE string to a 3D conformer, we import the SMILE string in RDKit. If the importing fails with sanitation we retry without sanitation and afterwards sanitise it. To get the 3D coordinates we use ETKDGv3 to generate up to 10 conformers. We first use attempt to generate with enforced chirality, but in case of failure we start from random coordinates. Since Atom-JEPA uses the SE(3) equivariant GNN EquiformerV3, the model can in theory learn to distinguish between enantiomers. The conformers are only used during training as a form of training augmentation. For several of the ADMET-TDC datasets, the SMILE string contains multiple fragmented components, such as salt forms of CN(C)C(…)Cc1c(Cl)c(Cl)c1.[Cl-].[H+]. This is a problem during conformer relaxation, as the disconnected components can arbitrarily be moved around, thus geometrically ill-defined. Table 29: Molecules in the TDC ADMET benchmark group that RDKit parses as more than one connected component (salts, hydrates, mixtures), over the de-duplicated train_val + test SMILES of each dataset. “Extra atoms” is the mean, over fragmented molecules only, of the share of atoms outside the largest fragment. Dataset N Fragmented Fragmented (%) Extra atoms (%) Caco2 Wang 906 9 1.0 4.6 HIA Hou 578 0 0.0 0.0 Pgb Broccatelli 1212 0 0.0 0.0 Bioavailability Ma 640 1 0.2 52.5 Lipophilicity AstraZeneca 4200 1 0.0 4.9 Solubility AqSolDB 9982 1098 11.0 30.2 B Martins 1975 105 5.3 11.7 PPBR AZ 1797 0 0.0 0.0 VDss Lombardo 1111 0 0.0 0.0 CYP2D6 Veith 13130 549 4.2 16.6 CYP3A4 Veith 12328 549 4.5 16.8 CYP2C9 Veith 12092 538 4.4 17.0 CYP2D6 Substrate Carbon-Mangels 664 0 0.0 0.0 CYP3A4 Substrate Carbon-Mangels 667 0 0.0 0.0 CYP2C9 Substrate Carbon-Mangels 666 0 0.0 0.0 Half-life Obach 665 10 1.5 13.2 Clearance Microsome AZ 1102 0 0.0 0.0 Clearance Mepatocyte AZ 1020 0 0.0 0.0 hERG 648 12 1.9 20.8 AMES 7255 0 0.0 0.0 DILI 475 18 3.8 30.0 LD50 Zhu 7342 0 0.0 0.0 H.1.3 Conformer Scaling Generation of conformers can be seen as a way of generating multiple 3D views of the same molecule. Previous work (Zhou et al., 2023) generates multiple conformers, specifically 11, as a type of augmentation for training. In this section we investigate the impact on performance these parameters have. To avoid leakage of data, we use the same data for the train and validation split as the hyperparameter tuning. First, we investigate the impact of the number of conformers available during training and test time. Some of the targets have only 7 datapoints in the validation set, so instead of averaging the over the tasks, we present error over the datapoints. For this experiment we generate up to 50 conformers (from 10) which equivalates a new conformer seen each epoch. In addition to using conformers during training, we also investigate the impact of ensembling the predictions over conformers. We average over 10 conformers for each run. The results are shown in Figure 16. In the figure, we see a large drop in validation MAE when going from training on 1 to 2 conformers, while the run to run variance dominates higher conformer numbers. This shows the importance of having multiple (>1) conformer available during training. Generating many conformers is a good trade-off for many ADMET tasks as cost of generating multiple conformers is comparatively cheap compared to the training time. In addition to training conformers, the figure reveals moderate gains from averaging multiple model predictions from conformers. This can be thought of as a way of performing test-time augmentations or ensemble averaging. Curiously, the figure reveals a weak trend of increasing error when ensembling conformer predictions when training on more conformers, although not statistically significant. We hypothesize, that the variance of the model’s predictions over conformers is reduced when trained with more conformers, which could negatively impact the ensembles prediction. Figure 16: The validation error on the validation error (MAE) against the number of conformers used during training. The different curves represent different numbers of predictions from conformers averaged over during test time (test time augmentations). We also investigate the impact of the conformer generation quality. Lower potential energy of conformers are typically associated with higher quality as they are more probable. To avoid a co-founding generation artifacts from the relaxation process of MMFF94 and using it to judge the same energies, we instead use the interatomic potential model UMA-s-1p2 (Wood et al., 2026) model to predict energy. The results are shown in Figure 17. Surprisingly, the figures shows no correlation between the error of the conformer prediction versus the conformer quality. Only for conformers of very high energy compared to the mean (∼10%)( 10\%) we see some small increase in error. Figure 17: Histogram of the prediction error of validation datapoints versus the energy of the conformer. Left plot is the relative energy of the conformer from the mean, right plot is the rank of the conformer. For none of the plots a strong correlation between error and conformer energy was observed. H.1.4 ADMET Parameters for Atom-JEPA The optimal hyperparameters obtained from hyperparameter tuning can be found in Table 30. Table 30: Hyperparameters used for Atom-JEPA fine-tuning on Biogen ADME, ChEMBL-MT, ExpansionRX, and ADMET-TDC. All three start from the same Uni-Mol-pretrained context encoder; encoder architecture is unchanged from Table 26. Every task gets its own MLP head on top of a shared trunk that takes the atom-pooled representation. Hyperparameter Biogen ADME ChEMBL-MT ExpansionRX Prediction Head Number of tasks 6 25 9 Trunk Architecture Linear (768,256)(768,256) → SiLU → Dropout Per Task Head Architecture Linear (256,64)(256,64) → SiLU → Linear (64,1)(64,1) Shared Trunk dropout .1 Pooling readout Mean Higher-order invariants Yes Conformers Conformers generated per molecule 10 Conformers used in training 10 Objective Per-task loss Huber Label standardization Per-task train mean/std Log-transformed targets None None All but LogD Regularization L2-SP coefficient λ 1×10−51× 10^-5 0 1×10−51× 10^-5 Shared Trunk weight decay .5 MTL log-σ weight decay .1 Encoder stochastic depth .05.05 Encoder normalization layers Eval mode Optimization Optimizer AdamW Learning-rate schedule Cosine + linear warmup Encoder learning rate 5×10−55× 10^-5 Head learning-rate multiplier 10 MTL log-σ learning rate 5×10−55× 10^-5 Minimum learning rate 1×10−71× 10^-7 Warmup epochs 10 Encoder-only warmup epochs 10 Frozen-encoder epochs 10 0 10 Batch size 8 Number of epochs 50 Gradient clipping norm 10 Weight EMA decay .99 H.2 QM9 Finetuning Table 31 summarizes the hyperparameters used for QM9 finetuning. We use three target-specific readouts operating on the encoder’s final atom representations following SchNet/SchNetPack-style atom-wise and electronic spatial-extent constructions (Schüt et al., 2018) and a PaiNN-style gated equivariant dipole construction (Schüt et al., 2021), adapted from the GotenNet implementation (Aykent and Xia, 2025). Let C denote the encoder feature width and ir_i the position of atom i. All readouts use sum aggregation of atom-wise contributions. Dipole moment (μ). Two gated equivariant blocks combine the last layer scalar (ℓ=0 =0) and vector (ℓ=1 =1) features, with SiLU activations, to predict an atomic scalar qiq_i and dipole vector id_i. The molecular dipole magnitude is then μ^=‖∑i(i+qii)‖2. μ= \| _i (d_i+q_ir_i ) \|_2. Electronic spatial extent (R2R^2). A two-layer atomwise MLP with dimensions C→C→1C→ C→ 1 and shifted-softplus activation predicts scalar weights aia_i. The molecular prediction is R^2=∑iai‖i−CM‖22, R^2= _ia_i \|r_i-r_CM \|_2^2, where CMr_CM is the molecular center of mass. Remaining targets. A two-layer atomwise MLP with dimensions C→C→1C→ C→ 1 and SiLU activation maps scalar atom features to scalar contributions, which are summed to obtain the molecular prediction. For U, U0U_0, H, and G, summed atomic reference energies are subtracted during training and restored for evaluation, following standard practice. No targets are standardized. Table 31: Hyperparameters used for QM9 fine-tuning. All targets share the same optimization settings, with target-specific readout architectures for μ and R2R^2. Encoder architecture hyperparameters are inherited from the pretrained model. Hyperparameter All targets Optimization Optimizer AdamW Adam ϵε 10−710^-7 Maximum training epochs 1,000 Batch size 32 Encoder base learning rate 10−410^-4 Readout base learning rate 10−410^-4 Weight decay 0 Learning-rate schedule Linear warmup + ReduceLROnPlateau Global learning-rate warmup (optimizer steps) 10,000 Learning-rate reduction factor .8 Learning-rate reduction patience (epochs) 15 Minimum scheduler learning rate 10−710^-7 Initial encoder freezing (epochs) 10 Encoder ramp after unfreezing (epochs) 10 Gradient clipping norm 5 Objective and Model Selection Training loss MSE Scheduler and early-stopping metric Validation MSE Early-stopping patience (epochs) 150 Checkpoint-selection metric Validation MAE Reproducibility Training seed 42 Split seed 1 H.3 Matbench Finetuning We use a common training configuration for the five small-to-medium tasks (phonons, dielectric, Log GVRH, Log KVRH and Perovskites), and a second configuration for the three large tasks (MP Gap, MP E Form, and MP Is Metal) as summarized in Table 32. MP Is Metal is the only classification task. All tasks use an atom-wise readout consisting of a two-layer MLP with SiLU activation with one output for regression or two logits for classification. The final layer has a bias only for classification, and no dropout is used. Atom-wise outputs are mean-pooled, except for phonons, which uses max pooling. Classification probabilities are obtained by applying softmax after pooling. We standardize targets for all regression tasks and not for classification. Training uses separate encoder and readout learning rates, with linear warmup followed by cosine decay. After the initial encoder-freezing phase, the encoder learning rate is ramped over ten epochs for small-to-medium tasks and 1 epoch for large tasks. We do not use L2-SP regularization for Matbench. For each official Matbench fold, we reserve 10% of the training fold for validation using split seed 42 and set the PyTorch random seed to 0. Regression normalization statistics are computed from the training fold. We select EMA weights by the lowest validation MAE for regression or highest validation F1 for classification. Scratch baselines use the same architecture and settings, but initialize the encoder randomly and train it from the first epoch without freezing. Table 32: Hyperparameters used for Matbench fine-tuning. Small-to-medium tasks comprise phonons, dielectric, Log GVRH, Log KVRH and Perovskites; large tasks comprise MP Gap, MP E Form, and MP Is Metal. Classification settings apply only to MP Is Metal. Encoder freezing applies only to pretrained runs; encoders trained from scratch are trainable from the first epoch. Hyperparameter Small-to-medium tasks Large tasks Optimization Optimizer AdamW AdamW Adam coefficients (β1,β2)( _1, _2) (.9,.95)(.9,.95) (.9,.95)(.9,.95) Adam ϵε 10−810^-8 10−810^-8 Number of epochs 100 12 Training batch size 4 32 Evaluation batch size 32 32 Encoder base learning rate 5×10−55× 10^-5 1.5×10−41.5× 10^-4 Readout learning-rate multiplier 10 10 Readout base learning rate 5×10−45× 10^-4 1.5×10−31.5× 10^-3 Learning-rate schedule Linear warmup + cosine decay Linear warmup + cosine decay Global learning-rate warmup (epochs) 10 2 Initial warmup factor .1 .5 Encoder learning-rate floor 10−710^-7 10−710^-7 Readout learning-rate floor 10−610^-6 10−610^-6 Initial encoder freezing (epochs) 10 2 Encoder ramp after unfreezing (epochs) 10 1 Encoder / readout weight decay 0/00/0 0/00/0 Gradient clipping norm 10 10 Model EMA decay .99 .99 Objectives and Aggregation Regression training loss MAE Huber Classification training loss — Weighted cross-entropy Aggregation of atomwise outputs Mean max for phonons Mean Appendix I Tukey HSD Plots I.1 Biogen Figure 18: Tukey HSD grid plot across the endpoints of Biogen Scaffold test data over 5 seeds. The analysis was conducted with a blocked ANOVA test followed by Tukey HSD using the Statsmodels Python package. I.2 ExpansionRX Figure 19: Tukey HSD grid plot across the endpoints of ExpansionRX temporal test data over 4 seeds. The analysis was conducted with a one-way ANOVA test followed by Tukey HSD using the Statsmodels Python package. I.3 ChEMBL-MT Figure 20: Tukey HSD grid plot across the endpoints of ChEMBL-MT test data over 4 seeds. The analysis was conducted with a blocked ANOVA test followed by Tukey HSD using the Statsmodels Python package.