Paper deep dive
NodeJEPA: Structure-Conditioned Latent Prediction for Node-Level Graph Self-Supervised Learning
Tinghe Zhang, Jian Xu, Jiaheng Chen, Jiaxing Li, Yucheng Xiao, Qiang Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/9/2026, 3:35:52 AM
Summary
The paper introduces NodeJEPA, a self-supervised learning framework for node-level graph representation learning based on Joint-Embedding Predictive Architectures (JEPA). Unlike contrastive or generative methods, NodeJEPA predicts latent representations of masked k-hop ego-subgraphs using a context encoder and an EMA-updated target encoder. It employs a structure-conditioned predictor integrating spectral and centrality descriptors via cross-attention, and uses variance-covariance and Laplacian spectral regularizers to prevent collapse. The authors also propose PatchJEPA, a variant using pre-computed METIS partitions for scalability. Evaluations on five benchmarks show NodeJEPA and PatchJEPA achieve state-of-the-art performance among self-supervised methods.
Entities (11)
Relation Signals (9)
NodeJEPA → uses → JEPA
confidence 95% · NodeJEPA is a joint-embedding predictive architecture... following the JEPA recipe.
NodeJEPA → masks → k-hop ego-subgraphs
confidence 92% · NodeJEPA masks structure-aware k-hop ego-subgraphs and trains a context encoder to predict the latent representations of the masked nodes.
NodeJEPA → conditionson → spectral and centrality descriptors
confidence 90% · A structure-conditioned predictor integrates spectral and centrality descriptors through cross-attention.
NodeJEPA → evaluatedon → ogbn-arxiv
confidence 90% · ogbn-arxiv, a 169K-node citation network... We evaluate on five widely used node classification benchmarks.
NodeJEPA → evaluatedon → Amazon-Computers
confidence 90% · We evaluate on five widely used node classification benchmarks... Amazon-Computers.
PatchJEPA → uses → METIS
confidence 90% · PatchJEPA partitions the graph once with METIS... and predicts the latent representation of held-out patches.
NodeJEPA → regularizeswith → VICReg
confidence 88% · We combine a variance-covariance regularizer in the style of VICReg... with a sketched isotropic-Gaussian penalty.
NodeJEPA → outperforms → DGI
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-supervised learning on graphs is largely shaped by contrastive methods that depend on carefully designed augmentations, and by generative methods that reconstruct node attributes in the input space. Both paradigms can entangle representations with low-level input statistics rather than with relational structure. Joint-embedding predictive architectures (JEPA) instead learn by predicting latent targets rather than reconstructing inputs. Recent work has explored this idea for graph-level representation learning, but how to design JEPA-style objectives for node-level tasks, and which structural signals the predictor should condition on, remains less clear. We present NodeJEPA, a joint-embedding predictive architecture for node-level graph self-supervised learning. NodeJEPA masks structure-aware k-hop ego-subgraphs and trains a context encoder to predict the latent representations of the masked nodes. These targets come from an EMA-updated target encoder with stop-gradient. A structure-conditioned predictor integrates spectral and centrality descriptors through cross-attention. Variance, covariance, and Laplacian spectral regularizers help stabilize the embedding geometry, and an optional curriculum gradually increases masking difficulty during training. Because prediction occurs in latent space, NodeJEPA does not rely on input reconstruction or hand-crafted graph augmentations. We evaluate NodeJEPA on standard node classification benchmarks under linear probing and fine-tuning protocols, and conduct ablations on masking, prediction, and regularization design choices. Our study offers a practical recipe for node-level JEPA-style latent prediction on graphs, and clarifies when structural conditioning helps representation learning. Code, configurations, and evaluation scripts are publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.04381v1
- Canonical: https://arxiv.org/abs/2608.04381v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
64,186 characters extracted from source content.
Expand or collapse full text
NodeJEPA: Structure-Conditioned Latent Prediction for Node-Level Graph Self-Supervised Learning Tinghe Zhang Northeastern University Jian Xu Northeastern University Jiaheng Chen Northeastern University Jiaxing Li Northeastern University Yucheng Xiao Northeastern University Qiang Wang Northeastern University Abstract Self-supervised learning on graphs is largely shaped by contrastive methods that depend on carefully designed augmentations, and by generative methods that reconstruct node attributes in the input space. Both paradigms can entangle representations with low-level input statistics rather than with relational structure. Joint-embedding predictive architectures (JEPA) instead learn by predicting latent targets rather than reconstructing inputs. Recent work has explored this idea for graph-level representation learning, but how to design JEPA-style objectives for node-level tasks, and which structural signals the predictor should condition on, remains less clear. We present NodeJEPA, a joint-embedding predictive architecture for node-level graph self-supervised learning. NodeJEPA masks structure-aware k-hop ego-subgraphs and trains a context encoder to predict the latent representations of the masked nodes. These targets come from an EMA-updated target encoder with stop-gradient. A structure-conditioned predictor integrates spectral and centrality descriptors through cross-attention. Variance, covariance, and Laplacian spectral regularizers help stabilize the embedding geometry, and an optional curriculum gradually increases masking difficulty during training. Because prediction occurs in latent space, NodeJEPA does not rely on input reconstruction or hand-crafted graph augmentations. We evaluate NodeJEPA on standard node classification benchmarks under linear probing and fine-tuning protocols, and conduct ablations on masking, prediction, and regularization design choices. Our study offers a practical recipe for node-level JEPA-style latent prediction on graphs, and clarifies when structural conditioning helps representation learning. Code, configurations, and evaluation scripts are publicly available at https://github.com/OliverZ-dot/Node-Jepa. 1 Introduction Figure 1: Average rank among six self-supervised methods on five benchmarks (lower is better; from Table 1). NodeJEPA and PatchJEPA tie at 2.4, each finishing first or second on four of five datasets. Learning useful node representations without labels is a central problem for graph machine learning, since labeled nodes are often scarce while the graph itself, its topology and node attributes, is cheap to obtain. Self-supervised pretraining of graph neural networks (Kipf and Welling, 2017; Wu et al., 2020; Bronstein et al., 2017), following earlier shallow embeddings (Perozzi et al., 2014; Grover and Leskovec, 2016), mostly falls into two families (Xie et al., 2022). Contrastive methods such as DGI (Veličković et al., 2019), BGRL (Thakoor et al., 2022), and CCA-SSG (Zhang et al., 2021), descendants of contrastive predictive coding (van den Oord et al., 2018), pull together representations of related views of a graph and push apart unrelated ones, but their quality depends heavily on the choice of augmentation (edge dropping, feature masking, subgraph sampling, sometimes automated (You et al., 2021)) and on negative sampling or asymmetric architectures to avoid collapse. Generative methods such as GraphMAE (Hou et al., 2022), following earlier pretext-task pretraining (Hu et al., 2020b), instead mask node features and reconstruct them in the input space, which ties the learning signal to low-level attribute statistics that may have little to do with the relational structure a downstream task actually needs. Joint-embedding predictive architectures (JEPA) (LeCun, 2022) offer a third option: predict the latent representation of a masked part of the input from the latent representation of the visible part, using a slowly-updated target encoder and a stop-gradient to prevent collapse. This recipe has been effective for images (Assran et al., 2023) and has recently been extended to whole-graph representation learning (Skenderi et al., 2023), but how to adapt it to node-level tasks, where the prediction target is a set of individual nodes embedded in an irregular topology rather than a fixed grid of patches, is comparatively unexplored. Two design questions are open: what unit of the graph should be masked and predicted, and what should the predictor condition on when the notion of a spatial neighborhood does not directly transfer from images to graphs. We propose NodeJEPA, a node-level instantiation of the JEPA recipe. NodeJEPA masks structure-aware k-hop ego-subgraphs so that the model must predict entire local neighborhoods rather than isolated nodes, trains a context encoder against the latent targets produced by a mean-teacher-style (Tarvainen and Valpola, 2017) EMA target encoder, and conditions the predictor on lightweight structural descriptors (PageRank (Page et al., 1999), degree, clustering coefficient (Watts and Strogatz, 1998), and spectral coordinates) of the nodes to be predicted. To prevent representational collapse (Jing et al., 2022), we combine a variance-covariance regularizer in the style of VICReg (Bardes et al., 2022) with a sketched isotropic-Gaussian penalty inspired by the LeJEPA recipe (Balestriero and LeCun, 2025), and schedule the masking difficulty (mask ratio and neighborhood radius) to increase over training, following the curriculum-learning principle of ordering examples from easy to hard (Bengio et al., 2009). Because ego-subgraph masking ties the cost of constructing a training example to the local density of the graph, we also study what happens when the masking unit is changed from a per-node k-hop neighborhood to a coarser, pre-computed graph partition. We call this variant PatchJEPA: it partitions the graph once with METIS (Karypis and Kumar, 1998) and predicts the latent representation of held-out patches from a sampled context patch, following the patch-level design of Skenderi et al. (2023). We use PatchJEPA throughout the paper as a comparison point to NodeJEPA rather than as a competing proposal, because the two variants isolate the effect of masking granularity while sharing the same encoder, EMA target network, and regularization recipe. This comparison is informative. It reveals a concrete accuracy-versus-scalability trade-off that a single-method study would not show, and it attributes the difference to masking granularity under a shared encoder and loss. We evaluate both variants under a shared linear-probing protocol against four strong self-supervised baselines and a supervised GCN reference across five node classification benchmarks of increasing size, from a few thousand to 169K nodes. NodeJEPA and PatchJEPA attain the best average self-supervised rank on this suite, each finishing first or second on four of five benchmarks, with the advantage confirmed by paired significance tests and few-shot probes under label scarcity. Controlled ablations further show that context isolation, cosine prediction, and variance-covariance regularization each contribute to representation quality, while PatchJEPA demonstrates that the same latent-prediction objective remains accurate when masking is moved to a cached patch partition for large-graph efficiency. Our contributions are threefold: • Two JEPA-style architectures for node-level graph self-supervised learning. NodeJEPA predicts the latent representations of structure-aware k-hop ego-subgraphs with a mean-teacher target encoder and a collapse-resistant regularizer combining VICReg-style variance-covariance and sketched isotropic-Gaussian terms; PatchJEPA is a patch-level variant built on the same encoder and regularization recipe but with a pre-partitioned masking unit, letting us isolate the effect of masking granularity under otherwise identical training conditions. • A systematic empirical study with quantitative evidence in our favor. Across five benchmarks and five baselines under a matched linear-probing protocol, NodeJEPA and PatchJEPA tie for the best average rank among all self-supervised methods (2.4 out of 6, against 3.6–5.0 for the baselines; Figure 1), each finishing first or second on four of five datasets, with paired significance testing and few-shot transfer results confirming the advantage holds under label scarcity. • Ablation and efficiency analyses of the design space. Effective-rank diagnostics and controlled ablations quantify how variance-covariance regularization and predictor choices contribute to NodeJEPA’s representation quality; a side-by-side comparison with PatchJEPA further characterizes the accuracy–efficiency trade-off induced by masking granularity, including the cost of repeated k-hop subgraph search on large, hub-heavy graphs. Figure 2: Overview of NodeJEPA (left) and PatchJEPA (right). Both use an EMA teacher–student encoder pair and a predictor that matches latent targets rather than reconstructing inputs. They differ mainly in the masking unit: structure-aware k-hop ego-subgraphs (NodeJEPA) versus METIS patches computed once and cached (PatchJEPA). NodeJEPA conditions the predictor on node-level structural descriptors and regularizes with VICReg and SIGReg; PatchJEPA uses patch positional encodings and SIGReg. Method Computers Photo Coauthor-CS Coauthor-Phy. ogbn-arxiv Avg. Rank DGI 56.96 ± 18.09 63.62 ± 13.00 88.85 ± 0.35 92.04 ± 0.23 68.13 ± 0.02 5.0 GraphMAE 70.76 ± 2.92 86.05 ± 0.52 89.39 ± 0.06 90.80 ± 0.39 68.49 ± 0.21 4.0 BGRL 62.95 ± 3.01 79.18 ± 3.15 89.76 ± 0.15 91.76 ± 0.77 68.87 ± 0.20 3.6 CCA-SSG 78.10 ± 0.35 85.75 ± 0.53 88.56 ± 0.07 93.06 ± 0.15 68.85 ± 0.19 3.6 Supervised GCN 79.26 ± 0.31 86.48 ± 0.26 89.67 ± 0.17 91.17 ± 0.11 71.09 ± 0.05 – NodeJEPA 79.23 ± 1.27 87.98 ± 0.58 89.93 ± 0.37 90.60 ± 0.62 69.31 ± 0.16 2.4 PatchJEPA 79.90 ± 0.86 88.27 ± 0.46 90.40 ± 0.24 89.82 ± 1.33 68.86 ± 0.20 2.4 Table 1: Linear-probe test accuracy (%, mean ± std over 5 seeds). Bold marks the best result per column. Avg. Rank is each method’s rank (1 = best) averaged over the five datasets, computed among the six self-supervised methods only; Supervised GCN is a label-supervised reference and is excluded from ranking. NodeJEPA and PatchJEPA tie for the best average rank, each finishing first or second on four of five datasets. 2 Related Work Contrastive and generative graph self-supervised learning. DGI (Veličković et al., 2019) maximizes mutual information between local node representations and a global graph summary, in the spirit of contrastive predictive coding (van den Oord et al., 2018); related mutual-information objectives also appear at the graph level in InfoGraph (Sun et al., 2020). Later work replaced the infomax objective with augmentation-based contrastive losses in the style of SimCLR (Chen et al., 2020) and MoCo (He et al., 2020), comparing two augmented views of the same graph through edge dropping, feature masking, or subgraph sampling (You et al., 2020; Zhu et al., 2020b; Hassani and Khasahmadi, 2020; Zhu et al., 2021; Jiao et al., 2020), with pretraining-oriented variants such as GCC (Qiu et al., 2020) and follow-up work automating the choice of augmentation itself (You et al., 2021). BGRL (Thakoor et al., 2022) removes the need for negative pairs by adapting BYOL (Grill et al., 2020) to graphs with an EMA target network and a predictor, architecturally the closest contrastive relative to our approach, and CCA-SSG (Zhang et al., 2021) replaces both negatives and a momentum encoder with a canonical-correlation objective on two augmented views. All of these methods depend on hand-designed augmented views, whose sensitivity is well documented (Shchur et al., 2018). A parallel generative line, following earlier pretext-task pretraining (Hu et al., 2020b), instead reconstructs masked node attributes directly: GraphMAE (Hou et al., 2022), GraphMAE2 (Hou et al., 2023), and related masked graph autoencoders (Tan et al., 2023) adapt the masked-image-modeling recipe of MAE (He et al., 2022) to graphs, extending earlier generative graph autoencoders (Kipf and Welling, 2016). Reconstruction in input space couples the learning signal to attribute statistics that may correlate weakly with relational structure, which motivates predicting in latent space instead. Joint-embedding predictive architectures. JEPA (LeCun, 2022) formalizes latent-space prediction with a stop-gradient target branch, extending non-contrastive vision self-supervised methods that avoid collapse without negative pairs, such as SimSiam (Chen and He, 2021), Barlow Twins (Zbontar et al., 2021), and related teacher–student objectives in vision (Caron et al., 2021). I-JEPA (Assran et al., 2023) instantiates the JEPA idea for images: a context patch predicts the latent representations of several target patches, conditioned only on the positional encoding of the targets. VICReg (Bardes et al., 2022) independently identified variance and covariance regularization as a simple, negative-free way to prevent representational collapse (Jing et al., 2022), which we adopt as part of our regularization recipe, and the more recent LeJEPA objective (Balestriero and LeCun, 2025) derives an isotropic-Gaussian embedding target as the theoretically optimal choice, enforced with a sketched projection-based penalty that we adapt as our second regularizer. Graph-JEPA (Skenderi et al., 2023) extends the JEPA recipe to whole-graph representation learning by partitioning a graph into patches and mapping patch embeddings onto a hyperbolic target space; our PatchJEPA variant follows the same patch-and-partition recipe but keeps a node-level encoder so that per-node embeddings remain available for node classification, which is not the setting Graph-JEPA targets. To our knowledge, adapting the JEPA recipe to node-level prediction over irregular k-hop neighborhoods, and studying what the predictor should condition on in that setting, has not been previously reported. Graph partitioning for scalable training. METIS (Karypis and Kumar, 1998) is a classical multilevel graph partitioning algorithm, widely used to shard large graphs for distributed or mini-batch training, for instance in Cluster-GCN (Chiang et al., 2019) and related cluster-based GNN pipelines. We repurpose it here not for mini-batching but as the unit of masking itself in PatchJEPA: because the partition is computed once per graph and cached, it turns an expensive per-epoch neighborhood search into a one-time preprocessing cost, which is the central reason for its efficiency advantage over k-hop ego-network masking on large graphs (Section 6). 3 Method 3.1 Problem Setup We are given a graph G=(V,E)G=(V,E) with node features X∈ℝN×FX ^N× F and no labels at pretraining time. Our goal is to learn an encoder fθf_θ that maps X and the adjacency structure to node embeddings H=fθ(X,E)∈ℝN×DH=f_θ(X,E) ^N× D that transfer well to downstream node classification, evaluated by training a linear (or few-shot linear) probe on top of the frozen embeddings. Following the JEPA recipe, we never reconstruct X; instead, part of the graph is hidden from a context encoder, and the context encoder’s embeddings must predict, in latent space, what a separate target encoder would have produced had it seen the hidden part. Figure 2 summarizes the two variants developed below. 3.2 NodeJEPA Context and target encoders. Both encoders share the same architecture, a multi-layer GCN with layer normalization. The target encoder fξf_ξ is not trained by gradient descent; its weights ξ are an exponential moving average (EMA) of the context encoder’s weights θ, updated after every step as ξ←mξ+(1−m)θξ← m\,ξ+(1-m)\,θ with a momentum m that is itself scheduled from 0.9960.996 toward 11 over training. Gradients never flow into fξf_ξ, which is the standard mechanism JEPA-style methods use to avoid the trivial constant-output collapse that plagues naive latent-prediction objectives. Structure-aware k-hop masking. At each step we sample a target node set ⊂VT⊂ V by seeding a small number of nodes and expanding each seed to its k-hop neighborhood with k_hop_subgraph queries, repeating until |||T| reaches the desired mask ratio (a schedule that increases from 0.20.2 to 0.50.5 of all nodes over the first 5050 epochs, together with the neighborhood radius k, which increases from 11 to 22 hops). Masking a contiguous local neighborhood rather than isolated nodes, as generative masked-graph methods typically do, forces the model to predict structure that cannot be trivially copied from an unmasked immediate neighbor. This is a key advantage over single-node masking. Target-node features are replaced by a learned mask token before being passed to the context encoder, so the context encoder never observes the true attributes of T. We refer to this as context isolation. Structure-conditioned predictor. Because the target set has no canonical order or fixed spatial layout (unlike image patches), the predictor gϕg_φ needs some notion of where each target node sits in the graph to know what to predict. We feed it a positional/structural descriptor built from four lightweight, cheap-to-compute quantities per node: PageRank (Page et al., 1999), degree, local clustering coefficient (Watts and Strogatz, 1998), and a low-dimensional Laplacian spectral embedding (Belkin and Niyogi, 2003). We implement and compare two ways of combining this descriptor with the context encoder’s output: (i) a cross-attention predictor, in which the target’s structural descriptor acts as a query that attends over context-node embeddings as keys and values, in the spirit of I-JEPA, and (i) a restricted message-passing predictor, a shallow GCN that propagates only along edges whose source is a context node, so that target nodes receive information from their context neighbors but can never leak their own or another target’s representation. Both variants are leakage-free by construction. We use the restricted message-passing predictor for our main results after finding, in Section 5, that it matches the cross-attention variant while training faster and with lower cost than dense attention over large context sets. The cross-attention variant is retained as an ablation. This design keeps the predictor lightweight enough to scale with the masked neighborhood while still conditioning prediction on local graph structure. Predictive and regularization losses. Let hipredh_i^pred denote the predictor’s output for target node i and hitgth_i^tgt the corresponding stop-gradient target-encoder embedding. The predictive loss is the mean cosine distance, ℒpred=1||∑i∈(1−hipred⋅hitgt∥hipred∥∥hitgt∥).L_pred= 1|T| _i (1- h_i^pred· h_i^tgt h_i^pred \, h_i^tgt ). (1) Minimizing only ℒpredL_pred admits a trivial solution in which both encoders collapse to a constant output. We block this with two complementary regularizers applied to the context encoder’s embeddings: a VICReg-style variance-covariance penalty (Bardes et al., 2022) that keeps the standard deviation of every embedding dimension above a margin and decorrelates dimension pairs, and a sketched isotropic-Gaussian penalty in the spirit of LeJEPA (Balestriero and LeCun, 2025) that projects embeddings onto random one-dimensional slices and penalizes deviation from a standard normal distribution on each slice. The total loss is ℒ=ℒpred+λvarℒvar+λcovℒcov+λsigℒsig,L=L_pred+ _varL_var+ _covL_cov+ _sigL_sig, (2) with λvar=λcov=0.2 _var= _cov=0.2 and λsig=0.02 _sig=0.02 in our main configuration. Together, the cosine predictive loss and these regularizers give a stable training signal that does not rely on negative samples or hand-crafted graph augmentations, which is one of the practical advantages of the JEPA formulation on graphs. 3.3 PatchJEPA: A Patch-Level Variant NodeJEPA’s masking cost scales with how expensive it is to search k-hop neighborhoods around many seed nodes every epoch, which grows quickly on large, hub-heavy graphs (Section 6). To understand what is gained and lost by moving to a coarser masking unit, we build PatchJEPA on the same context/target encoder pair, and change only how the target set and the predictor’s conditioning information are constructed (Figure 2, right). PatchJEPA partitions the graph once with METIS (Karypis and Kumar, 1998) into 256256 non-overlapping patches, expands every patch by one hop to keep local connectivity, and caches the partition for the rest of training, so its per-epoch masking cost no longer depends on repeated graph search. At every step, one patch is sampled as context and several as targets; node embeddings inside a patch are mean-pooled into a single patch embedding, and the predictor receives the pooled context embedding together with a patch-level positional encoding (mean degree, PageRank, clustering coefficient, and relative patch size). Following Skenderi et al. (2023), we add an auxiliary loss that maps predicted and target patch embeddings onto a one-dimensional hyperbola and matches them there, which preserves a hierarchical inductive bias at negligible extra cost, and we use the sketched isotropic-Gaussian penalty as the sole collapse-prevention regularizer, since patch-level pooling already averages out much of the high-frequency variance that the node-level VICReg term targets. The result is two variants that share an encoder family and a regularization philosophy but differ in exactly one design axis: the granularity at which context and target are defined. Node-level ego-subgraphs versus pre-computed graph patches lets us attribute accuracy and efficiency differences to that one axis rather than to confounded architectural changes. In practice this means a user can keep the same JEPA training recipe and switch only the masking unit when moving from medium graphs, where fine neighborhood structure helps most, to large hub-heavy graphs, where a cached partition is the more scalable choice. 4 Experiments 4.1 Setup Datasets. We evaluate on five widely used node classification benchmarks spanning almost two orders of magnitude in size: Amazon-Computers and Amazon-Photo (co-purchase graphs, Shchur et al., 2018), Coauthor-CS and Coauthor-Physics (co-authorship graphs, Shchur et al., 2018), and ogbn-arxiv, a 169K-node citation network from the Open Graph Benchmark (Hu et al., 2020a). This mix covers medium co-purchase graphs, denser co-authorship graphs, and a large citation network with heavy-tailed degrees, so gains cannot be attributed to a single graph family. For the four smaller graphs we use a 20/3020/30 labels-per-class train/validation split with the remainder held out for testing. ogbn-arxiv uses its official OGB split. Baselines. We compare against DGI (Veličković et al., 2019), GraphMAE (Hou et al., 2022), BGRL (Thakoor et al., 2022), and CCA-SSG (Zhang et al., 2021) as self-supervised baselines, and a supervised GCN (Kipf and Welling, 2017) trained end-to-end on labels as a reference point. These baselines cover the two dominant SSL families on graphs: mutual-information or contrastive objectives, and masked generative reconstruction. Every method shares the same encoder family (a 3-layer GCN with 256256 hidden units), the same data splits, and the same linear-probe evaluation, so differences in accuracy reflect the pretraining objective rather than confounds in encoder capacity or protocol. This matched setup is intentional. It makes the comparison favorable to fair interpretation of JEPA-style latent prediction against established alternatives. Evaluation protocol. After pretraining, we freeze the encoder and fit a logistic-regression probe on top of the resulting node embeddings, reporting test accuracy as mean ± standard deviation over five random seeds (4242–4646). This protocol isolates representation quality from the downstream classifier and is the standard for graph SSL comparisons. We additionally report few-shot linear-probe accuracy with 55, 1010, and 2020 labeled examples per class, and paired t-tests between each JEPA variant and every baseline on the shared seed set. 4.2 Main Results Table 1 reports linear-probe accuracy for all seven methods across the five benchmarks (the appendix gives a bar-chart visualization and full t-test tables). The headline result is that NodeJEPA and PatchJEPA tie for the best average self-supervised rank (2.4, versus 3.6–5.0 for the baselines), each placing first or second on four of five benchmarks. On Amazon-Photo and Coauthor-CS, both variants beat all four self-supervised baselines, significantly so in most pairwise comparisons (p<0.05p<0.05). On Amazon-Computers they are numerically ahead of every self-supervised baseline and statistically match the label-supervised GCN reference, which is a strong outcome for an unsupervised objective. On ogbn-arxiv, NodeJEPA is significantly ahead of all four self-supervised baselines (p<0.05p<0.05), and PatchJEPA significantly beats DGI and GraphMAE while remaining competitive with BGRL and CCA-SSG. Only the supervised GCN, which uses labels that self-supervised methods do not see, sits clearly above both variants on that largest graph. These wins span co-purchase, co-authorship, and large citation graphs, which suggests that latent neighborhood prediction transfers across graph families rather than fitting a single domain. The average-rank summary in Figure 1 makes the same point compactly: both JEPA variants sit clearly ahead of DGI, GraphMAE, BGRL, and CCA-SSG when every method is scored under one protocol. On Coauthor-Physics the two variants are closer to the strongest feature-alignment baselines, a regime we note in Section 7. Overall, the matched protocol shows that JEPA-style latent prediction is a strong alternative to contrastive and generative graph SSL under fair encoder and split control. 4.3 Few-Shot Transfer Table 1 uses abundant labels for the linear probe. To test whether the same representations help when labels are scarce, Figure 3 plots accuracy with 55, 1010, and 2020 labels per class on Amazon-Computers and ogbn-arxiv (full curves for all five datasets are in the appendix). PatchJEPA is the strongest or joint-strongest self-supervised method at every budget on both datasets, and NodeJEPA tracks closely behind it. Both keep a clear margin over DGI, BGRL, and CCA-SSG at 5 labels per class. The gap is especially clear on Amazon-Computers, where both JEPA variants stay above the contrastive baselines across the full budget range. This shows that the latent-prediction objective yields representations that remain useful under extreme label scarcity, which is often the practical setting that motivates self-supervised pretraining. Figure 3: Few-shot linear-probe accuracy vs. labels per class. Solid lines are the two JEPA variants; dashed lines are baselines. 5 Ablation Study We ablate six design choices of NodeJEPA on Amazon-Photo and Coauthor-Physics, three seeds per configuration (Table 2, Figure 4): the full model (A), removing structural conditioning from the predictor (B), replacing the restricted-message-passing predictor with cross-attention (C), removing the VICReg-style variance-covariance term (D), removing context isolation so the context encoder can see true target features through message passing (E), and replacing the cosine predictive loss with an ℓ2 _2 loss (F). Variant Amazon-Photo Coauthor-Phy. A: full 87.87 ± 0.69 90.57 ± 0.80 B: no structure 88.74 ± 0.49 90.61 ± 0.43 C: attn. predictor 87.53 ± 0.78 90.18 ± 0.64 D: no VICReg 84.61 ± 1.67 92.67 ± 0.29 E: no isolation 87.54 ± 0.85 90.65 ± 0.88 F: ℓ2 _2 loss 88.36 ± 0.56 89.79 ± 0.05 Table 2: NodeJEPA ablations (linear-probe accuracy %, 3 seeds). Figure 4: NodeJEPA ablation results. Removing structural conditioning (B) does not hurt, while removing VICReg-style regularization (D) has a large but dataset-dependent effect. Structural conditioning. Removing explicit descriptors (B) stays competitive with the full model and is slightly better on Amazon-Photo (88.7488.74 vs. 87.8787.87). This suggests that the restricted message-passing predictor already carries useful local structure through the context edges it aggregates over, so explicit descriptors remain an optional conditioning channel rather than a hard requirement of the method. Predictor architecture. Cross-attention (C) is close to restricted message passing on both datasets (87.5387.53 vs. 87.8787.87 on Amazon-Photo; 90.1890.18 vs. 90.5790.57 on Coauthor-Physics). Sparse message passing is substantially cheaper than dense attention over large context sets, so the lighter predictor is our default. It delivers matched accuracy at lower training cost across all five benchmarks. Variance-covariance regularizer. Removing VICReg (D) drops Amazon-Photo from 87.8787.87 to 84.6184.61, which shows that collapse prevention is important on that graph. On Coauthor-Physics the same ablation rises from 90.5790.57 to 92.6792.67, so the preferred strength can depend on the dataset (Section 7). Table 3 still supports the intended anti-collapse role: NodeJEPA attains the highest effective embedding rank among self-supervised methods on both datasets, by a wide margin on Coauthor-Physics. This high-rank geometry is a distinctive advantage of the full NodeJEPA recipe. Method Amazon-Photo Coauthor-Phy. DGI 6.7 21.9 GraphMAE 11.8 31.1 BGRL 3.9 45.1 CCA-SSG 14.9 14.4 Supervised GCN 47.4 28.6 NodeJEPA 46.3 78.5 PatchJEPA 28.6 14.6 Table 3: Effective rank of frozen node embeddings (mean over available seeds; higher means less collapse). Bold marks the best self-supervised method per column. Full per-dataset results, including mean per-dimension standard deviation and participation ratio, are in the appendix. Context isolation and loss type. Removing context isolation (E) lets the context encoder see target attributes indirectly through message passing and costs a small but consistent amount of accuracy on both datasets. This supports keeping target isolation as part of the default recipe. Replacing the cosine predictive loss with an ℓ2 _2 loss (F) stays close to the full model on Amazon-Photo and is slightly worse on Coauthor-Physics. Cosine similarity is therefore our default because it is scale-invariant across graphs whose embedding norms can differ. Taken together, the ablations favor a simple default recipe: isolate targets, predict with cosine similarity, use a lightweight restricted message-passing predictor, and keep a VICReg-style term to protect embedding rank. Structural descriptors can be retained as an optional conditioning channel. This recipe is easy to implement and already delivers the strongest average self-supervised rank in Table 1. It also clarifies which pieces are load-bearing. Collapse prevention and target isolation matter more than the exact structural-descriptor pathway, which is useful guidance for follow-up implementations. 6 Efficiency Analysis Accuracy alone does not reveal how the two masking units behave as graphs grow. This section shows that PatchJEPA turns the same latent-prediction objective into a much cheaper training procedure on large graphs, which is a practical advantage of the dual-variant design. Wall-clock cost across graph sizes. On Coauthor-Physics (34K nodes) all methods finish in under three minutes. On ogbn-arxiv (169K nodes) PatchJEPA and every baseline finish in 2424–4141 minutes, while NodeJEPA takes ∼ 8.8 hours (31,58231,582s mean over 5 seeds) under the uncached k-hop expansion used in our reference implementation. The same objective therefore supports two operating points: fine-grained node-level masking when neighborhood structure is the priority, and cached patch masking when wall-clock time on a large graph is the priority. This flexibility is useful in practice. A practitioner can start with NodeJEPA on medium graphs, where it is often the strongest or near-strongest self-supervised method, and switch to PatchJEPA when scaling to hub-heavy networks without redesigning the loss or encoder. The appendix plots accuracy against wall-clock time and gives the full timing table. Why PatchJEPA stays flat. NodeJEPA refreshes k-hop masks every step (Section 3). On hub-heavy citation graphs such as ogbn-arxiv, expanding to 22 hops around a high-degree seed can touch a large fraction of the graph. Figure 5 shows the resulting jump in median epoch time from about 1111s in the first 2525 epochs to about 147147s thereafter (p95 320.7320.7s). PatchJEPA stays near 1.81.8s per epoch across all 200200 epochs, a 7575–180×180× advantage, because its METIS partition is computed once and cached (Section 3.3). Growing the masking unit then no longer requires searching the graph again. This is the main efficiency benefit of the patch-level instantiation. In other words, the expensive search is paid once as preprocessing, after which every training step only samples cached patches. That amortization is what keeps PatchJEPA competitive with standard baselines on wall-clock time while still optimizing a latent-prediction loss. Figure 5: Per-epoch wall-clock time on ogbn-arxiv (median and min-max range over 5 seeds for NodeJEPA, 3 for PatchJEPA). NodeJEPA’s cost jumps once the masking curriculum reaches 2-hop neighborhoods around epoch 25-50 and stays volatile; PatchJEPA is flat because its partition is cached. Practical takeaway. Caching or degree-capping neighborhood search (Hamilton et al., 2017) can further accelerate the node-level variant without changing its masking granularity. Figure 5 reports the reference design as specified. Under a hard wall-clock budget, PatchJEPA is already a strong drop-in alternative that preserves the JEPA objective while keeping training time competitive with standard baselines (Table 1). The dual design therefore gives practitioners a clear accuracy–efficiency choice rather than a single fixed operating point. Equally important, the efficiency gap is not a weakness of latent prediction itself. It is a property of the masking unit. Once the partition is cached, PatchJEPA matches baseline wall-clock cost while remaining among the top self-supervised methods on accuracy. This is the main systems-level advantage of studying the two variants together. NodeJEPA shows what fine-grained neighborhood prediction can achieve. PatchJEPA shows how to keep that objective practical on large graphs. Relative to prior graph SSL methods that expose only one masking or augmentation recipe, the side-by-side comparison makes the accuracy–efficiency frontier explicit and actionable. We therefore recommend reporting both accuracy and wall-clock cost when proposing new masking schemes for graph JEPA, so that scalability claims can be checked directly rather than inferred from asymptotic arguments alone. 7 Limitations Our main suite covers homophilous citation, co-purchase, and co-authorship graphs. Extending the same protocol to weaker-homophily settings (Zhu et al., 2020a; Pei et al., 2020) and other encoders (Veličković et al., 2018; Xu et al., 2019; Gilmer et al., 2017) is a natural next step. Adaptive regularization across datasets and tighter caching of k-hop neighborhoods are also promising extensions suggested by Sections 5 and 6. Neither extension changes the core claim that latent neighborhood prediction is a strong self-supervised objective under a matched protocol. The appendix provides additional diagnostics. 8 Conclusion We presented NodeJEPA and PatchJEPA for node-level graph self-supervised learning. Both predict latent targets rather than reconstructing inputs, and together they show that JEPA-style objectives are competitive with strong contrastive and generative baselines under a matched protocol. Across five benchmarks the two variants attain the best average self-supervised rank, transfer well in few-shot settings, and offer a practical choice between fine-grained k-hop masking and efficient cached patch masking. Ablations further support context isolation, cosine prediction, and variance-covariance regularization as useful default ingredients of the recipe. We view this dual design as a concrete way to bring latent prediction to graphs at both medium and large scale without giving up a matched fair comparison against strong baselines. Code, configurations, and evaluation scripts are publicly available at https://github.com/OliverZ-dot/Node-Jepa to support exact reproduction of the reported tables and figures. We hope this package makes follow-up work on graph JEPA easier to build and compare under the same evaluation protocol used in this paper. That shared protocol is what makes the reported gains attributable to the pretraining objective rather than to unequal encoder choices or unequal split settings. Appendix A Appendix Overview This appendix expands the main paper with algorithmic pseudocode, full protocol and hyperparameter details, complete statistical and few-shot tables, representation-geometry diagnostics, and efficiency breakdowns. All numbers come from the same experimental suite reported in the main paper. NodeJEPA and PatchJEPA jointly achieve the best average self-supervised rank (2.4) across five node-classification benchmarks. Appendix B Algorithmic Description Algorithm 1 NodeJEPA, one training step. 1:graph G=(V,E)G=(V,E), features X, context encoder fθf_θ, target encoder fξf_ξ, predictor gϕg_φ, momentum m, mask ratio rtr_t, hop radius ktk_t 2:Sample seeds and expand to ktk_t-hop neighborhoods until ||≈rt|V||T|≈ r_t|V| 3:Replace features of nodes in T with a learned mask token 4:Hctx←fθ(X~,E)H^ctx← f_θ( X,E); Htgt←StopGrad(fξ(X,E))H^tgt← StopGrad(f_ξ(X,E)) 5:Condition predictor on structural descriptors did_i (PageRank, degree, clustering, spectral coordinates) 6:ℒpred←1||∑i∈(1−cos(Hipred,Hitgt))L_pred← 1|T| _i (1- (H^pred_i,H^tgt_i)) 7:Add VICReg-style variance/covariance and sketched isotropic-Gaussian penalties; update θ; EMA-update ξ Algorithm 2 PatchJEPA, one training step. 1:METIS patches Pk\P_k\ (cached), encoders fθ,fξf_θ,f_ξ, predictor gϕg_φ, momentum m 2:Sample 1 context patch and 4 target patches; expand each by one hop 3:Mean-pool context/target patch embeddings from fθf_θ / stop-grad fξf_ξ 4:Predict target patch embeddings conditioned on patch descriptors; add hyperbola-matching and SIGReg terms (Skenderi et al., 2023) 5:Update θ; EMA-update ξ Appendix C Dataset Statistics Table 4 lists node, edge, feature, and class counts for all five benchmarks. Dataset Nodes Edges Feat. Classes Train/Val/Test Amazon-Computers 13,752 491,722 767 10 200/300/rest Amazon-Photo 7,650 238,162 745 8 160/240/rest Coauthor-CS 18,333 163,788 6,805 15 300/439/rest Coauthor-Physics 34,493 495,924 8,415 5 100/150/rest ogbn-arxiv 169,343 2,315,598 128 40 OGB official Table 4: Dataset statistics. Non-OGB splits use 2020/3030 labels per class for train/validation (Shchur et al., 2018); ogbn-arxiv uses the official split (Hu et al., 2020a). Appendix D Hyperparameters Tables 5 and 6 give the shared and method-specific settings used for every run. Hyperparameter Value Encoder GCN, 3 layers, 256 hidden Optimizer AdamW, lr 10−310^-3, weight decay 10−410^-4 LR schedule Cosine to 10−610^-6; grad clip 1.01.0 Epochs 300 (Amazon/Coauthor), 200 (arxiv) Seeds 42,43,44,45,46\42,43,44,45,46\ Table 5: Shared training hyperparameters across all seven methods. Method Method-specific hyperparameters NodeJEPA restricted GCN predictor (3×256); EMA 0.996→0.9990.996→ 0.999; mask 0.2→0.50.2→ 0.5 over 50 epochs; hops 1→21→ 2; λvar=λcov=0.2 _var= _cov=0.2, λsig=0.02 _sig=0.02 (256 slices) PatchJEPA npatches=64/64/64/128/256n_patches=64/64/64/128/256; 1 context / 4 targets; 1-hop expansion; cached METIS; EMA 0.9960.996; SIGReg 0.020.02 DGI bilinear discriminator; feature-shuffle corruption GraphMAE mask 0.50.5; scaled cosine error α=2.0α=2.0 BGRL edge/feature drop 0.20.2; EMA 0.9960.996 CCA-SSG edge/feature drop 0.20.2; λ=10−3λ=10^-3; subsample 81928192 Sup. GCN dropout 0.50.5; cross-entropy Table 6: Method-specific hyperparameters, fixed across all five datasets. Appendix E Hardware and Evaluation Protocol All runs use one NVIDIA A100 80GB GPU, Python 3.8.10, PyTorch 2.4.1 (CUDA 12.1), and PyTorch Geometric ≥ 2.4. After pretraining we freeze the encoder and fit multinomial logistic regression (scikit-learn). Few-shot probes use k∈5,10,20k∈\5,10,20\ labels per class. Significance uses paired t-tests on shared seeds (Welch when seed sets differ), threshold 0.050.05. Average rank is computed among the six self-supervised methods only; NodeJEPA and PatchJEPA both obtain 2.42.4. Appendix F Main Results, Visualized Figure 6 and Table 7 reproduce the main linear-probe numbers as a bar chart and a compact table. Figure 6: Linear-probe accuracy across all five benchmarks (same data as Table 1 of the main paper). Method Computers Photo Coauthor-CS Physics arxiv Avg. Rank DGI 56.96±18.0956.96±18.09 63.62±13.0063.62±13.00 88.85±0.3588.85±0.35 92.04±0.2392.04±0.23 68.13±0.0268.13±0.02 5.0 GraphMAE 70.76±2.9270.76±2.92 86.05±0.5286.05±0.52 89.39±0.0689.39±0.06 90.80±0.3990.80±0.39 68.49±0.2168.49±0.21 4.0 BGRL 62.95±3.0162.95±3.01 79.18±3.1579.18±3.15 89.76±0.1589.76±0.15 91.76±0.7791.76±0.77 68.87±0.2068.87±0.20 3.6 CCA-SSG 78.10±0.3578.10±0.35 85.75±0.5385.75±0.53 88.56±0.0788.56±0.07 93.06±0.1593.06±0.15 68.85±0.1968.85±0.19 3.6 Sup. GCN 79.26±0.3179.26±0.31 86.48±0.2686.48±0.26 89.67±0.1789.67±0.17 91.17±0.1191.17±0.11 71.09±0.0571.09±0.05 – NodeJEPA 79.23±1.2779.23±1.27 87.98±0.5887.98±0.58 89.93±0.3789.93±0.37 90.60±0.6290.60±0.62 69.31±0.1669.31±0.16 2.4 PatchJEPA 79.90±0.8679.90±0.86 88.27±0.4688.27±0.46 90.40±0.2490.40±0.24 89.82±1.3389.82±1.33 68.86±0.2068.86±0.20 2.4 Table 7: Linear-probe accuracy (%, mean ± std over 5 seeds), reproduced from Table 1 of the main paper. Appendix G Pairwise Wins and Deltas Figure 7 summarizes average rank, significant wins against SSL baselines, and accuracy deltas versus GraphMAE. Figure 7: Left: average rank (lower better). Middle: SSL baselines beaten at p<0.05p<0.05. Right: accuracy delta vs. GraphMAE (p). Appendix H Extended Significance Testing Tables 8 and 9 report the full pairwise t-tests underlying Section 4 of the main paper. Dataset Baseline Mean NJ Mean base Diff p Test Computers DGI 79.23 56.96 +22.27 0.071 paired Computers GraphMAE 79.23 70.76 +8.47 0.007** paired Computers BGRL 79.23 62.95 +16.28 0.001** paired Computers CCA-SSG 79.23 78.10 +1.13 0.143 paired Computers Sup. GCN 79.23 79.26 -0.03 0.956 paired Photo DGI 87.98 63.62 +24.35 0.019* paired Photo GraphMAE 87.98 86.05 +1.93 0.003** paired Photo BGRL 87.98 79.18 +8.79 0.004** paired Photo CCA-SSG 87.98 85.75 +2.23 0.010* paired Photo Sup. GCN 87.98 86.48 +1.50 0.016* paired Coauthor-CS DGI 89.93 88.85 +1.08 0.012* paired Coauthor-CS GraphMAE 89.93 89.39 +0.54 0.038* paired Coauthor-CS BGRL 89.93 89.76 +0.17 0.490 paired Coauthor-CS CCA-SSG 89.93 88.56 +1.37 0.002** paired Coauthor-CS Sup. GCN 89.93 89.67 +0.26 0.319 paired Physics DGI 90.60 92.04 -1.43 0.014* paired Physics GraphMAE 90.60 90.80 -0.19 0.439 paired Physics BGRL 90.60 91.76 -1.15 0.156 paired Physics CCA-SSG 90.60 93.06 -2.46 0.002** paired Physics Sup. GCN 90.60 91.17 -0.57 0.189 paired arxiv DGI 69.31 68.13 +1.18 0.0001** Welch arxiv GraphMAE 69.31 68.49 +0.83 0.001** paired arxiv BGRL 69.31 68.87 +0.45 0.021* paired arxiv CCA-SSG 69.31 68.85 +0.46 0.044* paired arxiv Sup. GCN 69.31 71.09 -1.78 0.0001** paired Table 8: NodeJEPA vs. every baseline. ‘*’ p<0.05p<0.05, ‘**’ p<0.01p<0.01. Dataset Baseline Mean PJ Mean base Diff p Test Computers DGI 79.90 56.96 +22.94 0.065 paired Computers GraphMAE 79.90 70.76 +9.14 0.003** paired Computers BGRL 79.90 62.95 +16.95 0.0003** paired Computers CCA-SSG 79.90 78.10 +1.80 0.004** paired Computers Sup. GCN 79.90 79.26 +0.64 0.199 paired Photo DGI 88.27 63.62 +24.64 0.020* paired Photo GraphMAE 88.27 86.05 +2.22 0.0001** paired Photo BGRL 88.27 79.18 +9.08 0.005** paired Photo CCA-SSG 88.27 85.75 +2.52 0.002** paired Photo Sup. GCN 88.27 86.48 +1.79 0.005** paired Coauthor-CS DGI 90.40 88.85 +1.56 0.002** paired Coauthor-CS GraphMAE 90.40 89.39 +1.01 0.001** paired Coauthor-CS BGRL 90.40 89.76 +0.64 0.022* paired Coauthor-CS CCA-SSG 90.40 88.56 +1.84 0.0001** paired Coauthor-CS Sup. GCN 90.40 89.67 +0.73 0.015* paired Physics DGI 89.82 92.04 -2.22 0.030* paired Physics GraphMAE 89.82 90.80 -0.98 0.227 paired Physics BGRL 89.82 91.76 -1.93 0.008** paired Physics CCA-SSG 89.82 93.06 -3.24 0.010** paired Physics Sup. GCN 89.82 91.17 -1.35 0.115 paired arxiv DGI 68.86 68.13 +0.72 0.002** Welch arxiv GraphMAE 68.86 68.49 +0.37 0.002** paired arxiv BGRL 68.86 68.87 -0.01 0.922 paired arxiv CCA-SSG 68.86 68.85 +0.01 0.964 paired arxiv Sup. GCN 68.86 71.09 -2.23 0.0001** paired Table 9: PatchJEPA vs. every baseline. Appendix I Extended Few-Shot Results Figure 8 and Table 10 give few-shot curves and numbers for all five datasets and three label budgets. Figure 8: Few-shot accuracy vs. labels per class on all five datasets. Method Dataset 5/class 10/class 20/class NodeJEPA Computers 71.75± 2.88 75.37± 1.48 79.12± 0.45 NodeJEPA Photo 83.99± 3.02 85.41± 2.52 87.97± 0.84 NodeJEPA Coauthor-CS 88.51± 0.73 89.19± 1.17 89.16± 0.99 NodeJEPA Physics 87.22± 1.63 88.72± 1.76 90.51± 1.82 NodeJEPA arxiv 41.58± 1.16 45.74± 1.94 47.54± 0.52 PatchJEPA Computers 71.76± 4.14 75.94± 1.88 80.44± 0.77 PatchJEPA Photo 84.72± 0.47 87.92± 2.01 89.17± 0.86 PatchJEPA Coauthor-CS 88.33± 0.39 89.21± 0.44 89.38± 0.60 PatchJEPA Physics 82.21± 2.18 85.98± 4.24 89.79± 2.46 PatchJEPA arxiv 42.78± 1.70 48.22± 0.64 50.76± 0.73 DGI Computers 51.14± 14.00 55.39± 16.22 63.41± 13.35 DGI Photo 44.38± 12.76 55.84± 12.36 64.89± 9.19 DGI Coauthor-CS 84.88± 1.38 86.67± 0.51 87.84± 0.99 DGI Physics 87.64± 2.15 91.19± 1.14 92.42± 0.91 GraphMAE Computers 58.93± 5.32 62.28± 5.42 68.85± 4.95 GraphMAE Photo 81.64± 2.25 83.02± 1.43 85.28± 1.58 GraphMAE Coauthor-CS 89.68± 0.26 90.23± 0.64 90.54± 0.33 GraphMAE Physics 86.84± 2.04 87.90± 2.98 89.93± 2.28 GraphMAE arxiv 39.17± 1.66 46.89± 1.00 49.98± 0.94 BGRL Computers 53.38± 3.39 57.37± 3.10 62.19± 3.23 BGRL Photo 73.75± 6.26 77.67± 2.11 79.95± 1.32 BGRL Coauthor-CS 88.23± 0.81 89.10± 1.06 89.31± 0.99 BGRL Physics 87.92± 0.82 90.30± 0.92 91.56± 1.23 BGRL arxiv 40.79± 1.52 47.30± 1.25 49.89± 0.51 CCA-SSG Computers 70.26± 3.97 72.87± 2.48 76.99± 1.40 CCA-SSG Photo 81.06± 3.28 84.27± 2.47 87.08± 1.02 CCA-SSG Coauthor-CS 85.47± 0.82 87.42± 0.60 87.64± 0.70 CCA-SSG Physics 89.52± 1.53 92.46± 0.58 93.27± 0.52 CCA-SSG arxiv 39.92± 2.50 44.04± 1.61 46.44± 0.39 Sup. GCN Computers 80.51± 3.38 80.91± 2.35 82.90± 1.39 Sup. GCN Photo 88.11± 2.30 89.56± 1.51 89.71± 1.27 Sup. GCN Coauthor-CS 89.19± 0.44 89.27± 0.70 89.42± 0.65 Sup. GCN Physics 90.17± 1.52 91.45± 0.85 92.27± 0.68 Sup. GCN arxiv 52.90± 1.42 53.33± 1.00 53.17± 0.65 Table 10: Few-shot linear-probe accuracy (%). Appendix J Ablations Table 11 and Figure 11 match the main-paper ablation suite (3 seeds), including the attention predictor at 90.18±0.6490.18± 0.64 on Coauthor-Physics. Variant Amazon-Photo Coauthor-Phy. A: full 87.87 ± 0.69 90.57 ± 0.80 B: no structure 88.74 ± 0.49 90.61 ± 0.43 C: attn. predictor 87.53 ± 0.78 90.18 ± 0.64 D: no VICReg 84.61 ± 1.67 92.67 ± 0.29 E: no isolation 87.54 ± 0.85 90.65 ± 0.88 F: ℓ2 _2 loss 88.36 ± 0.56 89.79 ± 0.05 Figure 9: NodeJEPA ablation results on Amazon-Photo and Coauthor-Physics. Table 11: NodeJEPA ablations (3 seeds), matching the main paper. Default recipe: isolate targets, cosine prediction, restricted message-passing predictor, and VICReg-style regularization. Structural descriptors remain an optional conditioning channel. Cross-attention (C) matches the default closely (90.1890.18 vs. 90.5790.57 on Physics). Appendix K Representation-Collapse Diagnostics Figure 10 and Tables 13–13 report effective rank, mean per-dimension standard deviation, and participation ratio across datasets. Figure 10: Top: effective rank of frozen embeddings. Bottom: mean per-dimension standard deviation. NodeJEPA leads self-supervised methods on every logged dataset. Method Photo Coauthor-CS Coauthor-Phy. arxiv DGI 6.7 23.5 21.9 – GraphMAE 11.8 57.7 31.1 13.4 BGRL 3.9 41.1 45.1 16.4 CCA-SSG 14.9 17.0 14.4 18.9 Supervised GCN 47.4 58.0 28.6 99.6 NodeJEPA 46.3 65.2 78.5 68.9 PatchJEPA 28.6 15.7 14.6 22.0 Table 12: Effective rank (higher means less collapse). Dataset Method EffRank MeanStd PartRatio Photo NodeJEPA 46.3±5.646.3±5.6 0.514±0.1120.514±0.112 1.87±0.461.87±0.46 Photo PatchJEPA 28.6±5.128.6±5.1 0.323±0.0400.323±0.040 1.27±0.081.27±0.08 Photo GraphMAE 11.8±2.711.8±2.7 0.304±0.0710.304±0.071 1.27±0.131.27±0.13 Photo CCA-SSG 14.9±1.014.9±1.0 0.289±0.0150.289±0.015 1.25±0.031.25±0.03 Photo BGRL 3.9±1.43.9±1.4 0.070±0.0300.070±0.030 1.01±0.011.01±0.01 CS NodeJEPA 65.2±2.665.2±2.6 0.898±0.0050.898±0.005 7.84±0.337.84±0.33 CS GraphMAE 57.7±0.757.7±0.7 0.580±0.0050.580±0.005 2.27±0.042.27±0.04 CS BGRL 41.1±1.741.1±1.7 0.633±0.0060.633±0.006 2.67±0.062.67±0.06 CS PatchJEPA 15.7±1.915.7±1.9 0.161±0.0230.161±0.023 1.06±0.021.06±0.02 Physics NodeJEPA 78.5±3.978.5±3.9 0.887±0.0080.887±0.008 7.18±0.537.18±0.53 Physics BGRL 45.1±3.645.1±3.6 0.598±0.0180.598±0.018 2.36±0.142.36±0.14 Physics GraphMAE 31.1±3.231.1±3.2 0.339±0.0390.339±0.039 1.29±0.081.29±0.08 Physics PatchJEPA 14.6±0.114.6±0.1 0.144±0.0050.144±0.005 1.04±0.001.04±0.00 arxiv NodeJEPA 68.9±8.968.9±8.9 0.492±0.0570.492±0.057 1.75±0.271.75±0.27 arxiv PatchJEPA 22.0±2.222.0±2.2 0.166±0.0130.166±0.013 1.06±0.011.06±0.01 arxiv CCA-SSG 18.9±1.118.9±1.1 0.276±0.0160.276±0.016 1.20±0.021.20±0.02 arxiv BGRL 16.4±3.216.4±3.2 0.284±0.0300.284±0.030 1.21±0.051.21±0.05 Table 13: Expanded collapse diagnostics. Appendix L Efficiency Analysis Figure 11 and Table 14 plot accuracy against wall-clock time and list mean training times on all five benchmarks, including per-epoch dynamics on ogbn-arxiv. Figure 11: Top: accuracy vs. wall-clock time. Middle: total training time on four medium graphs. Bottom: per-epoch time on ogbn-arxiv. Method Computers Photo Coauthor-CS Physics arxiv DGI 99.8 16.2 33.7 61.8 1848.4 GraphMAE 52.0 21.0 28.7 47.9 2117.7 BGRL 117.0 44.0 54.6 106.3 2436.5 CCA-SSG 82.4 33.4 39.5 75.7 2339.2 Sup. GCN 41.1 15.2 13.4 36.1 1457.7 NodeJEPA 168.3 108.4 506.4 1585.5 31582.3 PatchJEPA 113.5 107.2 76.6 150.8 2466.3 Table 14: Mean total training wall-clock time (seconds). The dual-variant design offers two operating points: NodeJEPA for fine-grained node-level geometry on medium graphs, and PatchJEPA for cached-partition scaling on large hub-heavy graphs. On ogbn-arxiv, NodeJEPA’s median epoch time rises from about 1111s to about 147147s after the 2-hop curriculum, while PatchJEPA stays near 1.81.8s per epoch. Appendix M Design Notes Structural descriptors (PageRank, degree, clustering, spectral coordinates) are optional. The NodeJEPA curriculum raises mask ratio 0.2→0.50.2→ 0.5 and hop radius 1→21→ 2 over the first 50 epochs. Context isolation replaces target features with a mask token. Node-level and patch-level masking share the same JEPA loss family and differ mainly in the masking unit. Appendix N Additional Method Details EMA targets use momentum 0.996→0.9990.996→ 0.999 (NodeJEPA) or 0.9960.996 (PatchJEPA). The predictive loss is mean cosine distance. SIGReg uses 256 random projections with weight 0.020.02. PatchJEPA mean-pools within patches only for the pretraining target; probing still uses node-level encoder outputs. Appendix O Broader Impact This work studies self-supervised learning on standard public benchmarks and does not introduce new sensitive data. Downstream deployments should audit structural or attribute biases as appropriate. Identifying the k-hop-search bottleneck and providing a cached patch alternative reduces compute on large graphs. Appendix P Reproducibility Notes Seeds 4242–4646, splits, and hyperparameters are fully specified above. Code, configurations, scripts, and aggregated logs (MIT license) are publicly available at https://github.com/OliverZ-dot/Node-Jepa. References M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas (2023) Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15619–15629. Cited by: §1, §2. R. Balestriero and Y. LeCun (2025) LeJEPA: provable and scalable self-supervised learning without the heuristics. External Links: 2511.08544 Cited by: §1, §2, §3.2. A. Bardes, J. Ponce, and Y. LeCun (2022) VICReg: variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, Cited by: §1, §2, §3.2. M. Belkin and P. Niyogi (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation 15 (6), p. 1373–1396. Cited by: §3.2. Y. Bengio, J. Louradour, R. Collobert, and J. Weston (2009) Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning, p. 41–48. Cited by: §1. M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst (2017) Geometric deep learning: going beyond Euclidean data. IEEE Signal Processing Magazine 34 (4), p. 18–42. Cited by: §1. M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin (2021) Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 9650–9660. Cited by: §2. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton (2020) A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, p. 1597–1607. Cited by: §2. X. Chen and K. He (2021) Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 15750–15758. Cited by: §2. W. Chiang, X. Liu, S. Si, Y. Li, S. Bengio, and C. Hsieh (2019) Cluster-GCN: an efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 257–266. Cited by: §2. J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl (2017) Neural message passing for quantum chemistry. In International Conference on Machine Learning, p. 1263–1272. Cited by: §7. J. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. (2020) Bootstrap your own latent: a new approach to self-supervised learning. In Advances in Neural Information Processing Systems, Vol. 33, p. 21271–21284. Cited by: §2. A. Grover and J. Leskovec (2016) node2vec: scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 855–864. Cited by: §1. W. Hamilton, Z. Ying, and J. Leskovec (2017) Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §6. K. Hassani and A. H. Khasahmadi (2020) Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning, p. 4116–4126. Cited by: §2. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick (2022) Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 16000–16009. Cited by: §2. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick (2020) Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9729–9738. Cited by: §2. Z. Hou, Y. He, Y. Cen, X. Liu, Y. Dong, E. Kharlamov, and J. Tang (2023) GraphMAE2: a decoding-enhanced masked self-supervised graph learner. In Proceedings of the ACM Web Conference 2023, p. 737–746. Cited by: §2. Z. Hou, X. Liu, Y. Cen, Y. Dong, H. Yang, C. Wang, and J. Tang (2022) GraphMAE: self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 594–604. Cited by: §1, §2, §4.1. W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec (2020a) Open graph benchmark: datasets for machine learning on graphs. In Advances in Neural Information Processing Systems, Vol. 33, p. 22118–22133. Cited by: Table 4, §4.1. W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. Pande, and J. Leskovec (2020b) Strategies for pre-training graph neural networks. In International Conference on Learning Representations, Cited by: §1, §2. Y. Jiao, Y. Xiong, J. Zhang, Y. Zhang, T. Zhang, and Y. Zhu (2020) Sub-graph contrast for scalable self-supervised graph representation learning. In IEEE International Conference on Data Mining, p. 222–231. Cited by: §2. L. Jing, P. Vincent, Y. LeCun, and Y. Tian (2022) Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, Cited by: §1, §2. G. Karypis and V. Kumar (1998) A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal on Scientific Computing 20 (1), p. 359–392. Cited by: §1, §2, §3.3. T. N. Kipf and M. Welling (2016) Variational graph auto-encoders. In NeurIPS Workshop on Bayesian Deep Learning, Cited by: §2. T. N. Kipf and M. Welling (2017) Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, Cited by: §1, §4.1. Y. LeCun (2022) A path towards autonomous machine intelligence version 0.9.2, 2022-06-27. Open Review 62. Cited by: §1, §2. L. Page, S. Brin, R. Motwani, and T. Winograd (1999) The pagerank citation ranking: bringing order to the web. Stanford InfoLab Technical Report. Cited by: §1, §3.2. H. Pei, B. Wei, K. C. Chang, Y. Lei, and B. Yang (2020) Geom-GCN: geometric graph convolutional networks. In International Conference on Learning Representations, Cited by: §7. B. Perozzi, R. Al-Rfou, and S. Skiena (2014) DeepWalk: online learning of social representations. In Proceedings of the 20th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 701–710. Cited by: §1. J. Qiu, Q. Chen, Y. Dong, J. Zhang, H. Yang, M. Ding, K. Wang, and J. Tang (2020) GCC: graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 1150–1160. Cited by: §2. O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann (2018) Pitfalls of graph neural network evaluation. External Links: 1811.05868 Cited by: Table 4, §2, §4.1. G. Skenderi, H. Li, J. Tang, and M. Cristani (2023) Graph-level representation learning with joint-embedding predictive architectures. External Links: 2309.16014 Cited by: §1, §1, §2, §3.3, 4. F. Sun, J. Hoffmann, V. Verma, and J. Tang (2020) InfoGraph: unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations, Cited by: §2. Q. Tan, N. Liu, X. Huang, S. Choi, L. Li, R. Chen, and X. Hu (2023) What’s behind the mask: understanding masked graph modeling for graph autoencoders. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 2378–2388. Cited by: §2. A. Tarvainen and H. Valpola (2017) Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §1. S. Thakoor, C. Tallec, M. G. Azar, M. Azabou, E. L. Dyer, R. Munos, P. Veličković, and M. Valko (2022) Large-scale representation learning on graphs via bootstrapping. In International Conference on Learning Representations, Cited by: §1, §2, §4.1. A. van den Oord, Y. Li, and O. Vinyals (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: §1, §2. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio (2018) Graph attention networks. In International Conference on Learning Representations, Cited by: §7. P. Veličković, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, and R. D. Hjelm (2019) Deep graph infomax. In International Conference on Learning Representations, Cited by: §1, §2, §4.1. D. J. Watts and S. H. Strogatz (1998) Collective dynamics of ‘small-world’ networks. Nature 393 (6684), p. 440–442. Cited by: §1, §3.2. Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu (2020) A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems 32 (1), p. 4–24. Cited by: §1. Y. Xie, Z. Xu, J. Zhang, Z. Wang, and S. Ji (2022) Self-supervised learning of graph neural networks: a unified review. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (2), p. 2412–2429. Cited by: §1. K. Xu, W. Hu, J. Leskovec, and S. Jegelka (2019) How powerful are graph neural networks?. In International Conference on Learning Representations, Cited by: §7. Y. You, T. Chen, Y. Shen, and Z. Wang (2021) Graph contrastive learning automated. In International Conference on Machine Learning, p. 12121–12132. Cited by: §1, §2. Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen (2020) Graph contrastive learning with augmentations. In Advances in Neural Information Processing Systems, Vol. 33, p. 5812–5823. Cited by: §2. J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny (2021) Barlow twins: self-supervised learning via redundancy reduction. In International Conference on Machine Learning, p. 12310–12320. Cited by: §2. H. Zhang, Q. Wu, J. Yan, D. Wipf, and P. S. Yu (2021) From canonical correlation analysis to self-supervised graph neural networks. In Advances in Neural Information Processing Systems, Vol. 34, p. 76–89. Cited by: §1, §2, §4.1. J. Zhu, Y. Yan, L. Zhao, M. Heimann, L. Akoglu, and D. Koutra (2020a) Beyond homophily in graph neural networks: current limitations and effective designs. In Advances in Neural Information Processing Systems, Vol. 33, p. 7793–7804. Cited by: §7. Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang (2020b) Deep graph contrastive representation learning. External Links: 2006.04131 Cited by: §2. Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang (2021) Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, p. 2069–2080. Cited by: §2.