Paper deep dive
ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Models
Jingnan Pu, Zi-En Fan, Feng Lian
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models (LLMs) excel at token-level generation but may learn undesirable abstract semantics and lack comprehensive perception. LLM-JEPA mitigates this by aligning different views of the same underlying knowledge via a joint-embedding predictive architecture (JEPA). However, strong alignment does not necessarily lead to accurate, stable predictions. To address this, we propose ER-JEPA, which adds an episodic replay path to LLM-JEPA. ER-JEPA stores training pairs in a memory. At each step, it stores and retrieves relevant data to provide additional supervision. This enables learning from both the current batch and stored training pairs, providing additional supervision for token prediction and representation alignment. Experiments across multiple datasets (NL-RX, GSM8K, Spider, and NQ-Open) demonstrate that ER-JEPA consistently outperforms LLM-JEPA.
Tags
Links
- Source: https://arxiv.org/abs/2609.36952v1
- Canonical: https://arxiv.org/abs/2609.36952v1
Trouble viewing inline? Open PDF directly →
Full Text
60,285 characters extracted from source content.
Expand or collapse full text
ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Models Jingnan Pu Affiliation: School of Automation Science and Technology, Xi’an Jiaotong University No. 28, West Xianning Road, Xi’an, Shaanxi 710049, China Zi-En Fan Affiliation: School of Automation Science and Technology, Xi’an Jiaotong University No. 28, West Xianning Road, Xi’an, Shaanxi 710049, China Feng Lian Affiliation: School of Automation Science and Technology, Xi’an Jiaotong University No. 28, West Xianning Road, Xi’an, Shaanxi 710049, China Corresponding author Abstract Large language models (LLMs) excel at token-level generation but may learn undesirable abstract semantics and lack comprehensive perception. LLM-JEPA mitigates this by aligning different views of the same underlying knowledge via a joint-embedding predictive architecture (JEPA). However, strong alignment does not necessarily lead to accurate, stable predictions. To address this, we propose ER-JEPA, which adds an episodic replay path to LLM-JEPA. ER-JEPA stores training pairs in a memory. At each step, it stores and retrieves relevant data to provide additional supervision. This enables learning from both the current batch and stored training pairs, providing additional supervision for token prediction and representation alignment. Experiments across multiple datasets (NL-RX, GSM8K, Spider, and NQ-Open) demonstrate that ER-JEPA consistently outperforms LLM-JEPA. * 0pt1.25ex plus 1ex minus .2ex0.75em Figure 1: ER-JEPA outperforms LLM-JEPA across tasks. 1 Introduction Large language models (LLMs) have achieved remarkable advances in fluent generation (Pillutla et al., 2021; Yu et al., 2023; Li et al., 2025; Gong et al., 2024; Li et al., 2024; Liu et al., 2024), few-shot learning (Brown et al., 2020; Wei et al., 2021; Wang et al., 2024), and broad downstream transfer (Rafailov et al., 2023; Zeng et al., 2022; Huan et al., 2025). These abilities require models to extract shared structure from large-scale data and convert it into dependable predictions on new problems. Autoregressive pre-training addresses both needs through next-token prediction (NTP) (see Figure 2(a)), which provides a strong learning signal and can lead to semantic representations under the right conditions (Jin and Rinard, 2024). However, a token-level objective may also favor local coherence over the representations needed for comprehensive perception and reasoning. Joint-Embedding Predictive Architectures (JEPAs) shift prediction from the token level to the semantic level. Recent work on JEPAs has shown promising advances in representation learning and provable benefits for perception tasks (Assran et al., 2023; Bardes et al., 2024; Assran et al., 2025; Chen et al., 2026). Inspired by JEPAs, Huang et al. introduced LLM-JEPA (Huang et al., 2026), which adds an embedding space predictive loss to the standard language modeling loss (see Figure 2(b)). This loss aligns different views of the same underlying knowledge while preserving generative capability. It predicts both the next token and the target representation, improving the perception capability of LLMs. Despite these advances, better representation alignment does not by itself guarantee reliable predictions. We observe that LLM-JEPA continues to make errors even after its alignment loss has converged (Figure 3). Some previously correct predictions also become incorrect later in training. Therefore, progress in representation learning is not automatically converted into predictions that are learned and retained. This leaves an open question: how can LLM-JEPA correct persistent errors and retain correct predictions during training? We explore experience replay as a way to address this question by revisiting previously seen training examples and providing additional supervision as the model changes. Like standard fine-tuning, LLM-JEPA supervises an example only when it appears in the current mini-batch. Between two visits, many updates change the model, and the example is not checked again until the data schedule reaches it in the next epoch. Replay can reduce this gap by revisiting past examples during training. The replay mechanism is widely used in continual learning to preserve learned knowledge (Shi et al., 2024; Huang et al., 2024; Deng et al., 2025), and Complementary Learning Systems (CLS) theory gives it a similar role in the brain (Kumaran et al., 2016). In CLS, a fast system stores individual experiences and replays them to a slow system that extracts shared structure. From this perspective, LLM-JEPA is similar to the former but has no counterpart to the fast system. This motivates our research question: can experience replay improve the predictive performance of LLM-JEPA? To answer it, we propose ER-JEPA, which adds an episodic replay (ER) pathway that stores past training examples and replays them during later updates (see Figure 2(c)). It enables the model to learn from useful past experiences beyond the current mini-batch, providing additional supervision signals and strengthening the perception capabilities of LLMs. The replay path is removed after training, so ER-JEPA has the same architecture and inference cost as LLM-JEPA. In summary, the contributions of this paper are as follows: • We propose ER-JEPA, which extends LLM-JEPA with an episodic replay path that revisits past training examples to provide additional supervision for token prediction and representation alignment. We show that learning from past samples continues to provide alignment updates and reinforces the knowledge important for perception and reasoning. • We implement content-based, uniform, and hard selection within a shared replay path. We demonstrate that the replay branch itself, rather than any specific retrieval rule, is the central source of improvement. • We evaluate ER-JEPA across multiple datasets. The results show that ER-JEPA consistently outperforms LLM-JEPA, demonstrating the benefit of episodic replay for abstraction and generalization. Figure 2: Comparison of baseline training, LLM-JEPA, and ER-JEPA. (a) The baseline learns through next token prediction. (b) LLM-JEPA adds prediction in the embedding space to learn abstract relations between inputs and targets. (c) ER-JEPA adds a replay path to LLM-JEPA. It stores past training examples and replays their tokens, providing additional supervision for both token prediction and representation alignment. Past experience thus continues to shape learning as the model changes, helping it correct errors and reinforce learned knowledge for more reliable predictions. 2 Methodology ER-JEPA extends LLM-JEPA with a replay path over stored training examples. We first introduce the joint training objective in Section 2.1. We then explain how replay provides additional supervision in Section 2.2. 2.1 ER-JEPA Overview and Objective Our ER-JEPA builds on LLM-JEPA, retaining its token prediction and representation alignment objectives while adding an episodic replay path to learn from past training examples. The original LLM-JEPA objective is ℒLLM-JEPA=∑ℓ=2LℒLLM(Text1:ℓ−1,Textℓ)⏟generative capabilities (LLM)+λd(Predϕ(Encθ(Text)),Encθ(Code))⏟abstraction capabilities (JEPA).L_ LLM-JEPA= _ =2^LL_ LLM\! (Text_1: -1,Text_ )_generative capabilities (LLM)+λ\, d\! ( Pred_φ( Enc_θ(Text)),\; Enc_θ(Code) )_abstraction capabilities (JEPA). (1) Here TextText and CodeCode denote the source and target views, and L is the source sequence length. EncθEnc_θ and PredϕPred_φ denote the encoder and predictor. The function d measures cosine distance, and λ controls the JEPA loss weight. For a fresh mini-batch ℬtB_t of size B, let xiux_i^u and xiax_i^a denote the source and target tokens of example i, and xifull=xiu|xiax_i full=x_i^u\|x_i^a. Define ziu=Encθ(xiu),pi=Predϕ(ziu),zia=Encθ(xia).z_i^u=Enc_θ(x_i^u), p_i=Pred_φ(z_i^u), z_i^a=Enc_θ(x_i^a). (2) Let inewT_i new contain the target-token positions in xifullx_i full. The loss on new data is ℒLLM-JEPA-new(ℬt)=γZtnew∑i=1B∑ℓ∈inewℒLLM(xi,<ℓfull,xi,ℓfull)+λB∑i=1Bd(pi,zia),L_ LLM-JEPA-new(B_t)= γZ_t new _i=1^B _ _i newL_ LLM\! (x_i,< full,x_i, full )+ λB _i=1^Bd(p_i,z_i^a), (3) where Ztnew=∑i=1B|inew|Z_t new= _i=1^B|T_i new| and γ=1γ=1 is the language-modeling loss weight. The full ER-JEPA objective adds a replay term, ℒER-JEPA(ℬt)=ℒLLM-JEPA-new(ℬt)+βℒLLM-JEPA-replay(ℬt),L_ ER-JEPA(B_t)=L_ LLM-JEPA-new(B_t)+β\,L_ LLM-JEPA-replay(B_t), (4) where β≥0β≥ 0 controls the strength of replay. ℒLLM-JEPA-replayL_ LLM-JEPA-replay is defined in Section 2.2. 2.2 Replay-Based Training Content-based, uniform, and hard replay are three implementations of the same replay path. They share the replay objective below. The selection rules for all three strategies are given in Appendix D.3. Unless otherwise specified, ER-JEPA uses content-based replay. At each step, a replay strategy selects up to R entries from the episodic memory, where R is the replay budget. Let RtR_t be the number of selected entries and (τ~iu,τ~ia)( τ_i^u, τ_i^a) be the stored source and target tokens for entry i. Memory storage and retrieval are described in Appendix D.1. For each replay entry i, we concatenate the stored source and target tokens as τ~ifull=τ~iu|τ~ia τ_i full= τ_i^u\| τ_i^a. The current model recomputes the source and target representations, ziu,rep=Encθ(τ~iu),zia,rep=Encθ(τ~ia),pirep=Predϕ(ziu,rep).z_i^u, rep=Enc_θ( τ_i^u), z_i^a, rep=Enc_θ( τ_i^a), p_i rep=Pred_φ(z_i^u, rep). (5) Let irepT_i rep contain the target-token positions in τ~ifull τ_i full. For Rt>0R_t>0, the replay loss is ℒLLM-JEPA-replay(ℬt)=γZtrep∑i=1Rt∑ℓ∈irepℒLLM(τ~i,<ℓfull,τ~i,ℓfull)+λRt∑i=1Rtd(pirep,zia,rep),L_ LLM-JEPA-replay(B_t)= γZ_t rep _i=1^R_t _ _i repL_ LLM\! ( τ_i,< full, τ_i, full )+ λR_t _i=1^R_td\! (p_i rep,z_i^a, rep ), (6) where Ztrep=∑i=1Rt|irep|Z_t rep= _i=1^R_t|T_i rep|. The loss is zero when Rt=0R_t=0. The normalization matches the new-data loss in Eq. (3). The current and replay losses jointly update the shared model through Eq. (4). Replay-time JEPA errors refresh the selected memory scores, and replay counts are incremented as specified in Appendix D.2. At inference time, the episodic pathway is removed. Figure 3: LLM-JEPA can achieve strong alignment yet still make errors and lose previously correct predictions. For LLM-JEPA, the JEPA loss converges in (a). Yet in (b) and (c), errors remain even when cosine similarities between inputs and targets are close to one. Some correct predictions also become incorrect as training continues. For ER-JEPA, the JEPA loss also converges in (d). In (e) and (f), more predictions are correct despite slightly lower cosine similarities. At each checkpoint, ER-JEPA consistently has more correct answers than LLM-JEPA. These results suggest that replay supervision helps resolve errors that alignment alone leaves unresolved. This figure uses Llama-3.2-1B on 2000 SYNTH test examples with seed 82. Heatmap rows are sorted independently within each method by final-checkpoint correctness and prediction history. 3 Experiments: Episodic Replay Improves LLM-JEPA This section evaluates whether episodic replay improves JEPA-based LLM training. After describing the experimental setup in Section 3.1, we report results across tasks and compare accuracy at matched training compute in Section 3.2. We then examine the role of historical replay in Section 3.3, analyze prediction quality across regex structures and lengths and report resource costs in Section 3.4. Further ablations are provided in Appendix E. 3.1 Experimental Setup We evaluate ER-JEPA under the same task families used in the LLM-JEPA protocol. Experiments are conducted on Llama-3.2-1B-Instruct (Grattafiori et al., 2024). We use five datasets with naturally paired input–target structures: NL-RX-SYNTH and NL-RX-TURK (Locascio et al., 2016) for generating regular expressions from descriptions in natural language, GSM8K (Cobbe et al., 2021) for mathematical reasoning, Spider (Yu et al., 2018) for text-to-SQL generation, and NQ-Open (Lee et al., 2019) for open-domain question answering. Following the evaluation protocol of LLM-JEPA, we report exact match accuracy of the generated regular expression for NL-RX-SYNTH and NL-RX-TURK, exact match accuracy of the final answer for GSM8K and NQ-Open, and execution accuracy for Spider. Detailed experimental settings are provided in Appendix A. 3.2 ER-JEPA Outperforms LLM-JEPA Episodic replay improves LLM-JEPA across tasks. We first test whether replay provides benefits beyond representation alignment alone. We fine-tune Llama-3.2-1B-Instruct with different training objectives on all five datasets described above. As shown in Figure 1 (left), ER-JEPA achieves higher performance than LLM-JEPA on all five datasets. The consistent gains show that the benefit of ER-JEPA is not limited to one type of task. The method improves both structured generation tasks, such as regular expression and SQL generation, and more general generation tasks, such as mathematical reasoning and open-domain question answering. These results suggest that episodic replay improves the use of training data in LLM-JEPA. By storing separated traces of past examples and retrieving relevant traces for the current context, ER-JEPA allows the model to reuse useful past experiences during optimization. This mechanism provides additional training signals beyond the current mini-batch and leads to more effective fine-tuning. ER-JEPA achieves higher accuracy at the same training compute. To test whether the gains persist at the same training compute, we evaluate all methods at the same budgets from 40.0540.05 to 240.31240.31 PFLOPs on SYNTH with Llama-3.2-1B. We also evaluate three replay policies, including content, uniform, and hard replay, to test whether the benefit depends on the replay selection rule. The three policies share the same replay path and differ only in how stored examples are selected. Figure 4(a) shows that all three replay policies outperform LLM-JEPA at every evaluated compute budget. At 240.31240.31 PFLOPs, content replay reaches 83.65±1.48%83.65± 1.48\%, while LLM-JEPA reaches 68.75±4.92%68.75± 4.92\%, a gain of 14.9014.90 percentage points. The normalized AUC also increases from 56.61±3.41%56.61± 3.41\% to 73.73±1.15%73.73± 1.15\%. The gains across all three rules show that the benefit is not limited to content-based selection. At matched training compute, ER-JEPA achieves higher mean accuracy than LLM-JEPA on SYNTH with all three replay policies. (a) Five seeds. (b) Seed =84=84. Figure 4: ER-JEPA improves accuracy at matched training compute and reduces overfitting. (a) All three replay policies outperform LLM-JEPA at every evaluated budget. These gains support the benefit of introducing the replay path. (b) Baseline loses accuracy after an early increase. LLM-JEPA delays this decline but still loses accuracy later. Content and uniform replay maintain their gains, while hard replay shows only a small final decline. These trends suggest that replay mitigates overfitting. They complement the mean curves in Figure 1 (right). Results use Llama-3.2-1B on SYNTH at six compute budgets from 40.0540.05 to 240.31240.31 PFLOPs. Replay reduces accuracy drops during training. The mean curves in Figure 4(a) rise at every budget for all methods, but individual runs behave differently. Baseline and LLM-JEPA often lose accuracy that they reached earlier in training. These drops happen at different budgets in different seeds, so averaging hides them. Figure 4(b) illustrates this pattern for seed 84. Baseline loses accuracy after an initial increase, and LLM-JEPA shows a decline at a later checkpoint. ER-JEPA shows strong resistance to overfitting under all three replay policies. These results suggest that the replay path mitigates overfitting and helps retain the accuracy gained earlier in training. Appendix Figure 13 shows the other four seeds. 3.3 Understanding the Role of Historical Replay To analyze these gains, we first compare ER-JEPA with token-matched LLM-JEPA to assess whether additional token exposure alone can reproduce the improvement. We then compare historical replay with a control that applies the same replay loss to current-batch examples, assessing the additional benefit of revisiting past samples. We also examine which errors replay corrects and whether it keeps correct predictions. Appendix B analyzes a single replay update. Content replay achieves higher mean accuracy than token-matched LLM-JEPA. We first test whether extra token exposure can explain the gain. The token-matched control uses the token budget of content replay and retains the LLM-JEPA objective. Figure 6 shows that content replay reaches 83.65±1.48%83.65± 1.48\%, compared with 62.25±20.80%62.25± 20.80\% for token-matched LLM-JEPA. These results indicate that extra token exposure alone does not reproduce the gain. Historical replay achieves higher mean accuracy than the current-batch control. We next examine whether historical samples provide value beyond an additional loss on the current batch. The current-batch control applies the replay loss to current-batch examples instead of stored examples. Figure 6 shows that all three historical replay policies have higher mean accuracy than this control. Uniform replay reaches 85.04±0.54%85.04± 0.54\%, which is 3.803.80 percentage points (p) above the control, and it is higher in all five seeds. Content replay reaches 83.65%83.65\% (+2.41+2.41 p), and hard replay reaches 83.41%83.41\% (+2.17+2.17 p). These results are consistent with a benefit from reusing historical samples. Figure 5: ER-JEPA achieves higher mean accuracy with replay under matched token exposure. Results use Llama-3.2-1B on SYNTH over five seeds. Figure 6: Historical samples provide additional value. Results use Llama-3.2-1B on SYNTH over five seeds. ER-JEPA achieves higher correction rates on examples mispredicted by the baseline. As discussed in the Introduction, a model can align well with its target representation and still predict the wrong answer, sometimes even losing ground it had already covered. This raises a natural question: when Baseline errs, can LLM-JEPA and ER-JEPA fix its mistakes? To find out, we test both models on the SYNTH inputs where Baseline fails at the final matched compute point, checking whether each model recovers the exact target expression. We sort these failures by length relative to the target, including over-generation (extra regex tokens), under-generation (missing tokens), and same-length mismatch (correct length, wrong expression). Table 1 shows that ER-JEPA achieves a higher mean correction rate in all three categories. The largest gain occurs for over-generation, where the rate rises from 53.07%53.07\% with LLM-JEPA to 84.51%84.51\% with ER-JEPA. Complete correction counts and rates for each seed are reported in Appendix Table 3. Appendix Figure 10 further compares the two methods on inputs that LLM-JEPA predicts incorrectly. To further examine whether replay helps correct errors and retain correct predictions during training, we track predictions on the same test inputs across successive checkpoints. Figure 3 shows more correct predictions for ER-JEPA in (f) than for LLM-JEPA in (c) at every evaluated checkpoint. Figure 7(b) shows a higher mean correction rate for ER-JEPA than for LLM-JEPA. In Figure 7(c), the mean rate of correct predictions becoming incorrect decreases from 8.85%8.85\% with LLM-JEPA to 6.65%6.65\% with ER-JEPA. These results suggest that replay supports both error correction and the retention of correct predictions. Example inputs, targets, and predictions from the three methods are shown in Appendix Figure 11. Table 1: ER-JEPA achieves higher mean correction rates across all three Baseline error categories. Results use Llama-3.2-1B on SYNTH at 240.309 PFLOPs. Values are means ± one sample standard deviation over five seeds. Correction rates are computed separately for each seed before averaging. Baseline error category Baseline errors Method Corrected Correction rate (%) Over-generation 718.4±112.5718.4± 112.5 LLM-JEPA 378.8±64.6378.8± 64.6 53.07±7.9253.07± 7.92 ER-JEPA 610.8±126.1610.8± 126.1 84.51±4.7084.51± 4.70 Under-generation 2.6±1.52.6± 1.5 LLM-JEPA 0.6±0.50.6± 0.5 24.00±25.1024.00± 25.10 ER-JEPA 1.2±0.81.2± 0.8 54.67±44.0754.67± 44.07 Same-length mismatch 168.4±10.9168.4± 10.9 LLM-JEPA 34.8±5.834.8± 5.8 20.58±2.3520.58± 2.35 ER-JEPA 42.4±7.042.4± 7.0 25.07±2.6225.07± 2.62 Figure 7: ER-JEPA achieves higher accuracy and corrects more errors during training. (a) Test accuracy at comparable training compute for SFT, LLM-JEPA, and ER-JEPA. (b) The fraction of wrong predictions that become correct at the next checkpoint. (c) The fraction of correct predictions that become wrong at the next checkpoint. Compared with LLM-JEPA, ER-JEPA improves error correction across all five seeds, while the reduction in lost correct predictions is smaller and varies across seeds. Results use Llama-3.2-1B on the same 2,000 SYNTH test examples. 3.4 Fine-grained Analysis of Prediction Quality We next examine where the accuracy gains occur within the SYNTH test set, focusing on the structure and length of the target regular expressions. Accuracy gains span multiple forms of symbolic composition. To examine how these gains relate to symbolic structure, we analyze the SYNTH test set using six structural subsets defined by the target regular expressions, including complement, intersection, alternation, quantifiers, boundaries and anchors, and character classes. Detailed definitions and classification rules are provided in Appendix Table 2. Figure 9 shows that ER-JEPA achieves higher mean exact match accuracy than LLM-JEPA in all six subsets. The largest gains occur for expressions containing quantifiers, intersections, and character classes, with improvements of 16.9016.90, 16.0716.07, and 15.4215.42 percentage points, respectively. These structures encode repetition, conjunction, and restrictions on allowed characters, all of which must be preserved when translating a natural language description into a regular expression. The gains therefore span several forms of constraint composition. This pattern is consistent with replay reinforcing the mapping from linguistic constraints to their symbolic implementation. Replay reduces sensitivity to expression length. We next examine whether this advantage persists for longer expressions. Target regex length serves as a proxy for complexity. We divide the SYNTH test set into four groups: ≤10≤ 10, 1111–1515, 1616–2020, and >20>20 regex tokens. Figure 9 shows that Regular is highly sensitive to expression length. Its mean accuracy drops from 83.64%83.64\% in the shortest group to 33.53%33.53\% in the longest group, a decrease of 50.1150.11 percentage points. LLM-JEPA reduces this gap, but its accuracy still falls from 85.15%85.15\% to 56.76%56.76\%. ER-JEPA achieves the highest accuracy in all four groups. Its accuracy decreases from 89.80%89.80\% to 78.53%78.53\%, a drop of 11.2711.27 percentage points, compared with 28.3928.39 for LLM-JEPA. These results suggest that episodic replay helps the model generate longer symbolic expressions more reliably. Figure 8: ER-JEPA achieves higher prediction accuracy across all six evaluated regex structures with episodic replay. Compared with LLM-JEPA, ER-JEPA improves accuracy by 16.9016.90, 16.0716.07, and 15.4215.42 percentage points for expressions containing quantifiers, intersections, and character classes, respectively. Figure 9: ER-JEPA maintains higher accuracy on longer regular expressions with episodic replay. ER-JEPA leads in all four length groups and shows a smaller decline from the shortest to the longest group than Regular and LLM-JEPA. Lines show mean exact match accuracy over five seeds, and shaded bands indicate one standard deviation. Training gains come without additional inference cost. At inference, ER-JEPA uses the same generation procedure as LLM-JEPA. We remove the episodic memory and replay path after training. Following LLM-JEPA, we use only the fine-tuned language model for inference. Replay therefore adds no overhead to inference time and requires no additional computation or memory during inference. Ablation studies on memory addressing, memory capacity and JEPA objective hyperparameters are provided in Appendix E. 4 Related Work Joint-Embedding Predictive Architectures (JEPAs) learn representations by predicting target embeddings from context. LeCun (2022) proposed JEPA as a framework for learning world models that predict abstract states without reconstructing every input detail. A related approach, data2vec (Baevski et al., 2022), predicts contextualized representations from masked inputs using the same learning method across speech, vision, and language. In vision, I-JEPA (Assran et al., 2023) predicts the representations of target image regions from a context region, learning semantic features without relying on hand-crafted augmentations. V-JEPA (Bardes et al., 2024) extends latent prediction to video and learns representations that support downstream tasks with a frozen encoder. Image World Models (Garrido et al., 2024) broaden the prediction task beyond masked regions to include the effects of photometric transformations. To better understand latent prediction, Littwin et al. (2024) analyze its learning dynamics in deep linear models. Their analysis identifies an implicit bias toward features with high regression coefficients. Recent work also applies JEPA to language and vision-language learning. VL-JEPA (Chen et al., 2026) predicts continuous text embeddings for vision-language tasks and uses a decoder when text output is needed. For language models, LLM-JEPA (Huang et al., 2026) combines next-token prediction with a JEPA loss over paired textual views. This joint objective encourages semantic structure in the learned representations while preserving generative capabilities. Our work builds on LLM-JEPA by adding episodic replay, which reuses past training examples to provide additional supervision for token prediction and representation alignment. 5 Conclusion We propose ER-JEPA, which adds an episodic replay path to LLM-JEPA. The replay path revisits stored training pairs with the current model and provides additional learning signals. At matched training compute, ER-JEPA outperforms LLM-JEPA with all three selection rules. Replay also corrects more errors during training and avoids most large accuracy drops. These results show that revisiting past examples is a simple and effective way to improve LLM-JEPA. We view the current implementation of ER-JEPA as a first step toward introducing episodic replay into the JEPA framework, and future work could further refine and extend this direction. In particular, since the overall training objective is formulated as a linear combination of the new data JEPA loss and the replay loss, the relative weighting between different loss terms, such as the replay weight β, currently needs to be selected through grid search. This tuning process introduces additional computational cost and may motivate future work on adaptive weighting strategies. Moreover, ER-JEPA inherits the need of LLM-JEPA for datasets with paired views, such as text and code. We hope future work will explore more efficient and general data augmentation strategies for constructing informative paired views, thereby broadening the application regime of ER-JEPA. References Assran et al. (2023) M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 15619–15629. Cited by: §1, §4. Assran et al. (2025) M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, et al. V-jepa 2: self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985. Cited by: §1. Baevski et al. (2022) A. Baevski, W. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli Data2vec: a general framework for self-supervised learning in speech, vision and language. In International conference on machine learning, p. 1298–1312. Cited by: §4. Bardes et al. (2024) A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471. Cited by: §1, §4. Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877–1901. Cited by: §1. Chen et al. (2026) D. Chen, M. Shukor, T. Moutakanni, W. Chung, L. Yu, T. Kasarla, A. Bolourchi, Y. LeCun, and P. Fung VL-JEPA: joint embedding predictive architecture for vision-language. In The Fourteenth International Conference on Learning Representations, Cited by: §1, §4. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, J. Schulman, J. Hilton, M. Knight, A. Weller, D. Amodei, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §3.1. Deng et al. (2025) J. Deng, Q. Wu, P. Ju, S. Lin, Y. Liang, and N. Shroff Unlocking the power of rehearsal in continual learning: a theoretical perspective. External Links: 2506.00205, Link Cited by: §1. Garrido et al. (2024) Q. Garrido, M. Assran, N. Ballas, A. Bardes, L. Najman, and Y. LeCun Learning and leveraging world models in visual representation learning. arXiv preprint arXiv:2403.00504. Cited by: §4. Gong et al. (2024) L. Gong, S. Wang, M. Elhoushi, and A. Cheung Evaluation of llms on syntax-aware code fill-in-the-middle tasks. arXiv preprint arXiv:2403.04814. Cited by: §1. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §3.1. Huan et al. (2025) M. Z. Huan, Y. Li, T. Zheng, X. Xu, S. Kim, M. Du, R. Poovendran, G. Neubig, and X. Yue Does math reasoning improve general llm capabilities? understanding transferability of llm reasoning. ArXiv abs/2507.00432. Cited by: §1. Huang et al. (2026) H. Huang, Y. LeCun, and R. Balestriero LLM-JEPA: large language models meet joint embedding predictive architectures. In The Fourteenth International Conference on Learning Representations, Cited by: §1, §4. Huang et al. (2024) J. Huang, L. Cui, A. Wang, C. Yang, X. Liao, L. Song, J. Yao, and J. Su Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. External Links: 2403.01244, Link Cited by: §1. Jin and Rinard (2024) C. Jin and M. Rinard Emergent representations of program semantics in language models trained on programs. In Forty-first International Conference on Machine Learning, Cited by: §1. Kumaran et al. (2016) D. Kumaran, D. Hassabis, and J. L. McClelland What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in cognitive sciences 20 (7), p. 512–534. Cited by: §1. LeCun (2022) Y. LeCun A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review 62 (1), p. 1–62. Cited by: §4. Lee et al. (2019) K. Lee, M. Chang, and K. Toutanova Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez (Eds.), Florence, Italy, p. 6086–6096. External Links: Document Cited by: §3.1. Li et al. (2024) T. Li, W. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica From crowdsourced data to high-quality benchmarks: arena-hard and benchbuilder pipeline. ArXiv abs/2406.11939. Cited by: §1. Li et al. (2025) X. Li, H. Tu, M. Hui, Z. Wang, B. Zhao, J. Xiao, S. Ren, J. Mei, Q. Liu, H. Zheng, Y. Zhou, and C. Xie What if we recaption billions of web images with LLaMA-3?. In Proceedings of the 42nd International Conference on Machine Learning, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 267, p. 35957–35976. Cited by: §1. Littwin et al. (2024) E. Littwin, O. Saremi, M. Advani, V. Thilak, P. Nakkiran, C. Huang, and J. Susskind How jepa avoids noisy features: the implicit bias of deep linear self distillation networks. Advances in Neural Information Processing Systems 37, p. 91300–91336. Cited by: §4. Liu et al. (2024) A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. Cited by: §1. Locascio et al. (2016) N. Locascio, K. Narasimhan, E. DeLeon, N. Kushman, and R. Barzilay Neural generation of regular expressions from natural language with minimal domain knowledge. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, p. 1918–1923. Cited by: §3.1. Pillutla et al. (2021) K. Pillutla, S. Swayamdipta, R. Zellers, J. Thickstun, S. Welleck, Y. Choi, and Z. Harchaoui Mauve: measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems 34, p. 4816–4828. Cited by: §1. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §1. Shi et al. (2024) H. Shi, Z. Xu, H. Wang, W. Qin, W. Wang, Y. Wang, Z. Wang, S. Ebrahimi, and H. Wang Continual learning of large language models: a comprehensive survey. External Links: 2404.16789, Link Cited by: §1. Wang et al. (2024) S. Wang, Z. Chen, C. Shi, C. Shen, and J. Li Mixture of demonstrations for in-context learning. Advances in Neural Information Processing Systems 37, p. 88091–88116. Cited by: §1. Wei et al. (2021) J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. Cited by: §1. Yu et al. (2023) L. Yu, D. Simig, C. Flaherty, A. Aghajanyan, L. Zettlemoyer, and M. Lewis Megabyte: predicting million-byte sequences with multiscale transformers. Advances in Neural Information Processing Systems 36, p. 78808–78823. Cited by: §1. Yu et al. (2018) T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, et al. Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Cited by: §3.1. Zeng et al. (2022) A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia, et al. Glm-130b: an open bilingual pre-trained model. arXiv preprint arXiv:2210.02414. Cited by: §1. Appendix A Detailed Experimental Setup The experimental design follows the LLM-JEPA protocol. This design allows us to isolate the effect of the proposed episodic replay pathway, rather than introducing confounding changes in data, model scale, or training configuration. All experiments are conducted under the same LoRA fine-tuning setting with rank 512. Under this setting, we compare three fine-tuning objectives: the standard supervised fine-tuning objective, the LLM-JEPA objective, and the proposed ER-JEPA objective. For each combination of model and dataset, we use a learning rate of 1e−51e-5 and train for 66 epochs. We adopt this LoRA configuration because the LLM-JEPA study found that it outperforms fine-tuning all model parameters under the same setting. The standard supervised objective, LLM-JEPA, and ER-JEPA all share this same configuration, learning rate, and training schedule. This helps isolate the effects of the training objective and the episodic-replay extension on performance. All experiments are repeated with five fixed random seeds, 82,23,37,84,4\82,23,37,84,4\. We report the mean performance across seeds. Statistical significance is assessed using paired one-tailed t-tests over seed-matched runs. All training runs are conducted on a server equipped with four NVIDIA A800 GPUs. Appendix B How Historical Replay Can Provide Additional Correction Directions LLM-JEPA combines token generation with representation alignment, but both losses directly supervise only the current batch. We give a conditional explanation of how replay changes one gradient-descent step. Replay can supply a descent direction for the selected historical loss that current-batch gradients do not provide. The local limitation of the LLM-JEPA objective. Let ω collect the trainable LoRA coordinates, counting shared parameters once. For a nonempty current batch ℬB, let ℬT_B contain its supervised-token positions and Zℬ=|ℬ|>0Z_B=|T_B|>0. Write πq=softmax(fq(ω)) _q=softmax(f_q(ω)) for the token probabilities, yqy_q for the target token, and ei=pi/‖pi‖2−zia/‖zia‖2e_i=p_i/\|p_i\|_2-z_i^a/\|z_i^a\|_2 for the alignment residual. Equation (3) gives the LLM-JEPA objective Lℬ(ω)=γZℬ∑q∈ℬ−logπq,yq⏟token generation+λ2|ℬ|∑i∈ℬ‖ei‖22⏟representation alignment,L_B(ω)= γZ_B _q _B- _q,y_q_token generation+ λ2|B| _i \|e_i\|_2^2_representation alignment, (7) where γ,λ>0γ,λ>0 are fixed. Fix a parameter state ω0 _0. Hold the examples, sequences, masks, and forward-pass randomness fixed. Repeated occurrences use identical views and forward realizations. For all current and replay examples, assume finite logits, twice continuously differentiable maps, and representation norms above the normalization floor near ω0 _0. At ω0 _0, let Jq=∂fq/∂ωJ_q=∂ f_q/∂ω and Ai=∂ei/∂ωA_i=∂ e_i/∂ω, differentiating both representation branches. The two losses supply the gradient gℬ=∇Lℬ(ω0)=γZℬ∑q∈ℬJq⊤(πq−yq)+λ|ℬ|∑i∈ℬAi⊤ei,g_B=∇ L_B( _0)= γZ_B _q _BJ_q ( _q-e_y_q)+ λ|B| _i A_i e_i, (8) where yqe_y_q is the target-token basis vector. Neither term provides a component along directions in =v:Jqv∈span∀q∈ℬ,Aiv=0∀i∈ℬ.K=\v:\ J_qv \1\\ ∀ q _B, A_iv=0\ ∀ i \. (9) These directions preserve current token probabilities and alignment residuals to first order. They may still change losses on historical examples. Let P be the orthogonal projector onto K. The subspace K may contain only the zero vector. ER-JEPA adds βLℛβ L_R, with fixed β>0β>0, where LℛL_R applies the same joint objective to a nonempty replay multiset with supervised tokens, as in Eq. (6). The analysis also allows replay to overlap ℬB. Occurrences are counted separately and representations are recomputed. Hold the selected multiset ℛR fixed throughout the comparison. Write gℛ=∇Lℛ(ω0)g_R=∇ L_R( _0). Proposition B.1 (Conditional local effect of historical replay). At the fixed state above, Pgℬ=0Pg_B=0. The same holds for a gradient obtained by repeating current examples or reweighting them with fixed weights. Define d=−Pgℛd=-Pg_R. If Pgℛ≠0Pg_R≠ 0, then gℬ⊤d=0,gℛ⊤d=−‖Pgℛ‖22<0.g_B d=0, g_R d=-\|Pg_R\|_2^2<0. (10) For the gradient-descent updates ωC=ω0−ηgℬ _C= _0-η g_B and ωE=ω0−η(gℬ+βgℛ) _E= _0-η(g_B+β g_R), we have P(ωE−ω0)=ηβd,P(ωC−ω0)=0.P( _E- _0)=ηβ d, P( _C- _0)=0. (11) Their replay losses satisfy Lℛ(ωE)−Lℛ(ωC)=−ηβ(‖Pgℛ‖22+‖(I−P)gℛ‖22)+O(η2)L_R( _E)-L_R( _C)=-ηβ (\|Pg_R\|_2^2+\|(I-P)g_R\|_2^2 )+O(η^2) (12) as η→0η→ 0. If gℛ≠0g_R≠ 0, then Lℛ(ωE)<Lℛ(ωC)L_R( _E)<L_R( _C) for all sufficiently small η>0η>0. This loss comparison does not require Pgℛ≠0Pg_R≠ 0. Proof. Let v∈v . For each supervised token, write Jqv=cqJ_qv=c_q1. Then (Jqv)⊤(πq−yq)=cq(1−1)=0,(Aiv)⊤ei=0.(J_qv) ( _q-e_y_q)=c_q(1-1)=0, (A_iv) e_i=0. Thus every token and alignment contribution is orthogonal to K. Equation (8) gives Pgℬ=0Pg_B=0. Repeating these contributions or changing their fixed weights preserves this property. Since P=P⊤=P2P=P =P^2, we have gℬ⊤d=0g_B d=0 and gℛ⊤d=−gℛ⊤Pgℛ=−‖Pgℛ‖22g_R d=-g_R Pg_R=-\|Pg_R\|_2^2. This proves Eq. (10) when Pgℛ≠0Pg_R≠ 0. Applying P to the two updates gives Eq. (11). Taylor expansion at ω0 _0 gives Lℛ(ωC) L_R( _C) =Lℛ(ω0)−ηgℛ⊤gℬ+O(η2), =L_R( _0)-η g_R g_B+O(η^2), Lℛ(ωE) L_R( _E) =Lℛ(ω0)−ηgℛ⊤gℬ−ηβ‖gℛ‖22+O(η2). =L_R( _0)-η g_R g_B-ηβ\|g_R\|_2^2+O(η^2). Subtracting cancels the shared term −ηgℛ⊤gℬ-η g_R g_B. The orthogonal decomposition gℛ=Pgℛ+(I−P)gℛg_R=Pg_R+(I-P)g_R then gives Eq. (12). The smoothness assumptions bound the absolute remainder by Cη2Cη^2 for some fixed C>0C>0 and all sufficiently small η. If gℛ≠0g_R≠ 0, choosing η small enough that Cη<β‖gℛ‖22Cη<β\|g_R\|_2^2 makes the difference strictly negative. ∎ The term ‖Pgℛ‖22\|Pg_R\|_2^2 is the contribution of the additional directions to the first-order loss difference. It is positive only when Pgℛ≠0Pg_R≠ 0. The projector is an analytical device; ER-JEPA uses the full replay gradient. The loss comparison also applies to ordinary replay or any added smooth loss with a nonzero gradient. This analysis characterizes a conditional local effect on the replay loss; its implications for predictive performance are evaluated empirically. Appendix C Additional Empirical Analyses Figure 10: ER-JEPA predictions on examples that LLM-JEPA predicts incorrectly. Rows indicate LLM-JEPA error categories, and columns indicate ER-JEPA prediction categories for the same input and seed. The row total n counts LLM-JEPA errors pooled across five seeds. Each cell shows its count divided by n, expressed as a percentage to one decimal place. Results use Llama-3.2-1B on 2,000 SYNTH test examples per seed at a matched training compute budget of 240.309 PFLOPs. C.1 Regex Structural Subsets Table 2: Definitions of the six regex structural subsets. Each subset contains examples whose target regular expressions include the specified feature. Subset Classification rule Meaning Complement Contains the operator ~. Excludes strings matched by a subexpression. Intersection Contains the operator &. Requires multiple patterns to hold simultaneously. Alternation Contains the operator |. Allows a choice between alternative patterns. Quantifier Contains *, +, ?, or a brace quantifier such as m,n. Specifies repetition counts or optionality. Boundary / anchor Contains , , ^, or $. Constrains matching positions relative to word boundaries or string endpoints. Character class Contains a bracketed character class, such as [a-z]. Specifies a set of allowed characters. Classification follows the regex grammar. Escaped literal symbols and symbols inside character classes are not counted as separate operators. Each example is assigned to every applicable subset, so the subsets may overlap. C.2 Error Correction Across Seeds Table 3 reports the individual results summarized in Table 1. For each seed, we identify Baseline errors among the same 2,000 SYNTH test inputs and evaluate both methods on those inputs. Syntax errors are excluded from the three length-based categories. A prediction counts as corrected only when it exactly matches the target. Each correction rate uses the Baseline error count in its row as the denominator. ER-JEPA uses content replay. Table 3: Correction counts and rates for each seed on the same Baseline errors. Both methods use the same inputs and seed within each row. Corrected subsets may overlap between methods. The counts and rates in Table 1 are summarized across these five seeds. Seed Baseline errors LLM-JEPA corrected Correction rate (%) ER-JEPA corrected Correction rate (%) (a) Over-generation 4 901 448 49.72 807 89.57 23 697 435 62.41 583 83.64 37 705 375 53.19 608 86.24 82 591 344 58.21 455 76.99 84 698 292 41.83 601 86.10 (b) Under-generation 4 1 0 0.00 1 100.00 23 5 1 20.00 2 40.00 37 3 0 0.00 1 33.33 82 2 1 50.00 2 100.00 84 2 1 50.00 0 0.00 (c) Same-length mismatch 4 161 29 18.01 36 22.36 23 156 31 19.87 38 24.36 37 173 41 23.70 46 26.59 82 184 41 22.28 53 28.80 84 168 32 19.05 39 23.21 z C.3 Examples Selected by a Fixed Rule Figure 11: Example predictions from Baseline, LLM-JEPA, and ER-JEPA on the SYNTH dataset. Figure 12: Example predictions from Baseline, LLM-JEPA, and ER-JEPA on the NQ-OPEN dataset. Correct and incorrect answers are shown in dark green and dark red, respectively. For each prediction, only the first semicolon-separated answer is displayed. Figure 11 illustrates Baseline errors corrected by ER-JEPA, grouped by the Baseline error taxonomy. The selection contains exactly two LLM-JEPA-correct cases and seven LLM-JEPA errors; all three methods are compared on the same input and seed for each example. The full evaluation, using 2,000 test inputs for each of five seeds, gives mean accuracies of 53.63%53.63\%, 68.75%68.75\%, and 83.65%83.65\% for Baseline, LLM-JEPA, and ER-JEPA, respectively. Figure 10 reports ER-JEPA predictions on inputs that LLM-JEPA predicts incorrectly. C.4 Individual Seed Accuracy at Matched PFLOPs Figure 13: Raw accuracy at six shared target PFLOPs for seeds 4, 23, 37, and 82. Each panel shows one seed. Seed 84 is shown in Figure 4(b). The panels do not show means or error bars. Figure 13 complements the seed 84 results in Figure 4(b). Across all five seeds, Baseline has a local accuracy decline in four seeds, and LLM-JEPA has a decline in all five. Such declines occur less often with replay. These results suggest that replay mitigates overfitting across individual runs. Appendix D ER-JEPA Implementation Details D.1 Episodic Memory and Retrieval ER-JEPA maintains a fixed-capacity episodic memory ℳtM_t. Each active slot stores a source–target token pair, a sparse address, and memory-management metadata. We denote these entries by mn=(sn,τnu,τna,σn,rn),n∈t,m_n= (s_n, _n^u, _n^a, _n,r_n ), n _t, (13) where tA_t is the set of active slots, sns_n is the address stored at insertion, σn _n is the running JEPA error, and rnr_n is the replay count. Examples in the current mini-batch are excluded from replay. Writing, replacement, and score updates follow Appendix D.2. Content-based retrieval uses the current source representation zju=Encθ(xju)z_j^u=Enc_θ(x_j^u) as a cue. Let Φ be the frozen random projection with sparse Top-K selection defined in Appendix D.3.1. We normalize the cue and stored addresses as s^jcue=Φ(zju)‖Φ(zju)‖2,s^n=sn‖sn‖2. s_j cue= (z_j^u)\| (z_j^u)\|_2, s_n= s_n\|s_n\|_2. (14) For each cue, we retrieve up to κ slots with the highest cosine similarities, t,j=Topmin(κ,Nt)((n,⟨s^jcue,s^n⟩)∣n∈t),Nt=|t|.K_t,j=Top_ (κ,N_t) ( \ (n, s_j cue, s_n ) n _t \ ), N_t=|A_t|. (15) Here TopTop returns slot indices in descending similarity order. We concatenate the lists in batch order, keep the first occurrence of each slot, and retain at most R entries, where R is the replay budget. If fewer than R distinct slots are retrieved, we replay only those slots. Let t=(nt,1,…,nt,Rt)S_t=(n_t,1,…,n_t,R_t) denote the selected slot list, where Rt=|t|≤R_t=|S_t|≤ R. The token pair for replay entry i is (τ~iu,τ~ia)=(τnt,iu,τnt,ia),i=1,…,Rt.( τ_i^u, τ_i^a)=( _n_t,i^u, _n_t,i^a), i=1,…,R_t. (16) The selection rules for all three replay strategies are given in Appendix D.3. D.2 Shared Episodic Memory Each new slot is initialized with σn=d(pn,zna),rn=0, _n=d(p_n,z_n^a), r_n=0, (17) where pnp_n is the predicted target representation and znaz_n^a is the target readout. When the store is full, the slot with the lowest replay-adjusted significance is evicted, n⋆=argminn∈tσn1+rn.n = _n _t _n1+r_n. (18) After a replay forward pass, the significance of each selected slot is updated as σn←(1−η)σn+ηd(pnrep,zna,rep),rn←rn+1,η∈(0,1), _n←(1-η) _n+η\,d(p_n rep,z_n^a, rep), r_n← r_n+1, η∈(0,1), (19) where η is the update rate and pnrep,zna,repp_n rep,z_n^a, rep are recomputed during replay. Unselected slots retain their scores. These rules are shared by all three selection policies. D.3 Replay Selection Policies The content-based, uniform, and hard replay strategies differ only in how replay slots are selected. All use the memory rules in Appendix D.2 and the replay loss in Eq. (6). Let tA_t be the active slots, Nt=|t|N_t=|A_t|, and R the replay budget. A policy π∈content,uniform,hardπ∈\content,uniform,hard\ returns an ordered slot list tπS_t^π, and t=tπS_t=S_t^π. Repeated slots contribute once per occurrence. All policies return an empty selection when Nt=0N_t=0. D.3.1 Content Replay Content replay retrieves stored examples using current source representations zuz^u. Representations are mapped to sparse addresses with a frozen random projection and Top-K selection, WPS W_ PS ∈ℝS×H,[WPS]ℓk∼(0,1/H), ^S× H, [W_ PS]_ k (0,1/H), (20) [Φ(z)]ℓ [ (z)]_ =stop-grad([ℓ∈ℐK(WPSz)](WPSz)ℓ). =stop -grad\! (I[ _K(W_ PSz)](W_ PSz)_ ). H and S>HS>H are the representation and address dimensions, and ℐK(h)I_K(h) contains the indices of the K largest-magnitude entries of h, retaining their sign and magnitude. The projection is fixed throughout training. For current example j, the cue is s^jcue=Φ(zju)/‖Φ(zju)‖2 s_j cue= (z_j^u)/\| (z_j^u)\|_2. Let sns_n be the address stored at insertion and s^n=sn/‖sn‖2 s_n=s_n/\|s_n\|_2. The min(κ,Nt) (κ,N_t) slots with the highest cosine similarity are retrieved, t,j=Topmin(κ,Nt)((n,⟨s^jcue,s^n⟩)∣n∈t),K_t,j=Top_ (κ,N_t) ( \(n, s_j cue, s_n ) n _t \ ), (21) returned in descending score order. Lists from all examples in the batch are concatenated in batch order, deduplicated, and truncated to R entries, t=PrefixR(Unique(t,1‖⋯‖t,|ℬt|)),tcontent=t,C_t=Prefix_R\! (Unique\! (K_t,1\|·s\|K_t,|B_t| ) ), _t content=C_t, (22) where UniqueUnique keeps the first occurrence of each slot and PrefixRPrefix_R keeps at most R entries. The default policy uses the resulting list without repetition. For controls with a fixed replay count, we instead use tcontent=FillR(t)S_t content=Fill_R(C_t), where FillRFill_R repeats a shorter nonempty list until it reaches R entries; an empty list remains empty. Cosine similarity is used only for retrieval and contributes no training loss. D.3.2 Uniform Replay Uniform replay samples stored examples with equal probability, without using cue similarity or significance. When R≤NtR≤ N_t, R slots are sampled without replacement: for any subset ⊆tU _t of size R, Pr(set(tuniform)=∣ℳt)=(NtR)−1,Pr(n∈tuniform∣ℳt)=RNt. \! (set(S_t uniform)=U _t )= N_tR^-1, (n _t uniform _t)= RN_t. (23) When 0<Nt<R0<N_t<R, R slots are drawn independently with replacement, each with probability 1/Nt1/N_t. Uniform replay estimates the mean JEPA error over active memory. Let dn(t)d_n(t) be the error of stored example n under the current model. For a nonempty memory and fixed parameters, [1R∑n∈tuniformdn(t)|ℳt,θ]=1Nt∑n∈tdn(t).E\! [ 1R _n _t uniformd_n(t)\; |\;M_t,θ ]= 1N_t _n _td_n(t). (24) D.3.3 Hard Replay Hard replay selects the min(R,Nt) (R,N_t) slots with the largest significance σn _n (Appendix D.2), ℋt=Topmin(R,Nt)((n,σn)∣n∈t),thard=FillR(ℋt),H_t=Top_ (R,N_t) ( \(n, _n) n _t \ ), _t hard=Fill_R(H_t), (25) returned in descending score order, with FillRFill_R as in Appendix D.3.1. A large JEPA error indicates poor representation alignment, not necessarily an incorrect generated answer. Appendix E Ablation Studies Table 4: Some regular expressions generated by Llama-3.2-1B-Instruct after fine-tuning with ℒLLML_ LLM, ℒLLM−JEPAL_ LLM-JEPA, and ℒER−JEPAL_ ER-JEPA losses. Color code: wrong, extra, missing. Model / target Regular expression Input: lines not having the string “dog” followed by a number, 3 or more times Ground truth ~((dog.*[0-9].*)3,) ℒLLML_ LLM ~((dog.*[0-9].*)3,) ℒLLM−JEPAL_ LLM-JEPA ~((dog.*[0-9].*)3,) ℒER−JEPAL_ ER-JEPA ~((dog.*[0-9].*)3,) Input: lines containing ending with a vowel, zero or more times Ground truth .*(.*)(([AEIOUaeiou])*).* ℒLLML_ LLM .*(.*)(([AEIOUaeiou])*).*.*.* ℒLLM−JEPAL_ LLM-JEPA .*(.*)(([AEIOUaeiou])*).*.* ℒER−JEPAL_ ER-JEPA .*(.*)(([AEIOUaeiou])*).* Input: lines with a number or a character before a vowel Ground truth (([0-9])|(.)).*([AEIOUaeiou]).* ℒLLML_ LLM (([0-9])|(.)).*([AEIOUaeiou]).*.* ℒLLM−JEPAL_ LLM-JEPA (([0-9])|(.)).*([AEIOUaeiou]).* ℒER−JEPAL_ ER-JEPA (([0-9])|(.)).*([AEIOUaeiou]).* Input: lines ending with containing the string “dog”, 7 or more times Ground truth ((.*)(.*dog.*))7, ℒLLML_ LLM ((.*)(.*(dog.*))7,.*)* ℒLLM−JEPAL_ LLM-JEPA ((.*)((.*dog.*))7,) ℒER−JEPAL_ ER-JEPA (.*)((.*dog.*)7,)) E.1 Sparse and Dense Memory Addressing We compare content replay using sparse addresses and dense source representations. Sparse addresses are formed by a frozen random projection and Top-K selection. Both variants use cosine similarity, with all other settings fixed. Table 5 reports mean accuracies of 86.37%86.37\% for sparse retrieval and 86.21%86.21\% for dense retrieval. The two strategies achieve similar mean accuracy, with a difference of 0.160.16 percentage points. Table 5: Sparse and dense memory addressing for content replay. All other training settings are fixed. Results report mean accuracy, standard deviation, minimum, and maximum over five seeds. Method Accuracy (%) ↑ Min Max ER-JEPA (Sparse Retrieval) 86.37±0.3586.37± 0.35 85.7585.75 86.6086.60 Dense Retrieval 86.21±0.3786.21± 0.37 85.8585.85 86.7086.70 E.2 Hyperparameter Ablations Memory capacity. Table 6 reports accuracy after four epochs, and Figure 14 shows the learning curves. Increasing memory capacity from 1010 to 10410^4 raises accuracy from 84.22%84.22\% to 84.98%84.98\%. Table 6: Ablation on episodic memory capacity ℳM for Meta-Llama-3.2-1B-Instruct on NL-RX-SYNTH. Values at epoch 4 are mean ± standard deviation over five seeds. The observed training costs are not compute matched. Memory capacity ℳM Accuracy (%) ↑ PFLOPs Time (min) Tokens (M) 1010 84.22±0.7584.22± 0.75 181.536±0.211181.536± 0.211 10.48±0.0710.48± 0.07 8.581±0.1668.581± 0.166 10210^2 84.32±0.6584.32± 0.65 219.605±0.533219.605± 0.533 11.62±0.2211.62± 0.22 9.427±0.1759.427± 0.175 10310^3 84.76±0.5284.76± 0.52 219.766±0.439219.766± 0.439 16.74±1.3316.74± 1.33 10.969±0.09710.969± 0.097 10410^4 84.98±0.3584.98± 0.35 219.667±0.393219.667± 0.393 22.23±0.1722.23± 0.17 11.252±0.01011.252± 0.010 Objective hyperparameters. We also vary the JEPA loss weight λ and predictor depth k, following the LLM-JEPA design. Figure 15 reports the results. Figure 14: Accuracy across four training epochs for four episodic memory capacities. Points show means over five seeds. Error bars show one sample standard deviation. These runs are not compute matched. Figure 15: Ablation on the JEPA hyperparameters λ and k. Each entry reports accuracy (%), with the best result in bold.