Paper deep dive
ERASE: EaRly bAckpropagation SchEdule for Faster Training of Modern Recommendation Systems
Ergan Shang, Flavio Sales Truzzi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/20/2026, 4:33:55 AM
Summary
The paper introduces ERASE, a scheduling primitive that repurposes the detachment mechanism from the Forward-Forward (FF) algorithm to enable early backward pass execution in deep learning training. By detaching block outputs, ERASE removes downstream gradient dependencies, allowing backward passes to run concurrently with subsequent forward passes on separate CUDA streams. This technique improves training throughput by up to 9.51% on large-scale click-through-rate models while maintaining model quality, addressing underutilization of modern accelerators in lightweight proxy models.
Entities (8)
Relation Signals (6)
ERASE → improves → Training Throughput
confidence 95% · detaching six dense subarchitectures improves training throughput by up to 9.51%
ERASE → uses → Forward-Forward
confidence 95% · We reinterpret the detachment mechanism of Forward-Forward (FF) as a scheduling primitive
ERASE → appliedto → CTR Model
confidence 90% · On a large-scale click-through-rate model... ERASE launches each detached subarchitecture's backward pass
ERASE → utilizes → CUDA streams
confidence 90% · ERASE launches each detached subgraph's backward pass early on a separate CUDA stream
CUDA Graphs → usedby → ERASE
confidence 85% · Capturing the affected subgraphs as CUDA Graphs fixes the launch order
NanoChat → usedfor → ERASE
confidence 80% · We then evaluate ERASE on NanoChat... to demonstrate overlap
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Lightweight proxy models enable rapid experimentation without repeatedly training frontier-scale systems, but their small kernels often leave modern accelerators underutilized. Conventional training compounds this inefficiency by scheduling the forward and backward passes as disjoint phases, so spare capacity in one cannot be filled by work from the other. We reinterpret the detachment mechanism of Forward-Forward (FF) as a scheduling primitive: given a local objective, detaching a block's output removes downstream gradient dependencies, making its backward pass ready when its forward pass finishes. ERASE launches each detached subgraph's backward pass early on a separate CUDA stream, overlapping it with subsequent forward work. Execution trace on a lightweight transformer demonstrates this overlap and its limit: a kernel that saturates the device leaves no capacity for concurrency. On a large-scale click-through-rate model, detaching six dense subarchitectures improves training throughput by up to $9.51\%$ while keeping normalized entropy close to the baseline.
Tags
Links
- Source: https://arxiv.org/abs/2608.18469v1
- Canonical: https://arxiv.org/abs/2608.18469v1
Trouble viewing inline? Open PDF directly →
Full Text
19,037 characters extracted from source content.
Expand or collapse full text
ERASE: EaRly bAckpropagation SchEdule for Faster Training of Modern Recommendation Systems Ergan Shang Affiliation: Carnegie Mellon University Email: eshang@andrew.cmu.edu Flavio Sales Truzzi Affiliation: Meta Inc. Email: ftruzzi@meta.com Abstract Lightweight proxy models enable rapid experimentation without repeatedly training frontier-scale systems, but their small kernels often leave modern accelerators underutilized. Conventional training compounds this inefficiency by scheduling the forward and backward passes as disjoint phases, so spare capacity in one cannot be filled by work from the other. We reinterpret the detachment mechanism of Forward-Forward (F) as a scheduling primitive: given a local objective, detaching a block’s output removes downstream gradient dependencies, making its backward pass ready when its forward pass finishes. ERASE launches each detached subgraph’s backward pass early on a separate CUDA stream, overlapping it with subsequent forward work. Execution trace on a lightweight transformer demonstrates this overlap and its limit: a kernel that saturates the device leaves no capacity for concurrency. On a large-scale click-through-rate model, detaching six dense subarchitectures improves training throughput by up to 9.51%9.51\% while keeping normalized entropy close to the baseline. 1 Introduction Backpropagation [21, 22] remains the standard method for training modern neural networks across vision and language [4, 26], agent learning and evaluation [18, 14, 28], political and social network inference [16, 24], and the natural sciences [20, 27]. Among its most computationally demanding applications are large-scale recommendation and ranking systems [2, 3], which are continuously retrained on data streams that can outpace a single training job [1, 12]. Training throughput therefore directly affects model freshness and hardware resource burden [8]. For these workloads, throughput depends on accelerator utilization as well as peak arithmetic performance [29]. Collective communication, data movement, and scheduling dependencies can leave capacity unused [17, 8]. Conventional reverse-mode differentiation traverses the graph in reverse topological order, beginning the backward pass only after the entire forward pass has completed. The two phases therefore occupy disjoint intervals, preventing either from using spare capacity in the other. The same utilization pressure appears at the opposite end of the scale: frontier-scale models require too many resources for rapid, repeated experimentation, so automated research relies on lightweight proxies such as NanoChat [11], which retains a useful experimental signal at orders-of-magnitude smaller scale. The accelerator, however, does not shrink with the model. Small GEMMs may expose too little parallelism to approach peak arithmetic throughput, leaving capacity unused even without launch gaps; CPU scheduling computational overhead can reduce utilization further when gaps occur. We therefore ask how to improve model FLOPs utilization (MFU) when the model is, by design, too small to saturate the device. Meanwhile, the Forward-Forward (F) algorithm trains each block against a local objective and passes a detached activation to its successor [9]. This partitions the end-to-end graph into block-local subgraphs: gradients do not cross block boundaries, and activations need not be retained for a global backward pass. Importantly for scheduling, a block’s backward pass becomes ready once its forward pass and local objective are complete, without waiting for the terminal loss or later blocks. It can therefore overlap with subsequent forward passes, enabling work that the conventional forward-then-backward schedule forgoes. We turn this observation into ERASE (Section 2), which dispatches each detached subgraph’s backward pass as soon as its forward pass returns. Experiments (Section 3) demonstrate the intended overlap and its saturation limit on NanoChat, then measure the throughput improvement on a ranking model. Section 4 analyzes the staggering introduced by asynchronous dispatch. 1.1 Related Work Backpropagation-free learning. Backpropagation-free methods replace the global backward pass with block-local targets. NoProp and DiffusionBlocks attach an auxiliary denoising variable to each block [13, 25], borrowing their training signal from diffusion models [10, 23]; this permits detached activations and block-local updates. Earlier work on Forward-Forward and layer-wise learning likewise shows that local objectives can train competitive models without gradients crossing block boundaries [9, 15]. CUDA Graphs and execution-level acceleration. CUDA Graphs capture and replay kernel sequences, reducing per-kernel CPU launch computational overhead in small-kernel workloads [19, 6]; recent compiler work makes capture more robust in PyTorch [7]. ERASE combines this execution mechanism with F-style detachment, which creates independent pieces while preserving the usual loss and exact gradients within each piece. Early backward launches each piece on a separate CUDA stream as soon as its forward pass returns, while CUDA Graphs keep the asynchronous launch order consistent across ranks. 2 Backprop Free Algorithm 2.1 The Forward-Forward algorithm Data dependency shutdown. For an input embedding ∈ℝp x ^p, consider a network of B blocks with 0= z_0= x, activations b z_b, and parameters b θ_b. F [9] computes b=fb(sg[b−1];b),b=1,…,B, z_b\;=\;f_b (sg[ z_b-1]; θ_b ), b=1,…,B, (1) where sg[⋅]sg[·] is the stop-gradient operator: it is the identity on the forward pass and has zero Jacobian on the backward pass. A local loss on the undetached b z_b updates b θ_b alone and may, for example, apply binary cross-entropy to the goodness score Gb=∑jzbj2G_b= _jz_bj^2. Consequently, ∇b _ θ_b depends only on the subgraph between adjacent cut points and is ready once the block’s forward pass and local loss are complete, before the terminal loss is evaluated. ERASE exploits this scheduling consequence of detachment; Figure 1 illustrates the resulting graph. Figure 1: Schematic of F-style detachment. Each block receives a detached copy of the previous block’s activation, and contributes its own local loss. No gradient crosses a block boundary. 2.2 Early backward in ERASE: scheduling, streams, and determinism Detachment exposes independence but does not change the schedule: a single backward call on the aggregate objective ℒ=ℒmain+∑iλiℓiL\;=\;L_main\;+\; _i _i\, _i (2) still runs at the end of the step. ERASE instead launches each detached subgraph’s backward pass as soon as its forward pass returns, accumulating its parameter gradients while leaving the undetached remainder to the usual end-of-step backward. Because backward operations otherwise serialize with subsequent forward work on the default CUDA stream, ERASE uses separate streams and events to enforce the remaining dependencies. This allows the backward pass of block b to overlap with the forward pass of block b+1b+1. CPU dispatch introduces another tradeoff. Main-thread early backward blocks further dispatch until its autograd call is enqueued. A ThreadPoolExecutor avoids this stall but can vary the kernel and collective order across ranks, creating stragglers that we call staggering. Capturing the affected subgraphs as CUDA Graphs fixes the launch order. Section 4 compares the blocking and non-blocking variants. 3 Experiments As a small-scale sanity check via one A100 GPU, F-style detachment on a 3030-layer MLP using MNIST dataset ([5]) reduced compute-matched backward time by 58%58\% (23.523.5 to 9.99.9 ms) and total batch time by 30%30\% (41.241.2 to 28.828.8 ms). We then evaluate ERASE on NanoChat [11], using one A100 GPU to demonstrate overlap, and on a lightweight version of a large-scale recommendation model, using eight H100 GPUs to measure throughput. 3.1 NanoChat To verify the intended overlap before experiments on a recommendation model, we partition NanoChat’s multi-head-attention stack into three subgraphs using two detachment points. Figure 2 shows a single-batch execution trace. Figure 2: NanoChat single-batch trace with two detachment points and three streams. Backward work from one subgraph runs alongside forward work from the next. The trace demonstrates the execution pattern rather than measuring speedup: one subgraph’s backward work overlaps the next subgraph’s forward work, as Section 2.2 predicts. The exception is the fused multi-head attention backward kernel (fm in Figure 2), which runs alone. This is a resource limit rather than a dependency: by fusing the attention matrix multiplications with softmax and avoiding materialization of the sequence-by-sequence attention matrix, the kernel occupies every streaming multiprocessor and leaves no capacity for concurrent work. This limit reinforces the premise of Section 1: overlap helps only where the hardware is not already saturated. Reducing hidden width, depth, or batch size shrinks kernels without shrinking the accelerator, leaving resources that small kernels can share but device-filling kernels cannot. Lightweight proxies contain mostly the former, making them natural targets for early backward. Section 3.2 quantifies the benefit on the recommendation model. 3.2 CTR Model 3.2.1 Setup The remaining experiments use a click-through-rate (CTR) model with six detached subarchitectures, four of which are assigned separate CUDA streams alongside the main stream. Throughput is the number of training examples processed per second (QPS), summarized by the post-warm-up p90p90 of per-step samples (higher is better); unlike latency p90p90, this is the fast end of the distribution. Quality is normalized entropy (NE), the model’s cross-entropy divided by that of a constant predictor; lower is better. 3.2.2 Early backward on the CTR model ERASE launches each detached subarchitecture’s backward pass from a ThreadPoolExecutor as soon as its forward pass completes. Because the worker thread can introduce cross-rank launch-order nondeterminism, we capture a subset of the subarchitectures as CUDA Graphs to fix their order and reduce CPU dispatch computational overhead. This non-blocking configuration (row 1 of Table 1) improves throughput by 7.38%7.38\% with an NE gap of approximately 1.38%1.38\% (Figure 3). Section 4 examines its FUP=False setting and compares it with blocking dispatch. Figure 3: NE gap between the non-blocking early-backward variant with CUDA Graphs (red curve) and the baseline (blue curve), approximately 1.38%1.38\%. 4 Caveats Table 1 reports dispatch and FUP (find_unused_parameters) ablations. Blocking launches early backward from the main thread and stalls CPU dispatch during autograd enqueue; FUP controls which parameters participate in gradient synchronization. Table 1: Throughput on the CTR model, p90p90 queries per second # Configuration QPS (p90p90) Gain vs. baseline – baseline 184,535.51 +0.00%+0.00\% 3 blocking, FUP=False 185,223.71 +0.37%+0.37\% 2 blocking, FUP=True 194,244.72 +5.26%+5.26\% 1 non-blocking + CUDA Graphs, FUP=False 198,145.28 +7.38%+7.38\% 0 blocking, FUP=True† 202,078.43 +9.51%+9.51\% Blocking is deterministic but stalls the CPU: FUP=False gains only 0.37%0.37\% (row 3), whereas FUP=True gains 5.26%5.26\% (row 2) by reducing the parameters in the final aggregate backward. CUDA Graphs instead recover a 7.38%7.38\% gain with non-blocking dispatch and FUP=False (row 1). Together, these results suggest that deterministic, rank-synchronized collective order is useful and that either FUP or CUDA Graphs can provide it. 5 Conclusion ERASE repurposes F-style detachment as a scheduling primitive: cutting inter-block dependencies makes each backward pass ready after its forward pass, enabling separate-stream overlap while preserving exact within-subgraph gradients without F’s goodness objective. On a CTR model, ERASE improves p90p90 QPS by 55–9%9\% with a small NE gap under deterministic cross-rank collective order. References [1] F. Borisyuk, M. Zhou, Q. Song, S. Zhu, B. Tiwana, G. Parameswaran, S. Dangi, L. Hertel, Q. C. Xiao, X. Hou, et al. (2024) LiRank: industrial large scale ranking models at linkedin. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 4804–4815. Cited by: §1. [2] H. Cheng, L. Koc, J. Harmsen, T. Shaked, T. Chandra, H. Aradhye, G. Anderson, G. Corrado, W. Chai, M. Ispir, et al. (2016) Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, p. 7–10. Cited by: §1. [3] P. Covington, J. Adams, and E. Sargin (2016) Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, p. 191–198. Cited by: §1. [4] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) Imagenet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, p. 248–255. Cited by: §1. [5] L. Deng (2012) The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine 29 (6), p. 141–142. Cited by: §3. [6] J. Ekelund, S. Markidis, and I. Peng (2025) Boosting performance of iterative applications on gpus: kernel batching with cuda graphs. In 2025 33rd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing (PDP), p. 70–77. Cited by: §1.1. [7] A. Ghosh, A. Nayak, A. Panwar, and A. Basu (2025) Pygraph: robust compiler support for cuda graphs in pytorch. arXiv preprint arXiv:2503.19779. Cited by: §1.1. [8] V. Gupta, D. Choudhary, P. Tang, X. Wei, X. Wang, Y. Huang, A. Kejariwal, K. Ramchandran, and M. W. Mahoney (2021) Training recommender systems at scale: communication-efficient model and data parallelism. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, p. 2928–2936. Cited by: §1, §1. [9] G. Hinton (2022) The forward-forward algorithm: some preliminary investigations. arXiv preprint arXiv:2212.13345. Cited by: §1.1, §1, §2.1. [10] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. Advances in neural information processing systems 33, p. 6840–6851. Cited by: §1.1. [11] Autoresearch: ai agents running research on single-gpu nanochat training automatically Note: https://github.com/karpathy/autoresearch Cited by: §1, §3. [12] E. Kharitonov (2019) Federated online learning to rank with evolution strategies. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, p. 249–257. Cited by: §1. [13] Q. Li, Y. W. Teh, and R. Pascanu (2026) Noprop: training neural networks without back-propagation or forward-propagation. In Conference on Lifelong Learning Agents, p. 525–544. Cited by: §1.1. [14] Z. Li, Z. Li, Y. Shi, R. Wang, J. Yang, Z. Liu, X. Wu, A. Li, Y. Yu, N. Liu, L. Sun, H. Mi, and LeoweiLiang (2026) Long-horizon-terminal-bench: testing the limits of agents on long-horizon terminal tasks with dense reward-based grading. External Links: 2607.08964, Link Cited by: §1. [15] G. Lorberbom, I. Gat, Y. Adi, A. Schwing, and T. Hazan (2024) Layer collaboration in the forward-forward algorithm. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38, p. 14141–14148. Cited by: §1.1. [16] H. Lyu and J. Luo (2022) Understanding political polarization via jointly modeling users, connections and multimodal contents on heterogeneous graphs. In Proceedings of the 30th ACM international conference on multimedia, p. 4072–4082. Cited by: §1. [17] D. Mudigere, Y. Hao, J. Huang, Z. Jia, A. Tulloch, S. Sridharan, X. Liu, M. Ozdal, J. Nie, J. Park, et al. (2022) Software-hardware co-design for fast and scalable training of deep learning recommendation models. In Proceedings of the 49th Annual International Symposium on Computer Architecture, p. 993–1011. Cited by: §1. [18] L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al. (2025) Humanity’s last exam. arXiv preprint arXiv:2501.14249. Cited by: §1. [19] PyTorch Team (2021) Accelerating PyTorch with CUDA graphs. Note: PyTorch Blog, https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/Accessed 2026-08-03 Cited by: §1.1. [20] Y. Roohani, K. Huang, and J. Leskovec (2024) Predicting transcriptional outcomes of novel multigene perturbations with gears. Nature biotechnology 42 (6), p. 927–935. Cited by: §1. [21] D. E. Rumelhart, G. E. Hinton, and R. J. Williams (1986) Learning representations by back-propagating errors. nature 323 (6088), p. 533–536. Cited by: §1. [22] D. E. Rumelhart, G. E. Hinton, and R. J. Williams (1988) (1986) de rumelhart, ge hinton, and rj williams, learning internal representations by error propagation, parallel distributed processing: explorations in the microstructures of cognition, vol. i, de rumelhart and jl mcclelland (eds.) cambridge, ma: mit press, p. 318-362. Cited by: §1. [23] E. Shang, Y. Wei, and K. Roeder (2025) Predicting the unseen: a diffusion-based debiasing framework for transcriptional response prediction at single-cell resolution. Proceedings of the National Academy of Sciences 122 (52), p. e2525268122. Cited by: §1.1. [24] E. Shang, Y. Zhang, and W. Tang (2026) Inference for balance in dynamic signed networks. arXiv preprint arXiv:2606.08786. Cited by: §1. [25] M. Shing, M. Koyama, and T. Akiba (2026) Diffusionblocks: block-wise neural network training via diffusion interpretation. In International Conference on Learning Representations, Vol. 2026, p. 95053–95074. Cited by: §1.1. [26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Advances in neural information processing systems 30. Cited by: §1. [27] T. Zhang, E. Shang, and K. Roeder (2025) Genetic convergence analysis of crispr perturbations deciphers gene functional similarity. bioRxiv. Cited by: §1. [28] J. Zhou, Z. Sun, B. Li, J. Zhou, Y. Pan, H. Wang, H. Ren, X. Jia, X. Zhou, X. Cao, Y. Chen, Y. Feng, J. Wu, C. Zhang, S. Chen, H. Xue, C. You, H. Wang, K. Wu, P. Gao, J. Wu, W. Li, E. Shang, Q. Zheng, J. Zhou, R. Jia, Y. Xu, H. Zhang, X. Ma, Z. Cheng, Y. Hao, L. Mai, X. Ji, W. Zhang, Z. Chen, Y. Huang, C. Wang, W. Hua, Y. Hao, Y. Zhai, Z. Zhao, and J. Xie (2026) ASI-bench: at the dawn of artificial superintelligence. External Links: 2608.17271, Link Cited by: §1. [29] J. Zhu, Z. Fan, X. Zhu, Y. Jiang, H. Wang, X. Han, H. Ding, X. Wang, W. Zhao, Z. Gong, et al. (2025) Rankmixer: scaling up ranking models in industrial recommenders. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, p. 6309–6316. Cited by: §1.