Paper deep dive
NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale
Songlin Jiang, Zhiyu Li, Terry Kong, Yu Yao, Youngeun Kwon, Bernard Nguyen, Ashwath Aithal, Mario Di Francesco
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Agentic reinforcement learning (RL) disaggregates training from rollout, so each policy update must reach the rollout clusters before the next batch. Transferring a full 1T checkpoint for such weight synchronization (refit) takes 87.5 min between two AWS regions. Measurements of BF16 training show that about 1% of weights change their stored values per step. Recent systems exploit this sparsity but fall short on placement, exactness, or efficiency: they reimplement placement rules, assemble full tensors, rebuild values arithmetically, or use a cross-cluster collective, and none fully recovers from mid-refit failures. We present NeMo-DCR (Delta-Compressed Refit), which sends only changes yet is bit-exact: receivers obtain the same parameter and buffer bits as a dense refit. For placement, fixed affine mappings project changes from training shards into the checkpoint's canonical coordinates, residual conversion covers the other changes, and the serving runtime's native loader places all changes in receiver storage. For exactness, compressible XOR masks carry affine changes whose projection and loader preserve stored bits, and overwrites carry the others. Receivers apply both in place, retries overwrite partial writes, and a joint commit binds the policy to the baseline for the next delta. For efficiency, object storage or a relay tree streams payloads during delta construction, without a cross-cluster collective. Even at 3% and 5% change rates, NeMo-DCR refits of 30B-1T models are 12-40$\times$ faster than a transport-only full-checkpoint reference. A 1T relay-tree refit at 3% takes 150 s instead of 87.5 min, making refits practical for cross-cluster agentic RL at trillion-parameter scale.
Tags
Links
- Source: https://arxiv.org/abs/2610.08430v1
- Canonical: https://arxiv.org/abs/2610.08430v1
Trouble viewing inline? Open PDF directly →
Full Text
131,923 characters extracted from source content.
Expand or collapse full text
2026-10-07 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale Songlin Jiang 1,2* , Zhiyu Li 2 , Terry Kong 2 , Yu Yao 2 , Youngeun Kwon 2 , Bernard Nguyen 2 , Ashwath Aithal 2 and Mario Di Francesco 1 1 Aalto University, 2 NVIDIA Abstract. Agentic reinforcement learning (RL) disaggregates training from rollout, so each policy update must reach the rollout clusters before the next batch. Transferring a full 1T checkpoint for such weight synchronization (refit) takes 87.5 min between two AWS regions. Measurements of BF16 training show that about 1% of weights change their stored values per step. Recent systems exploit this sparsity but fall short on placement, exactness, or efficiency: they reimplement placement rules, assemble full tensors, rebuild values arithmetically, or use a cross-cluster collective, and none fully recovers from mid-refit failures. We present NeMo-DCR (Delta-Compressed Refit), which sends only changes yet is bit-exact: receivers obtain the same parameter and buffer bits as a dense refit. For placement, fixed affine mappings project changes from training shards into the checkpoint’s canonical coordinates, residual conversion covers the other changes, and the serving runtime’s native loader places all changes in receiver storage. For exactness, compressible XOR masks carry affine changes whose projection and loader preserve stored bits, and overwrites carry the others. Receivers apply both in place, retries overwrite partial writes, and a joint commit binds the policy to the baseline for the next delta. For efficiency, object storage or a relay tree streams payloads during delta construction, without a cross-cluster collective. Even at 3% and 5% change rates, NeMo-DCR refits of 30B–1T models are 12–40×faster than a transport-only full-checkpoint reference. A 1T relay-tree refit at 3% takes 150 s instead of 87.5 min, making refits practical for cross-cluster agentic RL at trillion-parameter scale. _ Documentation: NeMo RL weight refit guide§ Code: NeMo RL pull request #2444 1. Introduction Agentic reinforcement learning (RL) spends over 70% of its wall-clock time in rollout, so rollout is disag- gregated from training and runs on serving clusters with suitable hardware and parallel layouts [6,7]. Before the next rollout batch, a refit must deliver the updated policy to the receivers, the rollout ranks that generate responses with a serving runtime [15]. Disaggregation also lets rollout run on idle serving GPUs [6] or in other datacenters [40,41], where every refit must cross a wide-area network. A cross-cluster transfer of a 120B model’s 247.2 GB checkpoint through object storage takes 750 s, and at the 1T scale of models like Kimi K2 [14], such a transfer takes 87.5 min (Figure 1). Yet in the six models of Figure 2, only 0.6–1.2% of training-side source elements change their bfloat16 (BF16) values per step. We call this fraction the element change rate. With only 8 significant bits, BF16 rounds most optimizer updates away. Other reports give element change rates of 1.5–2.0% in Qwen3 RL [26] and av- erage deltas of 1.98% of a 1,024 GiB checkpoint [5]. A full-checkpoint transfer thus spends 98–99% of its bytes on unchanged elements. Training cluster Rollout cluster Transport-only full-checkpoint reference NeMo-DCR (relay tree, 3%) 051015 Time (min) 120B 1T 750 s 22.6 s (33×) 150 s (35×) 8590 87.5 min Figure 1|NeMo-DCR relay-tree refit at a 3% element change rate versus the transport-only full-checkpoint reference: 22.6 s instead of 750 s at 120B and 150 s instead of 87.5 min at 1T (§ 8.5 and 8.6). Qwen2.5 0.5B Qwen2.5 1.5B Qwen2.5 7B Llama-3.2 3B Gemma-3 4B Qwen3 30B-A3B 0.0 0.5 1.0 1.5 Element change rate per step (%) 0.7 0.6 0.90.9 1.1 1.2 Figure 2|Element change rate per optimizer step, averaged over each model’s first five steps with the setup of Appendix E: at most 1.2% in every model. * Work conducted during an internship at NVIDIA. © 2026 NVIDIA. All rights reserved.1 arXiv:2610.08430v1 [cs.DC] 6 Oct 2026 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale A refit that sends only the changed 1% faces three challenges: placement, exactness, and efficiency. First, placement: sparse source changes do not tell receivers where or what to write. A training-shard location does not map directly to a receiver’s stor- age location: refitting may gather, split, or permute tensors between training and rollout layouts [26]. A change to a shared scale factor can also alter con- verted values even when other source values stay un- changed [25]. Some systems reimplement the serving runtime’s placement rules [6,26], and others extract a delta after full-tensor assembly and conversion, which handles both problems but pays for both steps [11]. Second, exactness: a smaller transfer must still de- liver the exact updated bits, because rollout–training mismatch is known to destabilize RL [28]. Arithmetic reconstruction can introduce floating-point rounding errors [26], while absolute overwrites avoid round- ing [19] but resend unchanged bits within changed values. Failures also threaten exactness: applying updates in place avoids a separate receiver-side base- line copy, but an interrupted refit can leave a mix- ture of old and updated values that matches neither policy version. Because a delta applies only to the version it was computed from [19], recovery must complete the interrupted refit before rollout resumes and keep receivers aligned with the source baseline that the next refit compares against. Third, efficiency: delivery must overlap the other refit stages without a cross-cluster collective. The next roll- out batch waits for the complete update, so without overlap, delta construction and application can offset the transfer savings. A cross-cluster collective cou- ples receiver membership to the training cluster and introduces timeouts upon failure [23]. Some deploy- ments share weights only through object storage [3], and others have direct links between clusters [40], so delivery must support both. Existing systems meet these challenges only in part: they either send every weight or lack exactness, native placement, decoupled delivery, or recovery, so none meets all five requirements in Table 1. We present NeMo-DCR, a system for bit-exact delta refit that meets all five: it sends only changes, yet every re- ceiver obtains the same parameter and buffer bits as a dense refit that sends every weight. The key idea is to describe changes in canonical coordinates, the tensor names and indices of the Hugging Face check- point format that both training and serving already support. Fixed affine index mappings project changes directly into these coordinates and cover over 96% of the weight bytes in the mixture-of-experts (MoE) checkpoints. The serving runtime’s native loader then maps these coordinates to receiver storage, much as a page table maps virtual to physical addresses. Build- ing on canonical coordinates, this paper makes four contributions that address these three challenges: 1. Direct projection and residualconversion. One training rank per shard acts as its owner, which detects changes in stored values and projects affine changes into canonical coordinates. Residual conversion compares converted tensors with a distributed residual baseline to cover the other changes, including those from shared scale factors. Avoiding full-tensor assembly and con- version for affine changes makes Qwen3 delta construction1.08–1.16×faster at 3% and 5% element change rates (§ 4 and 8.3). 2. Mixed XOR/overwrite encoding with com- pression. XOR encodes non-overlapping affine changes whose projection and loader preserve stored bits, while overwrites encode all oth- ers. Both are exact, and mixed encoding cuts Qwen3 payload bytes by 38–40% versus over- writes (§ 5, 8.2, and 8.3). 3.Recoverable in-place refits. Receivers leave placement to the native loader and apply updates in place, keeping no separate receiver-side base- line copy. Retries overwrite the complete changed set, repairing partial writes. After all current re- ceivers apply the updates, a joint commit binds the policy to its source baseline, which owners then update in place. Proposition 1 guarantees bit-exactness. With receivers killed mid-refit, both NeMo-DCR transports follow the mean re- ward and KL trajectories of dense NCCL refits (§ 6, 8.1, and 8.4). 4.Pipelined delivery without a cross-cluster collective. Payloads travel over object stor- age or a relay tree as the delta is built and, in synchronous RL, applied. In asynchronous RL, rollout continues serving requests until the delta has been transferred and staged, then pauses while receivers apply it. At 3% and 5%, relay- tree transport overlaps delta construction, and the transport lower bound accounts for 77–94% of the refit latency (§ 7 and 8.7). Together, these contributions make NeMo-DCR refits of 30B–1T models 12–40×faster than the transport- only full-checkpoint reference (§ 8.5 and 8.6), even in stress cases whose 3% and 5% element change rates exceed every rate in Figure 2. The same code handles Qwen3 and hybrid Mamba-Transformer Nemotron models without model-specific logic. At 1T and 3%, the relay tree takes 150 s instead of 87.5 min (Fig- ure 1), making refits practical for cross-cluster agentic RL at trillion-parameter scale. © 2026 NVIDIA. All rights reserved.2 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale Table 1 | Refit approaches scored against the five requirements of § 2.3. Only NeMo-DCR meets all five. ApproachSystems Bit-exact Delta Native Decoupled Recovery G1G2G3G4G5 ADense collective refit NCCL-Reshard [23]✓–(✓)–(✓) checkpoint-engine [20]✓–✓–(✓) BFull-checkpoint transferTensorHub [40]✓–(✓)✓(✓) C Arithmetic reconstruction ROSE [6]–✓–✓– AReaL-DTE [26]–✓(✓)✓(✓) AuroraRL [30]–✓–✓(✓) D Absolute overwrites PULSE [19](✓)✓–✓(✓) SparseRL-Sync [11](✓)✓– verl [36](✓)✓– Bit-exact delta refitNeMo-DCR✓ ✓ means met, (✓) partly met, and – not met or not described in the cited source. Details: Appendix F. 2. Motivation and Requirements 2.1. Refit across layouts and versions Training systems and serving runtimes store the same policy in different layouts. Training systems shard parameters, gradients, and optimizer state to fit into GPU memory while maximizing update through- put [21,29]. Serving runtimes organize weights for inference and maintain request state such as a paged key-value (KV) cache [15]. RL systems that coordi- nate the training and rollout sides must bridge the two layouts [10,13,32]. The two sides choose their paral- lel degrees independently, so one training shard can overlap several rank-local slices on receivers, and one slice can combine parts of several shards (Figure 3). convertL r trainingcanonicalW v rollout rank 0 rank 1 012rankone element 0x3F00 0.5 0x3F01 0.50390625 0x3F02 0.5078125 0x3F03 0.51171875 updates Figure 3|Top: one tensor in the training, canonical, and rollout layouts. Bottom: BF16 values near 0.5, labeled by their stored bits in hexadecimal. Values in the shaded interval round to0x3F01, and only an update crossing its boundary changes the stored bits. Both layouts map to canonical tensors: the model’s tensors in the Hugging Face checkpoint’s naming and layout, independent of either cluster’s sharding. Let 푊 푣 denote the canonical tensors at version푣, includ- ing versioned buffers. A receiver푟does not store푊 푣 directly. Its native loader퐿 푟 is the serving runtime’s own weight loader and may split, combine, skip, trans- pose, or cast tensors before writing rank-local storage 푃 푣 푟 =퐿 푟 (푊 푣 ). Forexample, vLLM fuses the query, key, and value projections and splits routed experts across ranks. Each sequence of such operations is a loader path, and we consider loader paths whose out- put elements each come from one canonical element. A refit must bring each receiver to푃 푣+1 푟 =퐿 푟 (푊 푣+1 ), the result of a dense refit. Dense collective refit and full-checkpoint transfer produce this result by sending every weight, and Table 1 lists them as approaches A and B. We call the update to ver- sion푣+1 a transition, which may take several at- tempts. Each attempt updates the current receivers, called its required receivers. 2.2. Few BF16 values change per step How many values a refit must update depends on training: an update changes a stored BF16 value only when it crosses a rounding boundary (Figure 3), so larger learning rates and early training steps change more stored values. Yet even in the first five steps of Group Relative Policy Optimization (GRPO) [31], the six models in Figure 2 show only 0.6–1.2% average element change rates, counting each unique source element once. Appendix E details the setup, which follows an agentic software-engineering (SWE) RL recipe with a constant 10 −6 learning rate. 2.3. Requirements and related work To exploit this sparsity, a refit must meet five re- quirements that make placement (G3), exactness (G1 and G5), and efficiency (G2 and G4) concrete (Ta- ble 1).G1 Bit-exact:starting from receiver storage 푃 푣 푟 , a delta refit must produce the same parameter and buffer bits as a dense refit.G2 Delta:transfer volume must scale with the emitted changed values, their locations, and payload metadata.G3 Native: the native loader must determine placement, with no model-specific placement rules in the refit system. G4 Decoupled:no collective may span the training and rollout clusters.G5 Recovery:receivers must © 2026 NVIDIA. All rights reserved.3 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale training rollout commitprepare ACKopen_apply compare with baseline ̸= project or convertencode validate, dedupnative loader012rank coordinator 123 56 4 7 d canonical coordinates d object storagerelay tree 4 retry with overwrites controldata Figure 4|NeMo-DCR architecture and the numbered steps of a refit. The coordinator callsprepareto start an attempt andopen_applyto let receivers apply payloads and then acknowledge (ACK). suppress duplicate payloads, partial writes must be re- coverable, and one authoritative commit record must bind each new policy version to its source baseline af- ter every required receiver holds its new storage푃 푣+1 푟 . No existing system in Table 1 fully meets G5, and each fully meets at most twoof the five requirements. Other systems that ship policy weights to serv- ing runtimes are limited in scalability or efficiency: they send full checkpoints, rebuild full weights from deltas, or support only unsharded runtimes. INTELLECT-2’sSHARDCAST relays checksummed full checkpoints in pipelined shards [27]. In Com- poser 2 and Fireworks, trainers upload per-step deltas to shared storage, from which inference clusters re- build full weights [3,5]. TRL’s delta sync likewise rebuilds full tensors from a CPU snapshot [12], and the slime framework applies XOR or overwrite deltas to a full local checkpoint and reloads it at every re- fit [33]. vLLM patches runtime parameters in place, but doesn’t support any parallelism [37]. Related techniques target training traffic, fine-tuning, file transfers, or storage rather than policy re- fits: gradient compression [17,38], low-rank adap- tation [9], delta file transfer and lossless weight com- pression [8,34], and compression of fine-tuning deltas and checkpoints [4, 16, 18, 39]. 3. Design Overview Canonical coordinates split a refit into delta construc- tion and receiver-side placement. Figure 4 numbers the seven steps: owners construct deltas, a trans- port delivers them, receivers apply them, and the coordinator, which drives each attempt, publishes the policy with a joint commit. The next four sections (§ 4–7) explain these steps in detail and develop the four contributions in turn: projection and conversion, encoding, application and recovery, and delivery. Projection and conversion (❶–❷, §4).Shard owners compare their weights with the source baseline and directly project changes covered by affine map- pings, which are fixed and convert each source element independently. The rest are residual: their canonical tensors must be constructed first (contribution 1). Encoding (❸, §5). The codec encodes a change as an XOR entry when projection and the loader path preserve stored bits and no other write overlaps it, and as an overwrite entry otherwise (contribution 2). The first three steps together meet G1 and G2. Application and recovery (❺–❼, §6). Each re- ceiver validates and deduplicates payloads, then ap- plies their XOR or overwrite entries in place through the native loader. Failed attempts are retried with overwrites. After every required receiver acknowl- edges, the coordinator’s joint commit binds the policy version to its source baseline (contribution 3). These steps together meet G1, G3, and G5. Delivery (❹, §7). Object storage or the relay tree delivers the entries in payloads without a cross- cluster collective (G4). Streaming overlaps deliv- ery with delta construction and, in synchronous RL, also with application (contribution 4). Example.Suppose that a BF16 element changes its stored bits from0x3F01to0x3F02in a matrix sharded by rows in training and by columns in rollout. Its owner detects the change (❶), projects it to its canonical destination푑(❷), and emits the XOR mask 0x0003in a compressed payload (❸). The payload reaches every rollout node (❹), and each node’s re- ceivers validate it (❺). Each receiver whose column slice holds푑intercepts the loader’s final storage copy in order to XOR0x0003into0x3F01(❻). The joint commit publishes the updated policy, and the owner’s baseline becomes 0x3F02 (❼). Implementation. NeMo-DCR is about 7K lines of Python. It builds on Megatron Bridge conversion [25] and the vLLM native loader [15]. 4. Direct Projection and Residual Conversion In❶–❷of Figure 4, NeMo-DCR compares each train- ing shard with its source baseline in place, on the shard owners: direct projection avoids full-tensor as- sembly and conversion for affine changes, and residual conversion covers the other changes. The changes that an owner can directly project de- pend on the conversion tasks through which the Megatron Bridge library converts training shards to canonical tensors [25]. Affine tasks convert © 2026 NVIDIA. All rights reserved.4 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale each source element independently and determinis- tically with the fixed affine index mappings of sup- ported mapping classes, such as those for replicated and row- or column-split shards. Tasks for fused query/key/value (Q/K/V) source tensors and other tasks outside the supported mapping classes are residual and require constructing canonical tensors. Affine mappings cover 96.3% of the weight bytes in the Qwen3-30B-A3B checkpoint and 97.0% in the Nemotron-3-Ultra-550B-A55Bcheckpoint, both MoE models, as Appendix D details. 4.1. Ownership and change detection In❶, only one owner compares each unique source element against the source baseline, because data- parallel training keeps its replicas bitwise identical. A stable name hash balances ownership across data- parallel ranks (Figure A.2) and keeps it fixed across the attempts of a transition. Owners detect changes against the committed source baseline퐵 푣 . It contains a task tracker for each con- version task, holding the version-푣values of the task’s source shard, and a distributed residual baseline of canonical tensors for residual tasks. Because each unique source element has one owner, the trackers together hold one copy of the source weights. For task 푇, let푆 푣+1 푇 be the candidate shard,퐴 푣 푇 its tracker, and퐽 푇 its owned indices. Let퐼(푋) 푗 denote the stored bit pattern of 푋 at 푗. The changed indices are 퐶 푇 =푗 ∈ 퐽 푇 | 퐼(푆 푣+1 푇 ) 푗 ̸= 퐼(퐴 푣 푇 ) 푗 . Comparing stored bits detects all changes, even those that floating-point equality would miss (Figure A.2). Baseline memory.The baseline uses host mem- ory, not GPU memory. Beyond one copy of the source weights in the trackers, it holds the canon- ical tensors of residual tasks, which are 3.69% of theQwen3-30B-A3Bcheckpoint and 3.01% of theNemotron-3-Ultra-550B-A55Bcheckpoint. It thus totals about 1.04×and 1.03×the respec- tive checkpoint. Balanced ownership spreads it across the training ranks: for the 1,121 GB Nemotron-3-Ultra-550B-A55Bcheckpoint, each of the 32 training ranks of § 8.1 holds about 36 GB. 4.2. Direct projection In❷, fixed shard geometry and index mappings let an owner compute the canonical destination of each change locally. For a changed index푗of affine task푇, the index mapping휋 푇 identifies the canonical destination, and the source projection푄 푇 produces its updated value: 푑 = 휋 푇 (푗), 표 푣+1 푇 (푗) = 퐼(푄 푇 (푆 푣+1 푇 )) 푑 . (1) π T 2 entries convert24 entries = affine XOR w canonical coordinates i 0 k = (i 0 +i)w + y(i,y) residual 2 owners λ W v+1 u ̸= R v u overwrite 0000000000000011 0011111100000010 applied twice to 0x3F0114 leading zeros XOR 0x3F01 ⊕ 0x3F02 ⊕ 0x3F01 overwrite 0x3F01 = 0x3F02 = 0x3F02 Figure 5|Top: affine and residual changes. Bottom: XOR and overwrite entries of the example, each ap- plied twice to 0x3F01. Here,푑= (푛,푘) identifies a canonical tensor name and flat index, and 표 푣+1 푇 (푗) is the updated stored value. The owner emits each change once for all receivers without knowing their storage offsets. In the example of § 3, a changed local element (푖,푦) in a row shard that begins at row푖 0 maps to canonical destination (푛,푘), where푘= (푖 0 +푖)푤+푦for푤columns (Figure 5). Each receiver’s native loader then selects its own column slice. 4.3. Residual conversion In❷, residual conversion covers residual tasks, which fall outside direct projection because they lack a sup- ported mapping, can spread a source change across several outputs, or combine inputs from several own- ers. Examples include stacking, padding, tied weights, adapters, custom postprocessing, and quantization scales. For these tasks, NeMo-DCR constructs the affected canonical tensors and compares their bits with the committed residual baseline (Figure 5). Change detection covers every input of a conversion because a change in a shared scale factor휆can alter many converted values even when the other source values stay unchanged. For example, a fixed conver- sion rule may multiply each source value by휆and cast the product. Owners set a flag when a resid- ual task’s shard changes (Figure A.4). Combined with known conversion dependencies, the flags se- lect every canonical tensor affected directly or in- directly. The selected tensors form the required tensor set풰 req . The delta omits outputs whose converted bits remain unchanged. For each selected tensor푢, the owners convert their parts, and a canonical-name hash selects the residual-baseline owner that assembles these parts (Figure A.4). The parts must jointly cover every element of푢, and overlapping parts must agree bit- for-bit. Comparing the assembled result푊 푣+1 푢 with its committed residual baseline 푅 푣 푢 gives 퐷 푢 =(푛 푢 ,푘)| 퐼(푊 푣+1 푢 ) 푘 ̸= 퐼(푅 푣 푢 ) 푘 , © 2026 NVIDIA. All rights reserved.5 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale where푛 푢 is the canonical name. Residual tensors outside 풰 req keep their committed values. Both paths feed one changed set: the emitted destina- tions combine direct projections휋 푇 (퐶 푇 ) and residual changes퐷 푢 . Each emitted destination has a delta record with its updated value, and these records form the complete changed set ℰ that the codec encodes. 5.Mixed XOR/Overwrite Encoding with Compression In❸, the XOR/overwrite encoding keeps reconstruc- tion exact, and XOR also improves compression. Delta records in canonical coordinates share one codec and one payload format: the affine/residual path de- cides how a record is built, and the XOR/overwrite encoding decides how it is applied. Exact encoding rules out arithmetic reconstruction, which can fail to recover the updated stored values: for an old value훼and an updated value훽, rounding can make훼 + (훽− 훼)differ from훽, and such errors can accumulate across versions. NeMo-DCR instead sends either XOR masks of stored values or absolute overwrites. Both reproduce the new bits exactly: a mask flips only the bits that differ, and an overwrite replaces the whole stored value. For direct projection, the mask of changed index푗is 푥 푣+1 푇 (푗) = 퐼(푆 푣+1 푇 ) 푗 ⊕ 퐼(퐴 푣 푇 ) 푗 .(2) However, the mask yields the target bits only on a representation-preserving path, where the source projection푄 푇 and native loader preserve dtype and stored bits. In the shared-scale example of § 4.3, mul- tiplication and casting change stored bits, so a source mask does not carry over to the converted values (Fig- ure D.1). Comparison with the residual baseline finds such changes after conversion, and absolute overwrites carry the converted bits. To exploit unchanged bits within changed values, the default mixed mode uses XOR entries (푑,푥) wherever an affine path is representation-preserving and no other write touches the same bytes. These conditions are decided in advance for each mapping class and loader path (Figure D.1). All other records become overwrite entries (푑,표). In the two MoE checkpoints of § 4, most conversions are affine and preserve stored bits, so XOR covers more than 96% of the weight bytes, as Appendix D details. XOR entries also compress well: small updates of- ten leave high-order bits unchanged, so XOR masks contain many leading zeros. In the example of § 3, the XOR mask0x0003has 14 leading zero bits (Fig- ure 5). Run-length and gap coding shrink location bytes, and level-1 zstd [2] then compresses both values and locations in each payload. Yet XOR makes duplicate delivery unsafe: apply- ing a mask twice restores the old bits (Figure 5), so receivers suppress duplicate payloads by their identifiers. For the same reason, retries repair partial writes with overwrites. 6. Recoverable In-Place Refits In❺–❼, native-loader placement and overwrite recov- ery allow in-place refits from canonical coordinates without a receiver-side baseline copy. Reimplement- ing placement would duplicate the rules for fused Q/K/V, routed experts, slicing, padding, transposi- tion, and tied weights. NeMo-DCR instead lets the native loader apply these rules, intercepts the loader’s final storage copies, and specifies loader conditions that guarantee dense-refit bits. Overwrite recovery re- pairs partial writes, and a joint commit keeps receiver storage and the source baseline in version alignment: each transition starts with every required receiver at the committed version. 6.1.Applying deltas through the native loader Receivers apply payloads through the native loader one item at a time, where an item holds one owner’s entries for one tensor. After validating and deduplicat- ing payloads (Figure B.2), each receiver scatters each item into a reusable host scratch buffer with the canon- ical tensor’s shape and dtype. Positions without an entry are inactive and hold a placeholder푞(Figure 6, top). The loader resolves names and transforms the scratch, while a PyTorch dispatch hook updates only active positions at the loader’s final storage copies. q q q q q q q q rank 0 rank 1 scratchXOR:q = 0overwrite:q = NaN native loaderfinal copyown slice 0x3F01 ⊕ 0x0003 = 0x3F02 encodetransportapply W v+1 B v ∆ v+1 L r P v+1 r = L r (W v+1 ) version alignment payload deduplication native-loader placement Figure 6|Top: applying one item in❻of Figure 4. Bottom: the chain of Proposition 1 and the property used at each arrow. XOR path. Scratch holds the placeholder푞= 0 except for masks at changed positions, such as0x0003 at푑in the example. The intercepted copy XORs each © 2026 NVIDIA. All rights reserved.6 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale mask into the matching resident element, and the copy’s input must share storage with scratch so the masks arrive unchanged. XOR with푞= 0 leaves inactive positions unchanged. Overwrite path.Scratch holds overwrite values and a placeholder that must stay distinct from these values until the first intercepted copy into each desti- nation view. At that copy, non-placeholder positions form the view’s boolean active mask of positions to up- date (Figure A.10). NaN is the default placeholder. Receiver memory. Beyond resident weights, re- ceivers keep only the scratch, staged payloads, and temporary buffers for active masks and optional post-apply checks of intercepted copies (Figures C.2 and A.12). Aprewarmcall reserves scratch for the largest canonical tensor and identifies the tensors that the native loader skips on each rank, such as routed experts held by other ranks. Loader conditions.Exact delta application re- quires intercepting every storage write: the ten- sors the loader reports as loaded must match the intercepted copies, and an item without an inter- cepted copy requires an explicit native skip (Fig- ure A.8). The loader must transform each canonical value independently and deterministically through- out thetransition. XOR byte ranges must be dis- joint from all other writes, and overlapping over- writes must yield the same bits as a dense refit. NeMo-DCR rejects unsupported loader paths in ad- vance and fails any attempt whose copy breaks these conditions (Table A.2). 6.2. Equivalence to a dense refit Under these conditions, a NeMo-DCR refit is bit- exact. Let Δ 푣+1 encodeℰ, and letapply 푟 apply Δ 푣+1 at required receiver 푟, starting from its storage 푃 푣 푟 . Proposition 1 (Dense-refit equivalence). If a first attempt passes every check before its commit and the assumptions of Table A.3 hold, including fixed candidate weights푊 푣+1 , committed baseline, source ownership, conversion rules, and loader settings until the commit, then bits (︀ apply 푟 (Δ 푣+1 ,푃 푣 푟 ) )︀ = bits (︀ 퐿 푟 (푊 푣+1 ) )︀ ∀푟, (3) wherebitsreads all rank-local parameter and buffer storage but excludes request state such as the KV cache, which each transition invalidates. Proof sketch.XOR reconstructs each new value from the resident version-푣bits (Figure A.13, top). Overwrite supplies the value itself. Every changed destination has a record, and destinations with- out a record keep their version-푣bits, which al- ready match a dense refit because their canoni- cal elements did not change. The bottom of Fig- ure 6 outlines the argument, and Appendix A gives the full statement and proof. 6.3. Overwrite recovery and joint commit Overwrite retries extend Proposition 1 to transitions whose first attempt fails. Partial writes from a failed attempt leave receivers with a mixture of old and updated values. NeMo-DCR stops the failed attempt and waits until its writes can no longer take effect, then recomputes the complete changed set from the same candidate weights and unchanged퐵 푣 . Every record becomes an absolute overwrite, supplying the target bits regardless of which earlier writes completed (Figure A.14). The retry thus yields퐿 푟 (푊 푣+1 ). Re- ceiver membership changes take effect only between at- tempts, and new or restarted receivers restore the com- mitted version through a dense refit before joining. At the end of an attempt, the joint commit publishes the updated policy and binds it to its source baseline in a durable, versioned commit record that a control plane holds and changes only by compare-and-set. The coordinator commits after all required receivers have received every expected payload and finished application, device synchronization, and optional post- apply checks. A commit succeeds only if the commit record still equals the version-푣record퐾 푣 . Owners then update the task trackers and residual baseline in place (Figure B.3). This update finishes before the transition ends or change detection resumes, so the next delta uses the committed baseline. The commit record also makes coordinator failover safe: a new coordinator reads the same commit record before deciding whether to retry. Appendix B specifies the protocol as Algorithm B.2, with commit deadlines and the control-plane conditions for failover. 7. Pipelined Delivery without a Cross-Cluster Collective In❹, pipelined delivery over object storage or a relay tree overlaps other refit stages without a cross-cluster collective. Owners emit the payloads, and both trans- ports share the refit protocol of Appendix B. 7.1. Object-storage and relay-tree delivery Payload bytes scale with emitted values, locations, and metadata, but transport topology and retries de- termine how often payloads cross the cluster boundary. © 2026 NVIDIA. All rights reserved.7 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale Let푉 payload be the total bytes in unique compressed payloads before replication to receivers, and푉 cross the bytes crossing that boundary. O O N N N O O N N N Object storage trainingrollout no direct link Relay tree trainingrollout one crossing per payload Figure 7|Logical payload movement per attempt in ❹of Figure 4, with owners O, rollout nodes N, and the object-storage manifest of expected payloads. Object storage.This transport separates upload- ing from downloading (Figure 7, left). Owners upload each payload once and list it in a per-attempt mani- fest of expected payloads. Rollout nodes fetch every listed payload without a direct link. One stored copy in canonical coordinates serves every rollout node, but each download crosses the cluster boundary, so 푉 cross grows with the number of rollout nodes. A retry uploads the complete changed set again. Relay tree. Where direct links exist, this trans- port streams each owner’s payloads to one root in the rollout cluster (Figure 7, right). The root for- wards payloads through a balanced tree over local links, and each node’s receivers apply them. With- out retransmission, each payload crosses the cluster boundary once per attempt, so푉 cross ≈ 푉 payload . Ap- pendix B details both transports. 7.2. Overlapping refit stages Overlap matters because payload volume scales with the changed set, whereas delta construction cost scales with model size: construction scans source shards and converts required residual tensors. To overlap delta construction and delivery, owners group whole tensors into chunks for comparison and pack the resulting records into buckets, each compressed into one payload (Figure 8, top). Buckets amortize the overhead of metadata, compression, and transfer. In Figure 8 bottom, owners construct bucketℓ+1 while payloadℓis sent. Afteropen_apply, receivers in synchronous RL applyℓwhileℓ+1 arrives. Bounded queues slow upstream stages when receivers fall be- hind, limiting the number of decoded payloads held in memory. Pipeline startup anddrain leave stages idle. A request gate shields requests from partially applied weights and closes at different times in synchronous and asynchronous RL. In synchronous RL, the coor- dinator closes it when the first attempt starts, and receivers apply payloads during delivery (Figure 8, bottom). In asynchronous RL, receivers stage the owner transport rollout sync RL rollout async RL gate closed bucketℓbucketℓbucketℓ+1 convcmpzstd convcmpzstd sendℓ sendℓ+1 applyℓapplyℓ+1 applyℓ+1 servevservev+1 servevservev+1 tensors chunks records buckets ∅ startupdrainfinalization bucketℓ bucketℓ+1 time Figure 8|Top: chunks group whole tensors, and buckets pack their records. An unchanged chunk has none (∅) and adds nothing to bucketℓ, which continues past it. Bottom: stage timeline of residual- task buckets in❷–❼of Figure 4 with conversion (conv), comparison (cmp), and the request gate. compressed delta in host memory while rollout keeps serving version푣. After the last payload arrives, the coordinator closes the gate, and receivers apply the delta in place once current requests finish. The gate reopens when the transition ends. 8. Evaluation Research questions (RQs) 1–3 and 6 test the four contributions, and each subsection names the contri- butions it tests. RQs 4–5 test their combined effect. RQ1. How much smaller are XOR masks than over- write values after compression?(§ 8.2) RQ2. How do direct projection and XOR encoding change construction time and bytes?(§ 8.3) RQ3. Are refits bit-exact, and does training under kills follow dense NCCL?(§ 8.4) RQ4. How much faster is a NeMo-DCR refit than the reference at 120B?(§ 8.5) RQ5. How do latency and speedup change as models grow from 30B to 1T?(§ 8.6) RQ6. How much of refit latency do relay-tree trans- port and delta construction occupy?(§ 8.7) 8.1. Experimental setup and bit-exactness Testbed.The latency testbed has 32 NVIDIA GB300 GPUs on eight training nodes in one AWS region and 64 H100 GPUs on eight rollout nodes in another. The object-storage transport uses Ama- zon S3. Each node’s cross-cluster flow was measured at up to 5 Gbps. The latency runs use the BF16 Nemotron and Qwen3-235B-A22B checkpoints of Ta- ble C.1, with sizes from 63.2 to 2,242 GB, where 1 GB is 10 9 bytes. The 1T checkpoint doubles the layers of Nemotron-3-Ultra-550B-A55B. We run five warm-up refits and then ten timed refits for each training-side © 2026 NVIDIA. All rights reserved.8 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale configuration of parallel layout and replica count. The ten timings of each configuration span less than 1 s from minimum to maximum. We average them, then average these means with equal weight. Refit settings. Both transports use direct projec- tion and the same mixed-mode codec and bounded- queue policy, with the chunk, bucket, and queue sizes of Appendix C. The latency runs use synchronous RL, the training runs use asynchronous RL, and neither uses model-specific logic. Element change rates.Latency and delta con- struction runs use 3% or 5% element change rates as stress cases, since both exceed every rate in Fig- ure 2. Changed source elements are chosen uniformly from each owner’s shards and updated with a rel- ative amplitude of about 10 −2 . Replicated copies receive the same changes. Timed window. Refit latency covers comparison and projection, residual conversion, encoding and zstd, transport, receiver staging, loader application, device synchronization, ACK, joint commit, and the in-place baseline update (Figure C.5). It excludes benchmark change injection, post-apply checks, and one-time work such as model loading, initial baseline construction, and receiver prewarm. Reference.The two clusters share no InfiniBand or Elastic Fabric Adapter (EFA) route for NCCL. Of approaches A–D in Table 1, only full-checkpoint transfer meets both G1 and G4. We therefore use a transport-only full-checkpoint reference: each training node uploads an eighth of the checkpoint to shared ob- ject storage, and each rollout node then downloads all of it (Figure C.5). The reference excludes checkpoint save and load, to its advantage, whereas NeMo-DCR refit latency includes every stage of the timed window. Speedup is reference time divided by refit latency. Bit-exactness. Comparisons with dense refits from the same candidate weights confirmed bitwise equal- ity of every parameter and buffer element, including unchanged elements, in the evaluated BF16 receiver configurations. Every measured NeMo-DCR refit also passed its post-apply checks over all written parame- ter and buffer bits. 8.2. XOR masks are 1.7–2.2× smaller XOR masks are 1.7–2.2×smaller than overwrite val- ues after zstd level-1 compression (contribution 2, Table 2). We generate 20 million standard-normal values directly in BF16 and add independent Gaussian noise with standard deviation푎∈10 −4 ,10 −3 ,10 −2 in FP32. We round each result to the nearest BF16 value, as BF16 training does. The comparison isolates value encoding: XOR and overwrite streams encode the same changed words in input order, without loca- tions or metadata, as Appendix C details. Table 2|Compressed changed-value sizes for 20 mil- lion synthetic BF16 inputs, where 1 MB is 10 6 bytes. Noise standard Overwrite ↓ XOR ↓ Reduction ↑ deviation푎(MB)(MB) 10 −4 0.740.332.21× 10 −3 7.023.232.18× 10 −2 25.7014.931.72× 8.3. Projection saves time, XOR saves bytes Direct projection makes Qwen3 delta construction 1.08–1.16×faster than full conversion, and XOR en- coding cuts payload bytes by 38–40% versus over- writes only (contributions 1 and 2, Table 3). We com- pare three delta construction modes in the 3% and 5% stress cases on Qwen3-30B-A3B and Qwen3-235B- A22B, which use the same mapping classes. Mode X uses direct projection with mixed XOR/overwrite encoding, P uses direct projection with overwrites only, and F uses full conversion, which assembles and converts full tensors, with overwrites only. P versus F isolates direct projection, and X versus P isolates XOR encoding. Mode X spends 7–13% more con- struction time than P, yet it is the default because transport dominates relay-tree refit latency (§ 8.7). 8.4. Training follows dense NCCL under receiver kills With receivers killed mid-refit, both NeMo-DCR trans- ports follow dense NCCL’s mean reward and KL trajectories (contribution 3, Figure 9). To exercise in-place application and overwrite retries on receivers that survive a kill, we train Qwen3-30B-A3B on agentic tool calls for 50 GRPO steps with the set- tings of Appendix E. Training and rollout each use four H100 nodes on a second testbed. We compare three methods: a dense NCCL refit that sends ev- ery weight, object-storage NeMo-DCR, and relay-tree NeMo-DCR. These runs use real training updates, which change 1.2% of elements per step on aver- age. KL is the estimated per-token KL divergence between rollout and training policies. Every five steps in each NeMo-DCR run, we pick a vLLM instance uniformly at random, kill it mid-refit, and restart it five steps later (Figure E.1). Each kill fails the in-flight attempt, and the overwrite retry repairs any partial writes on the other receivers. Restarted receivers restore the committed version before rejoining. We run each method five times, and each NeMo-DCR run uses independent random © 2026 NVIDIA. All rights reserved.9 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale Table 3|Qwen3 delta construction time and payload size before replication. An arrow names the only difference between its modes. X PF direct projectionfull conversion mixed XOR/overwrite overwrites only XOR projection Qwen3 Change rate X (s) P (s) F (s) X (GB) P=F (GB) 30B 3%2.96 2.77 3.201.492.44 30B 5%3.23 2.99 3.292.293.80 235B 3%20.23 18.38 19.87 11.3718.42 235B 5%23.07 20.42 21.96 17.5829.36 0.3 0.5 0.7 Reward 11020304050 Step 0.0018 0.0021 0.0024 KL Dense NCCL NeMo-DCR (object storage) NeMo-DCR (relay tree) Figure 9|Qwen3-30B-A3B reward and per-token KL, averaged over five runs. Mid-refit receiver kills occur only in NeMo-DCR runs. choices. In these runs, every NeMo-DCR refit and overwrite retry passed all of its configured post-apply checks. Over the 50 steps, mean reward is 0.41 for dense NCCL and both NeMo-DCR transports. 8.5. 15–33× faster refits at 120B Including delta construction and loader application, 120B refits on both transports take 22.6–49.7 s and are 15.1–33.2×faster than the reference (Table 4). Before replication, compressed payloads including locations total 2.4–3.8% of the 247.2 GB checkpoint, 21–24% less than the uncompressed changed values alone, about 3% or 5% of the checkpoint. Table 4|Mean 120B refit latency versus the transport- only full-checkpoint reference. Volume is푉 payload for NeMo-DCR and the checkpoint size for the reference, both measured before replication. Method Change rate Latency ↓ Volume ↓ Speedup ↑ Object storage 3% 24.5 s5.86 GB30.6× Relay tree3% 22.6 s5.82 GB33.2× Object storage 5% 49.7 s9.38 GB15.1× Relay tree5% 41.6 s9.35 GB18.0× ReferenceN/A 750 s247.2 GB1× 8.6. 12–40× faster from 30B to 1T With either transport, refits of 30B–1T models stay 12–40×faster than the reference in the 3% and 5% stress cases, although refit latency grows with check- point size, from 8.3–14.5 s at 30B to 150–300 s at 1T (Figure 10). At 5%, the relay tree beats object storage at every size, and its advantage grows from 1.13× at 30B to 1.36× at 1T. 01,0002,000 BF16 checkpoint size (GB) 0.1 1 10 100 Minutes (log scale) 12–22× 15–33× 18–40× 12–26× 18–35× 3.0 12.5 28.9 39.3 87.5 Transport-only full-checkpoint reference NeMo-DCR refit, 3% and 5% element-change stress cases 01,0002,000 0 150 300 Re fi t latency (s) 3% element-change stress case Object storage Relay tree 01,0002,000 0 150 300 5% element-change stress case BF16 checkpoint size (GB) Figure 10|Top: reference time and mean NeMo- DCR refit latency. The band spans both transports and change rates, labels give reference minutes, and speedups sit on the dotted lines. Bottom: mean refit latency of each transport. 8.7. Relay-tree transport dominates latency Nsight Systems traces show that relay-tree transport dominates refit latency and overlaps delta construc- tion (contribution 4, Figure 11). The construction lower bound covers comparison, projection, resid- ual conversion, encoding, and zstd. The transport lower bound covers intervals from payload submis- sion until the sending endpoint receives the rollout node’s queueing acknowledgment. Each lower bound is the time its intervals cover on the busiest owner or sending endpoint, counting overlaps once (Fig- ure C.5). No transport interval includes the staging or application of its own payload. 30B120B235B550B 0 50 100 150 Time (s) 79% 82% 92% 94% 3% element-change stress case 30B120B235B550B 0 50 100 150 77% 93% 94% 93% 5% element-change stress case Before finalization Finalization Construction lower bound Transport lower bound Figure 11|Relay-tree refit latency and lower bounds. The percentages give the transport lower bound’s share of mean refit latency. In the stress cases, the transport lower bound accounts for 77–94% of refit latency, versus 18–45% for the © 2026 NVIDIA. All rights reserved.10 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale construction lower bound. Pipelined delivery lets delta construction overlap relay-tree transport. Neither lower bound covers finalization (Figure 8, bottom), which takes 0.2–1.7 s after application and covers device synchronization, ACK, the joint com- mit, and the in-place baseline update. Refit latency exceeds the larger bound plus finalization by 1.8–9.2 s, spent in startup, drain, and stage handoffs. 9. Conclusion Because only about 1% of BF16 values change per step, a delta refit can be much cheaper than a full- checkpoint transfer if it addresses placement, exact- ness, and efficiency, which NeMo-DCR does in canon- ical coordinates through four contributions. Direct projection avoids full-tensor assembly and conversion for affine changes, residual conversion covers the rest, and mixed XOR/overwrite encoding cuts Qwen3 pay- load bytes by 38–40% versus overwrites. Recoverable in-place refits match dense refits bitwise without a receiver-side baseline copy, and both transports follow dense NCCL’s mean reward and KL trajectories de- spite receiver kills. The source baseline stays in host memory and totals about one checkpoint across train- ing ranks. Pipelined delivery without a cross-cluster collective overlaps delta construction with relay-tree transport, and the transport lower bound accounts for 77–94% of refit latency. In the cross-cluster 3% and 5% stress cases for 30B–1T models, refits are12–40×faster than the transport- only full-checkpoint reference, and the relay tree takes 150 s instead of 87.5 min at 1T and 3%. These re- sults make per-update refits practical for cross-cluster agenticRL at trillion-parameter scale. NeMo-DCR is open source as part of NeMo RL [22]. References [1] J.-P. Aumasson, S. Neves, Z. Wilcox-O’Hearn, and C. Winnerlein. BLAKE2: Simpler, smaller, fast as MD5. In Proceedings of the 11th International Conference on Applied Cryptography and Network Security, volume 7954 of Lecture Notes in Computer Science, pages 119–135. Springer, 2013. doi: 10.1007/978-3-642-38980-1_8. [2] Y. Collet and M. S. Kucherawy. Zstandard compression and the ‘application/zstd’ media type. RFC 8878, Internet Engineering Task Force, 2021. URL https://w.rfc-editor.org/rfc/rfc8878. [3]Cursor Research Team. Composer 2 technical report. arXiv preprint arXiv:2603.24477, 2026. [4] A. Eisenman, K. K. Matam, S. Ingram, D. Mudigere, R. Krishnamoorthi, K. Nair, M. Smelyanskiy, and M. Annavaram. Check-N-Run: A checkpointing system for training deep learning recommendation models. In Proceedings of the 19th USENIX Symposium on Networked Systems Design and Implementation, pages 929–943. USENIX Association, 2022. [5] Fireworks AI. Frontier RL is cheaper than you think. Blog post, archived copy, 2026. Accessed August 2026. [6] W. Gao, Y. Zhao, D. Muhtar, D. An, X. Shang, T. Wu, L. Cao, S. Xiong, W. Wang, J. Huang, T. Ma, S. Yang, J. Wang, L. Qu, B. Zheng, and W. Wang. ROSE: Rollout on serving GPUs via cooperative elasticity for agentic RL. arXiv preprint arXiv:2605.06534, 2026. [7] W. Gao, Y. Zhao, T. Wu, S. Xiong, W. Wang, D. An, L. Cao, D. Muhtar, Z. Liu, H. Zhao, J. Huang, S. Yang, Y. Li, W. Su, J. Wang, L. Qu, B. Zheng, and W. Wang. RollArt: Disaggregated multi-task agentic RL training at scale. In Proceedings of the 20th USENIX Symposium on Operating Systems Design and Implementation, pages 863–881. USENIX Association, 2026. [8] M. Hershcovitch, A. Wood, L. Choshen, G. Girmonsky, R. Leibovitz, O. Ozeri, I. Ennmouri, M. Malka, P. Chin, S. Sundararaman, and D. Harnik. ZipNN: Lossless compression for AI models. In Proceedings of the 18th IEEE International Conference on Cloud Computing, pages 186–198. IEEE, 2025. doi: 10.1109/CLOUD67622.2025.00028. [9] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In Proceedings of the 10th International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. [10] J. Hu, X. Wu, W. Shen, J. K. Liu, W. Wang, S. Jiang, H. Wang, H. Chen, B. Chen, W. Fang, Xianyu, Y. Cao, H. Xu, and Y. Liu. OpenRLHF: A Ray-based easy-to-use, scalable and high-performance RLHF framework. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 656–666. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.emnlp-demos.48. [11] L. Hu, R. Zhao, I. Zhu, Z. Zhang, H. Zhang, H. Yin, and J. Zhao. SparseRL-Sync: Lossless weight synchronization with ∼100× less communication. arXiv preprint arXiv:2605.07330, 2026. [12] Hugging Face. Shipping a trillion parameters with a Hub bucket: Delta weight sync in TRL. Blog post, 2026. Accessed September 2026. © 2026 NVIDIA. All rights reserved.11 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale [13] S. Jiang, T. Shi, S. Zhang, Z. Wang, M. Di Francesco, and B. Zhao. Nereus: Adaptive parallelism for LLM post-training. arXiv preprint arXiv:2609.34645, 2026. [14] Kimi Team. Kimi K2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. [15] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626. ACM, 2023. doi: 10.1145/3600006.3613165. [16] W. Li, X. Chen, H. Shu, Y. Tang, and Y. Wang. ExCP: Extreme LLM checkpoint compression via weight-momentum joint shrinking. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 27575–27588, 2024. URL https://proceedings.mlr.press/v235/li24m.html . [17] Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. In Proceedings of the 6th International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=SkhQHMW0W. [18] J. Liu, G. Xiao, K. Li, J. D. Lee, S. Han, T. Dao, and T. Cai. BitDelta: Your fine-tune may only be worth one bit. In Advances in Neural Information Processing Systems 37, pages 13579–13600. Curran Associates, 2024. doi: 10.52202/079017-0434. [19] E. Miahi and E. Belilovsky. Understanding and exploiting weight update sparsity for communication-efficient distributed RL. arXiv preprint arXiv:2602.03839v2, 2026. [20] Moonshot AI. Checkpoint Engine: A simple middleware to update model weights in LLM inference engines. GitHub repository, commit d1de07b3, 2026. Accessed August 2026. [21] D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia. Efficient large-scale language model training on GPU clusters using Megatron-LM. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–15. ACM, 2021. doi: 10.1145/3458817.3476209. [22] NVIDIA. NeMo RL: A scalable and efficient post-training library. GitHub repository, commit 72149d09, 2026. Accessed August 2026. [23] NVIDIA. NCCL-Reshard refit. NeMo RL pull request #2971, 2026. Accessed August 2026. [24] NVIDIA. Two-stage SWE RL for Qwen3-30B-A3B-Thinking. NeMo RL guide, commit b7a4d95d, 2026. Accessed September 2026. [25] NVIDIA. NeMo Megatron Bridge: Training library for Megatron-based models with bidirectional Hugging Face conversion capability. GitHub repository, commit 554c7b93, 2026. Accessed August 2026. [26] Y. Peng, J. Zhang, W. Zhou, R. Xu, R. Yan, W. Dong, Y. Gao, Z. Ding, T. Yang, and B. Yuan. AReaL-DTE: Sparse policy-weight transfer for online agentic reinforcement learning. arXiv preprint arXiv:2608.00455, 2026. [27]Prime Intellect Team, S. Jaghouar, J. Mattern, J. M. Ong, J. Straube, M. Basra, A. Pazdera, K. Thaman, M. Di Ferrante, F. Gabriel, F. Obeid, K. Erdem, M. Keiblinger, and J. Hagemann. INTELLECT-2: A reasoning model trained through globally decentralized reinforcement learning. arXiv preprint arXiv:2505.07291, 2025. [28] P. Qi, Z. Liu, X. Zhou, T. Pang, C. Du, W. S. Lee, and M. Lin. Defeating the training-inference mismatch via FP16. arXiv preprint arXiv:2510.26788, 2025. [29] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. ZeRO: Memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. doi: 10.1109/SC41405.2020.00024. [30] C. Ruan, G. Luo, X. Wan, L. Zhao, Q. Wang, J. Zhu, D. Xu, G. Xu, D. Wei, X. Liu, C. Li, H. Sun, L. Luo, C. Miao, and J. Li. AuroraRL: Fast, fault-tolerant, and cost-efficient reinforcement learning over decentralized network. arXiv preprint arXiv:2602.11456, 2026. [31] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. [32] G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu. HybridFlow: A flexible and efficient RLHF framework. In Proceedings of the 20th European Conference on Computer Systems, pages 1279–1297. ACM, 2025. doi: 10.1145/3689031.3696075. [33] THUDM. Delta weight sync. slime documentation, commit 474861a, 2026. Accessed September 2026. [34] A. Tridgell and P. Mackerras. The rsync algorithm. Technical Report TR-CS-96-05, Australian National University, 1996. URL https://rsync.samba.org/tech_report/. © 2026 NVIDIA. All rights reserved.12 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale [35] verl. EP-aware sharded delta export (fused expert stacks). GitHub pull request #7085, 2026. Accessed September 2026. [36] verl. Sharded delta weight sync over NCCL for disaggregated rollout. GitHub pull request #6974, 2026. Accessed September 2026. [37] vLLM. Add sparse NCCL weight transfer support for in-place updates. GitHub pull request #40096, 2026. Accessed September 2026. [38] T. Vogels, S. P. Karimireddy, and M. Jaggi. PowerSGD: Practical low-rank gradient compression for distributed optimization. In Advances in Neural Information Processing Systems 32, pages 14259–14268. Curran Associates, 2019. [39] X. Yao, Q. Hu, and A. Klimovic. DeltaZip: Efficient serving of multiple full-model-tuned LLMs. In Proceedings of the 20th European Conference on Computer Systems, pages 110–127. ACM, 2025. doi: 10.1145/3689031.3717468. [40] C. Ye, H. Zhang, M. Han, B. Zhong, X. Li, Q. Chen, X. Zhang, W. Zhang, K. Jiang, W. Zhang, H. Sun, W. Xiao, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau. TensorHub: Scalable and elastic weight transfer for LLM RL training. In Proceedings of the 32nd Symposium on Operating Systems Principles, pages 882–897. ACM, 2026. doi: 10.1145/3830418.3843859. [41] Y. Zhong, Z. Zhang, X. Song, H. Hu, C. Jin, B. Wu, N. Chen, Y. Chen, Y. Zhou, C. Wan, H. Zhou, Y. Jiang, Y. Zhu, and D. Jiang. StreamRL: Scalable, heterogeneous, and elastic RL for LLMs with disaggregated stream generation. arXiv preprint arXiv:2504.15930, 2025. Appendices Appendix A states the source and native-loader con- ditions and proves the dense-refit equivalence. Ap- pendix B specifies the refit protocol and its failure handling. Appendix C gives integration and mea- surement details, Appendix D the affine and XOR coverage and the payload format, Appendix E the GRPO settings, and Appendix F the evidence behind Table 1. Table A.1 summarizes the main notation, and Figure A.1 places it on the refit path. A. Dense-Refit Equivalence This appendix states the conditions of Proposition 1 and proves it. Appendix A.1 also lists the direct- projection mapping classes of § 4. Table A.1|Main notation. Other symbols are defined in the sections where they are used. SymbolMeaningWhere 푊 푣 Canonical tensors at version 푣, including versioned buffers 2.1 퐿 푟 , 푃 푣 푟 Receiver 푟’s native loader and rank-local storage 퐿 푟 (푊 푣 ) 2.1 퐵 푣 Committed source baseline: task trackers and residual baseline 4.1 퐽 푇 , 퐶 푇 Owned and changed source indices of task 푇 4.1 푆 푣+1 푇 , 퐴 푣 푇 Candidate shard and tracker of task푇 4.1 퐼(푋) 푗 Stored bit pattern of 푋 at index 푗4.1 휋 푇 , 푄 푇 Index mapping and source projection of an affine task 4.2 푑 = (푛,푘) Canonical destination: tensor name and flat index 4.2 푅 푣 푢 , 퐷 푢 Residual baseline and changed destinations of tensor 푢 4.3 풰 req Required tensor set of an attempt4.3 ℰ Complete changed set of delta records4.3 (푑,푥), (푑,표) XOR entry and overwrite entry5 푞Placeholder at inactive scratch positions 6.1 Δ 푣+1 Encoding ofℰapplied at each receiver6.2 apply 푟 , bits Delta application at receiver 푟 and stored bits of rank-local parameters and buffers 6.2 퐾 푣 Commit record naming version 푣 and its baseline 6.3 푉 payload Compressed payload bytes before replication 7.1 푉 cross Bytes crossing the cluster boundary 7.1 푒Delta record: destination, overwrite value, XOR mask, and path tag A.1 풯 aff , 풯 res , 풰 Affine tasks, residual tasks, and residual-baseline tensors A.1 푔, ℳCandidate-baseline identifier and integration setup A.1 휌 푟 (푒)Destination byte ranges written on receiver 푟 for record 푒 A.2 휏Attempt identifier, returned by prepare B 푚Encoding mode: mixed or overwrite B A.1. Source conditions and version state This subsection states the source-side conditions and version state of Proposition 1: each unique source element has one owner and one conversion task, the owners receiver r S v+1 T vs A v T C T e = (d,o,x,χ) W v+1 u vs R v u D u E scratch, qP v r → P v+1 r π T ,Q T ∆ v+1 L r Figure A.1|Main notation of Table A.1 on the refit path. Owners compare the candidate shard푆 푣+1 푇 with its tracker퐴 푣 푇 , or a converted tensor푊 푣+1 푢 with its residual baseline푅 푣 푢 , and emit records푒intoℰ, whose encoding Δ 푣+1 each receiver applies through scratch and its native loader 퐿 푟 . © 2026 NVIDIA. All rights reserved.13 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale committed baseline stays fixed until the joint commit, and version alignment keeps every required receiver at the committed version. Ownership. Replication adds copies but not own- ers. The parallel layout identifies replica groups, which include the data-, context-, and expert-data- parallel replicas and the tensor-parallel copies of repli- cated parameters. Every training step keeps the copies within each replica group bitwise identical, so a stable name hash selects one owner among the copies of each shard (Figure A.2). Tied weights likewise have one owner and one task. Change detection. As in § 4.1, owners com- pare stored bit patterns, not floating-point values. Floating-point equality treats +0.0 and−0.0 as equal although their BF16 patterns0x0000and0x8000 differ. A zero that changes sign would then get no record, and receivers would keep the old sign bit. Thebitdiffcall of Table B.1 compares the stored patterns, so it detects this change (Figure A.2). shard 0shard 1shard 2shard 3 replica 0 replica 1 replica 2 owner −0.0+0.00x8000 0x80000x00000x0000 A v T S v+1 T receiver floating-point equality = no record bit patterns ̸= x = 0x8000 Figure A.2|Top: ownership across data-parallel repli- cas. Each replica holds a copy of every shard, and a stable name hash picks one owner per shard and spreads the owners across the replicas, so each unique source element is compared once. Bottom: change detection at a signed zero. Floating-point equality treats−0.0 and +0.0 as equal, so the change gets no record and the receiver keeps0x8000. Comparing bit patterns finds the change, and the XOR mask0x8000 flips the receiver’s sign bit, which gives0x0000, the value a dense refit writes. Affine tasks. The mapping classes Direct, Repli- cated, ColumnParallel, RowParallel, and GatedMLP of Megatron Bridge determine canonical coordinates from local indices, shard geometry, parallel ranks, and global expert indices [25]. Figure A.3 shows each class for two tensor-parallel ranks. For direct projec- tion, the Megatron Bridge integration supports only these affine mapping classes and Auto mappings that π T π T π T π T π T DirectReplicated ColumnParallelrows of [out, in]RowParallelcolumns of [out, in] GatedMLP gate up rank 0rank 1 non-owner copygate/up split Figure A.3|Affine mapping classes for two tensor- parallel ranks. Each rank’s shard maps through휋 푇 to fixed canonical coordinates, and only the owner’s copy of a replicated tensor is projected. permute no dimensions and act as ColumnParallel, RowParallel, or Replicated mappings. All of them use Bridge metadata for shard axes, offsets, and gate/up splitting. A custom export hook postprocesses con- verted tensors, so it disables direct projection. Residual tasks. Change flags select the residual tensors that each attempt converts. Tied embed- dings, the output head that depends on them, and any source tensor that also feeds a residual conver- sion always use residual tasks. For residual tasks, the training cluster’sall_reduce_flagscollective combines the change flags of § 4.3 by task identifier over all owners, and each owner contributes zero for tasks it does not hold (Figure A.4). Before assem- bly, the flags and conversion dependencies fix the required tensor set풰 req ⊆풰for the attempt, where 풰indexes the residual-baselinetensors. A custom export hook leaves conversion dependencies unknown, so any flagged task selects every residual tensor. Residual assembly. Residual tasks whose outputs form one canonical tensor are converted together when any of their shards changes. For example, per-expert tasks are converted together when the checkpoint stores all experts as one stacked tensor. A canonical-name hash selects the residual-baseline owner of each residual tensor푢from the fixed owners. Each owner’s part of푢carries the tensor name, shape, dtype, and canonicalindices it covers.FigureA.4 shows one such assembly, and Figure 5 contrasts this path with direct projection. Records. The affine and residual paths both emit records푒= (푑,표,푥,휒) inℰ(Figure A.5), with destina- tion푑, overwrite value표, XOR mask푥, and path tag 휒∈affine, residual. For residual records, 푥 =⊥. © 2026 NVIDIA. All rights reserved.14 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale all_reduce_flags owners T 1 T 2 T 3 100 001 000 1 1 0 u 1 u 2 u 3 u 4 U req ̸= R v u shardsparts of uW v+1 u D u E 0 E 1 E 2 convert owner h(n u ) Figure A.4|Top: residual change flags. Each owner sets 1 for its residual tasks whose shards changed, 0 for its other tasks, and a gray 0 for tasks it does not hold. The combined flags and the conversion dependencies select풰 req . Bottom: residual assembly of a stacked expert tensor푢. One changed element in expert퐸 1 makes every owner convert its part. The owner selected by a hashℎof the canonical name푛 푢 assembles푊 푣+1 푢 , and the comparison with푅 푣 푢 puts only the changed element into the set 퐷 푢 . (n,k)0x3F020x0003affine (n u ,k)0x4080⊥residual doxχ e = e = Figure A.5|Delta records, one per path. The affine record of the example carries the destination, the overwrite value0x3F02, and the XOR mask0x0003, while a residual record has no XOR mask (푥=⊥) and carries only its overwrite value. Version state. For affine tasks풯 aff and residual tasks 풯 res , the committed source baseline is 퐵 푣 =퐴 푣 푇 푇∈풯 aff ∪풯 res ∪푅 푣 푢 푢∈풰 . The baseline퐵 푣 reconstructs푊 푣 : the source pro- jections푄 푇 map the affine-task trackers into canon- ical coordinates, and the residual-baseline tensors supply the rest. The authoritative commit record 퐾 푣 = (푣,푔 푣 ) names version푣and the identifier푔 푣 of its baseline퐵 푣 . With푊 푣+1 and퐵 푣 fixed, the preparecall of Table B.1 binds a fresh candidate- baseline identifier푔to the attempt. The logical can- didate baseline퐵(푔) =퐴 푔 푇 ∪푅 푔 푢 is defined by 퐴 푔 푇 =푆 푣+1 푇 for every task and푅 푔 푢 =푊 푣+1 푢 for ev- ery required residual tensor, with푅 푔 푢 =푅 푣 푢 for every other residual tensor. Likewise,퐵(푔) reconstructs 푊 푣+1 when every changed residual tensor is in풰 req , as Figure A.6 shows for both baselines. Fixed state. The candidate weights푊 푣+1 , source ownership and membership, and integration setupℳ remain fixed across the attempts of the transition, and퐵 푣 remains fixed until commit. The integration setupℳcontains the conversion rules with their map- ping and dependency metadata, the native loaders with their settings and XOR/overwrite/skip classi- Q T , copy Q T , copy trackersresidual baselinecanonical B v A v T 1 A v T 2 R v u 1 R v u 2 W v B(g) S v+1 T 1 S v+1 T 2 W v+1 u 1 R v u 2 W v+1 Figure A.6|Committed and candidate baselines. Each row shows a baseline and the canonical ten- sors it reconstructs through the projections푄 푇 and the residual tensors.퐵(푔) replaces every tracker by its candidate shard and the required residual tensor푢 1 by its converted value, while 푢 2 keeps 푅 푣 푢 2 . fications, and the transport adapter of Figure C.1. To keep푊 푣+1 fixed, training takes its next opti- mizer step only after the transition ends (Figure A.7). Source membership is the set of training ranks, and changing it requires redistributing or reinitializing the baseline before another transition. Version alignment. The protocol preserves ver- sion alignment between the source baseline and re- ceiver storage: when a transition starts, every required receiver holds푃 푣 푟 for the version푣that the commit record names. Receivers obtain푃 푣 푟 from the com- mitted transition to version푣or from a dense refit, which also restores new or restarted receivers. Resi- dent weights change only through refits. To keep this alignment, a commit requires completion acknowl- edgments from every required receiver, and a new or restarted receiver joins only after restoring the com- mitted version. Figure A.7 follows one transition, and Appendix B.2 specifies both rules. A.2. Native-loader conditions For each source and loader path, Table A.2 lists the operations it allows, the rules the integration enforces, and the required assumptions. Loader paths whose required assumptions do not hold are rejected in ad- vance, and a copy that breaks an integration rule at run time fails the attempt. Native skips. Theprewarmcall runs the loader with tensor metadata to discover which tensors the loader skips on each receiver rank. Any reported load without an intercepted copy fails the attempt, as Figure A.8 shows. Loader properties. Beyond the conditions of § 6.1, the integration must establish two loader prop- erties: scratch bits remain unchanged before an XOR copy, and model parameters and buffers keep their existing storage (Figure A.9). Because run- time checks at intercepted copies cannot establish © 2026 NVIDIA. All rights reserved.15 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale Table A.2 | Operations, integration rules, and assumptions for the default BF16 source and loader paths. PathOperationsIntegration ruleRequired assumptions Source Affine Direct, Replicated, ColumnParallel, RowParallel, GatedMLP, and qualifying Auto Supported mapping classes; no custom export hook Fixed deterministic Bridge mappings; stored bits preserved for XOR entries Residual Other mapping classes, stacking, padding, tied weights, adapters, and custom postprocessing Residual conversion; forced overwrite Complete dependency metadata, or every residual tensor selected when unknown Loader XOR Identity, views, slices, splits, fusions Copy input shares scratch storage; same dtype; no within-item overlap Stored bits preserved; no overlap across items Overwrite Identity, views, slices, splits, fusions, casts, pointwise transforms such as − exp(·) Copy input is floating-point; non-NaN active values; active mask remains valid Placeholders preserved until each view’s first copy; each output depends only on its input Native skipRank-local skips Explicit skip report; reported loads match intercepted copies Accurate reports; interception of every storage write τ 2 step v+1step v+2 τ 1 commitupdate W v+1 , ownership, source membership,M B v B(g) training transition fixed baseline (v,g v )(v,g v )(v+1,g) record baseline receivers committed vattempt τafter commit B v B v B(g)B(g) P v r mixtureP v+1 r preparecommit Figure A.7|Top: what stays fixed during a transi- tion. Training takes its next optimizer step only after the transition ends. The candidate weights푊 푣+1 , source ownership and membership, and the integra- tion setupℳstay fixed across the failed attempt휏 1 and its retry휏 2 , and퐵 푣 stays fixed until the commit. Bottom: version state at three points of the transition. During an attempt, the commit record and퐵 푣 stay fixed, the dashed candidate baseline퐵(푔) exists only logically, and receivers may hold a mixture of old and updated values. After the commit, owners update the baseline in place to 퐵(푔). these properties alone, the integration also validates the configured loader in advance. XOR copies. The loader must preserve every bit pattern an XOR mask can contain at its dtype width, including NaN, infinity, subnormal, and signed- zero patterns (Figure A.9). Operations such as nan_to_num, clipping, and casts therefore use over- write only. An XOR copy fails the attempt if its E 2 E 3 rank 0 rank 1 rank 1 E 2 E 2 E 2 skips E 0 E 1 E 2 E 3 E 2 E 3 Figure A.8|Native skips for one routed-expert item퐸 2 , which every receiver gets. The loader on rank 0 skips퐸 2 , asprewarmrecorded, and writes nothing. On rank 1 it writes퐸 2 through the final storage copy inside the dashed dispatch hook. In the bottom lane, a write bypasses the hook, so a reported load has no intercepted copy and the attempt fails. input no longer shares scratch storage, its input and destination dtypes differ, or its destination byte range overlaps another copy within the item. Overwrite copies. The default overwrite paths require a floating-point copy input, whose non-NaN positions form each destination view’s active mask. An active value indistinguishable from the placeholder fails the attempt before the item is written. Ac- tive values are cast to the destination dtype. Later intercepted writes to the same destination view by a pointwise loader transform may reuse that active mask. The ordered writes of that transform must pre- serve the active mask and produce the same final bits as a dense refit (Figure A.10). Transforms that erase the placeholder require another validated placeholder scheme satisfying the conditions below. Placeholder schemes. To validate a placeholder scheme for an overwrite path, the integration traces © 2026 NVIDIA. All rights reserved.16 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale ×2 0x00030x00030x3F010x3F02 0x00030x00060x3F010x3F07 0x00030x0003?? scratchcopy inputresidentresult both hold ⊕= scratch changed ⊕= new storage ⊕= 0x7FC00x7FC00x3F800x4040 0x7FC00x00000x3F800x3F80 scratchcopy inputresidentresult identity ⊕= nan_to_num ⊕= Figure A.9|Top: loader properties on the XOR path of the example. With unchanged scratch bits and ex- isting parameter storage, the intercepted copy XORs the mask into the resident version-푣bits and yields 0x3F02. An in-place loader operation on scratch al- ters the mask, and new parameter storage lacks the resident bits, so both give wrong bits. Bottom: XOR masks on two loader paths. The mask between the resident 1.0 (0x3F80) and the updated 3.0 (0x4040) is the BF16 NaN pattern0x7FC0. An identity path keeps the mask and yields 3.0, whereasnan_to_num turns the mask into0x0000and leaves 1.0 unchanged. placeholders and active values throughnan_to_num, clipping, casts, quantization, and other loader oper- ations. Outputs must be deterministic and depend only on their corresponding inputs across supported shapes, dtypes, and value ranges, under fixed loader settings and cast behavior. The transformed place- holder must remain distinct from all active values until the active mask is formed. Overwrite retries keep the integration setupℳfixed but may choose another of the integration’s validated placeholder schemes. Figure A.10 contrasts a cast with nan_to_num. Write overlap. Let휌 푟 (푒) denote the set of destina- tion byte ranges written on receiver푟when applying the entry encoded from record푒. The overlap rules of § 6.1 apply to these ranges across entries, items, payloads, and loader calls. Runtime checks for XOR overlap are reset at each item, so the integration val- idates in advance that XOR writes do not overlap writes from other items. A loader change requires anotherprewarmcall and revalidation of these rules and the loader properties. Figure A.11 shows휌 푟 (푒) on a fused path. Post-apply checks. For an XOR copy, the op- tional post-apply check saves the destination’s pre- copy bits and verifies that XORing the mask into the updated bits restores the pre-copy bits (Fig- ure A.12). For an overwrite copy, the check com- pares the written bits with the overwrite values transformed by the native loader. cast nan_to_num ? scratchcopy inputactive mask NaN0.5NaN0.0NaN0.5NaN0.0 NaN0.5NaN0.00.00.50.00.0 copy q3q5 copy input version vafter copy φ on mask φ everywhere 24682365 26610 461210 φ=×2 Figure A.10|Top: the NaN placeholder on two over- write paths, with placeholders in gray and active values in green. A cast keeps the NaN placeholders, so the non-NaN positions of the copy input form the active mask.nan_to_nummaps the placeholders to 0.0, which is also an active value, so the mask can- not be formed and the path needs another validated placeholder scheme. Bottom: ordered writes to one destination view. The first intercepted copy forms the active mask from the non-placeholder positions of its input. The intercepted write of a later point- wise transform휙reuses that mask and yields the same bits as a dense refit. Applied everywhere, the doubling휙would transform the inactive positions a second time, because their version-푣values are al- ready transformed. gateup split, fuse rank 0 rank 1 ρ 1 (e) fused Figure A.11|A default loader path. The native loader splits the canonical gate and up projections across two tensor-parallel ranks at the dashed lines and fuses them per rank at the solid lines. Each output element has one canonical source, so a changed element of the up projection has one destination range 휌 1 (푒) on rank 1. 0x3F020x00030x3F010x3F01 0x3F020x3F02 XOR overwrite ⊕= ? = updatedxsaved writtenL r (o) ? = Figure A.12|Optional post-apply checks on the ex- ample. For an XOR copy, XORing the mask into the updated bits must restore the saved pre-copy bits. For an overwrite copy, the written bits must equal the overwrite value 표 transformed by the native loader. A.3. Equivalence proof The argument combines complete change detec- tion and the XOR/overwrite encoding with the three properties of Figure 6: version alignment be- tween the source baseline and receiver storage, pay- © 2026 NVIDIA. All rights reserved.17 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale load deduplication at ingestion, and native-loader placement at the receiver. Full statement of Proposition 1. For a first at- tempt that passes every check up to and including theflushcall of Table B.1, receiver storage satis- fies(3)under the following assumptions (Table A.3). The candidate weights푊 푣+1 , baseline퐵 푣 , source ownership and membership, and integration setup ℳremain fixed until commit, andℳincludes the conversion rules and loader settings. Each required receiver initially holds푃 푣 푟 , and receiver membership stays fixed during the attempt. Replicated source copies agree bit-for-bit, and the owners cover each unique source element exactly once. Every required residual tensor is fully assembled, with overlapping parts agreeing bit-for-bit. Projected values and over- write values are computed deterministically, and every XOR-encoded record uses a representation-preserving path. The residual dependency metadata is com- plete, or unknown dependencies select every residual tensor, so that푊 푣+1 푢 equals푅 푣 푢 bit-for-bit for ev- ery푢 /∈풰 req . The value written to each destination byte range depends only on its corresponding canon- ical element. The loader classification covers every final storage copy, and the loader conditions of § 6.1 and Appendix A.2 hold. Table A.3|Assumptions of Proposition 1, how each is established, and the figures that illustrate it. AssumptionEstablished byFigure 푊 푣+1 , 퐵 푣 , source ownership and membership, and ℳ fixed until commit Training waits for the transition; owners update 퐵 푣 only after commit A.7 Receivers start from 푃 푣 푟 ; receiver membership fixed per attempt Version alignment; receiver membership changes only between attempts A.7, B.5 Copies agree; one owner per source element Identical replica updates; name-hash ownership A.2 Residual tensors fully assembled and consistent Checks ofbuild_complete in Table B.1 A.4 Deterministic values; XOR only on representation- preserving paths Fixed conversion rules; encoding decided per mapping class and loader path D.1 Dependencies complete, or every residual tensor selected Change flags with dependency metadata A.4 Each written value depends only on its canonical element Supported loader pathsA.11 Every final storage copy classified; loader conditions hold prewarm skip discovery, copy interception, and advance validation A.8, A.9 XOR invariant. For each XOR-encoded record 푒from affine task푇and source index푗, the XOR invariant states that every range휌∈ 휌 푟 (푒) holds the matching source-baseline bits until its XOR copy: bits(푃 푣 푟 ) ⃒ ⃒ 휌 = 퐼(퐴 푣 푇 ) 푗 , 푒 = (휋 푇 (푗),표,푥, affine), where each range휌spans one stored value. Version alignment ensures that the receiver holds these bits: its resident weights are at version푣, the baseline version used to compute the XOR mask. Under the proposition’s assumptions, the projection푄 푇 and loader preserve dtype and stored bits, and no other entry writes to that range, so the invariant holds without transmitting baseline values (Figure A.13). Proof. Under the assumptions of the proposition, owners compare every unique source element and required residual tensor bitwise, affine outputs depend only on their source elements, and residual tensors outside풰 req keep their committed bits. Hence every changed canonical destination has a record, and the conflict checks leave it exactly one record or several overwrite records with identical value bits. For an XOR-encoded record푒from affine task푇and source index푗, the mask is푥=퐼(푆 푣+1 푇 ) 푗 ⊕ 퐼(퐴 푣 푇 ) 푗 . Scratch carries the item’s XOR masks at changed positions and zeros elsewhere. Payload deduplication, together with the arrival check and queue draining offlushin Appendix B.1, ensures that the receiver applies every XOR and overwrite entry exactly once. For every휌∈ 휌 푟 (푒), the invariant and the representation-preserving path give bits(푃 푣 푟 )| 휌 ⊕ 푥 = bits(퐿 푟 (푊 푣+1 ))| 휌 . For overwrite entries, a validated placeholder scheme establishes the active mask, and the loader conditions require the ordered writes of every pointwise loader transform to preserve it. Because each written value depends only on its canonical element, the complete overwrite sequence writesbits(퐿 푟 (푊 푣+1 )) at active positions and nowhere else. Every byte range written for an emitted destination therefore holds its target bits, and every remaining byte range keeps its version-푣bits (Figure A.13). The canonical element of each remaining range did not change, and the value a dense refit writes there de- pends only on that element, so these bits already equal bits(퐿 푟 (푊 푣+1 )). This establishes (3). Retries. After thestop_and_waitcall of Table B.1 completes for a failed attempt, a retry that passes every check up to and includingflushhas written every byte range of an emitted destination as an absolute overwrite. Every attempt recomputesℰde- terministically from the same푊 푣+1 ,퐵 푣 , ownership, © 2026 NVIDIA. All rights reserved.18 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale andℳ. If every write of the earlier attempts passed the dispatch hook, those attempts wrote only these ranges, and such a retry turns any mixture they left intobits(퐿 푟 (푊 푣+1 )) before commit (Figure A.14). Retries are therefore safe to repeat. A write that by- passes the hook can reach bytes outside these ranges. Its attempt fails (Figure A.8), as does every retry with that receiver, since the loader repeats the write. After retirement or the operator abort of Appendix B.2, a dense refit restores the receiver. B. Refit Protocol This appendix specifies the protocol operations behind the two transports of § 7 and the commit and recovery rules of § 6.3. The active coordinator runs one transition at a time. Each attempt has a fresh attempt identifier휏and candidate-baseline identifier푔, under which the com- plete changed setℰis streamed. Appendix A.1 defines the committed and candidate baselines and the state 0x3F020x3F010x0003 0x3F010x00030x3F02 owner receiver r ⊕= I(S v+1 T ) j I(A v T ) j x ⊕= bits(P v r )| ρ xbits(L r (W v+1 ))| ρ = 0x3F010x00030x3F020x3F02 0x40400x40800x40800x4080 0xBF800xBF800xBF80 residententryafterdense refit XOR ⊕== overwrite == no entry = Figure A.13|Top: XOR invariant for the changed element of the example. Version alignment keeps the receiver’s resident bits equal to the owner’s baseline bits, marked by the dashed equality, so the owner sends only the mask푥across the cluster boundary. Bottom: the three kinds of receiver byte ranges in the proof. An XOR entry turns the resident version-푣 bits into the target bits, an overwrite entry replaces the resident bits, and a range without an entry keeps its version-푣bits, which equal the dense-refit bits because its canonical element did not change. P v r mixtureL r (W v+1 ) mixed τ 1 overwrite τ 2 Figure A.14|Overwrite retry on one receiver, with emitted ranges outlined and updated bits in green. Ev- ery attempt emits the same ranges and writes nowhere else. The mixed-mode attempt휏 1 fails after writing two of them, and the overwrite retry휏 2 writes all four, which turns the mixture of old and updated values into the dense-refit result 퐿 푟 (푊 푣+1 ). that remains fixed across retries. In the signatures of Table B.1,ℳis the fixed inte- gration setup of Appendix A.1. Mode푚is mixed or overwrite, and overwrite mode encodes every record as an absolute overwrite. The symbols푝,id,ack, and 휎denote a payload, its identifier, an acknowledgment, and asuccessorfailedstatus, and푐is a commit out- come as defined in Appendix B.2. Task푇’s change flag is푓 푇 . Insidebuild_complete,bitdiffdetects changes, owners project affine changes into canoni- cal coordinates,all_reduce_flagsfixes풰 req , and convert produces the residual parts. Failure model. Owners, receivers, transports, and coordinators may crash or lose messages (Table B.2), but they do not act maliciously, and channels are trusted or encrypted. A control plane, assumed avail- able and clock-synchronized with coordinators, holds the durable commit record, acceptsprepareonly from the active coordinator, and raises aconflictex- ception for any laterprepareorcommitof a replaced coordinator. An attempt’s payloads, control calls, and gate operations carry the attempt identifier휏 that itspreparereturned, and receiver and transport endpoints ignore them once a laterpreparehas re- placed that attempt, except that itsstop_and_wait returns at once. If retries cannot repair a failure, operator intervention is required. B.1. Admission and delivery Each attempt runsprepare, then streams, checks, applies, and flushes its payloads as in Algorithm B.1. Admission. Thepreparecall checks the meta- data and loader classifications inℳ, then binds 푔, mode푚, and the set of required receivers to the fresh attempt. The first successfulprepare admits the transition. A failedpreparecheck raisesreject, and any other failure raisesfailed. Before admission, a rejectionends the transition. Any other failure before admission retries the first preparein mixed mode. After admission, every rejectorfailedexception triggers overwrite recov- ery, wherever in the attemptthe failure occurs. Al- gorithm B.2 starts in mixed mode, or in overwrite mode when it takes over an admitted transition after coordinator failover (Figure B.1). Streaming. Receivers validate and deduplicate pay- loads as they arrive. In synchronous RL, they decode them into the bounded queue but apply them only after the coordinator closes the gate with © 2026 NVIDIA. All rights reserved.19 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale Table B.1|Protocol operations. Operations that take an attempt identifier휏act only on that attempt. Appendix B.2 defines commit outcomes and the recovery rules. OperationEffect Source and coordinator refit(푊 푣+1 ,퐵 푣 ,퐾 푣 ,ℳ)Run a transition with overwrite recovery prepare(푣+1,ℳ,푔,푚)→ 휏Validateℳ, then bind푔,푚, and required receivers to a fresh attempt build_complete(푊 푣+1 ,퐵 푣 ,ℳ,푔)→ℰ ℓ ℓ Stream changed-set buckets under푔and check each for completeness and conflicting canonical values bitdiff(푋,푌 )Compare equal-width stored values and return changed indices all_reduce_flags((푇,푓 푇 ))→풰 req Reduce flags within the training cluster to find all affected tensors convert(푇,푢)Produce task 푇’s part of canonical tensor 푢 encode(ℰ ℓ ,푚)→ (푝 ℓ ,휎 ℓ )Encode bucket ℰ ℓ as compressed XOR or overwrite entries send(휏,푝 ℓ )Send identified payload 푝 ℓ under 휏 wait_for_requests(휏,푣)Close the request gate if open, then drain version-푣 requests open_apply(휏 )Let receivers decode staged payloads and apply queued payloads flush(휏 )→ (ack,휎) Check arrivals, drain receiver queues, synchronize devices, report any configured post-apply checks, acknowledge commit(휏 )→ 푐Conditionally replace 퐾 푣 with (푣+1,푔) after flush read_commit(휏 )→ 퐾Return the record once no commit of 휏 can change it mark_obsolete(휏 )Invalidate 휏 for later payloads, open_apply, and commit stop_and_wait(휏 )→ ackStop delivery, then wait for writes and acknowledgments cleanup(휏 )Release the payloads and, on failure, the delta records Receiver and transport endpoints prewarm(ℳ)→ 휎Reserve scratch and discover native skips ingest(휏, id,푝) Validate and deduplicate, then queue decoded payloads in synchronous RL or stage compressed payloads in asynchronous RL apply(푝)Scatter into scratch, then call the native loader Table B.2|Failures and how the protocol handles them, with the figures that illustrate each case. FailureHandlingFigure Rejected prepare Before admission, ends the transition; after admission, triggers overwrite recovery B.1 Payload of an obsolete attempt Dropped at receptionB.2 Invalid payload The attempt fails before the payload is applied B.2 Duplicate payload Ignored if its bytes match; otherwise the attempt fails B.2 Missing payload The arrival check fails, followed by an overwrite retry B.4 Receiver killed mid-attempt Retired by stop_and_wait and rejoins after a dense refit; the overwrite retry updates the others B.5, A.14 Lost commit reply After the deadline, read_commit decides the outcome B.4 Coordinator failure Takeover drains the attempt, calls read_commit, then retries or ends with a conflict B.6 Persistent failureOperator abort; a closed gate stays closed until a dense refit B.6 wait_for_requestsand then callsopen_apply, so version-푣requests finish before any payload is applied. In asynchronous RL, they retain validated compressed payloads in a separate host staging area. After the coordinator confirms that every expected payload has arrived, closes the gate, and callsopen_apply, a de- coder feeds the bounded queue in declared order while the application drains it in that order. Transports. Both transports follow Algorithm B.1 but carry payloads differently. On object storage, each payload is uploaded as a multipart object. Payloads and the manifest remain in storage un- til cleanup, which follows commit or a completed stop_and_wait. The relay tree streams payloads over ZeroMQ DEALER/ROUTER sockets. Reception checks. Receivers learn the version and identifier of the active attempt atprepare. Before queueing, they check that each payload belongs to the active attempt, so they drop late payloads from obsolete attempts. Receivers then check each pay- load’s compressed-body digest, tensor names, shapes, admittedsuccess rejectfailed failed success reject or failed takeover first preparemixed attemptcommitted overwrite attemptend,v stays Figure B.1|Admission and attempt modes. A re- jected firstprepareends the transition with version푣 still committed, and a failed one is retried. After admission, anyrejectorfailedexception leads to over- write attempts, which repeat until commit, a conflict, or an operator abort. A coordinator that takes over an admitted transition starts in overwrite mode. © 2026 NVIDIA. All rights reserved.20 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale Algorithm B.1 Concurrent streaming schedule of one attempt. Failures propagate to Algorithm B.2. Require: Prepared 휏, 푔, 푣, 푚, 푊 푣+1 , 퐵 푣 , ℳ, bounded queues Ensure: (ack, success) or (⊥, failed) 1: spawn owners 2: for all buckets ℰ ℓ streamed by build_complete do 3:(푝 ℓ ,휎 ℓ )← encode(ℰ ℓ ,푚) 4:if 휎 ℓ ̸= success then 5:raise failed 6:end if 7:send(휏,푝 ℓ ) 8: end for 9: spawn receiver ingestion 10: for all arriving (id,푝) do 11:ingest(휏, id,푝) 12: end for 13: spawn receiver application 14: wait until the coordinator calls open_apply(휏 ) 15: if asynchronous RL then 16:spawn a decoder for staged payloads in declared order 17: end if 18: for all 푝 yielded by the bounded queue in declared order do 19:apply(푝) 20: end for 21: if asynchronous RL then 22: await all sends 23: wait for delivery and forwarding 24: raise failed if any expected payload is missing 25: end if 26: wait_for_requests(휏,푣) 27: open_apply(휏 ) 28: await all sends 29: return flush(휏 ) dtypes, schema, and loader classification. If any of these checks fails, the attempt fails before that pay- load is applied. Receivers key each payload by its attempt, owner, and payload identifiers, apply it at most once, and ignore identical duplicates. Reusing an identifier with different bytes fails the attempt. Fig- ure B.2 summarizes these checks and their outcomes. Application order. After these reception checks, all receivers apply payloads in the single order that the payload identifiers declare for the attempt rather than in arrival order (Figure B.2). The staging queue reserves a slot for the next payload to apply, so later payloads cannot fill it first. Receivers may reorder only independent loader operations that have no cross- rank collectives or cross-payload side effects. Invalid payloads, conflicting canonical values, or conflicting destination writes stop application and fail the at- tempt, whether it is the first attempt or a retry. Flush. Theflushcall closes the attempt to new sends, waits for delivery and relay-tree forwarding, and checks that every expected payload arrived: every payload in the manifest on object storage, or every active attemptτ? late: drop valid digest, schema, names, shapes, dtypes, classification? no: fail the attempt new identifier? same bytes: ignore, else: fail queue apply in declared order rank 0 rank 1 arrivaldeclared order 213 132 123 123 queue Figure B.2|Top: checks on each received payload. An obsolete attempt’s payload is dropped, a failed check fails the attempt, and a duplicate payload is ignored when its bytes match and fails the attempt otherwise. Bottom: application order on two receiver ranks, with a loader collective per payload drawn as a dashed link. Payloads arrive in different orders, so applying them on arrival would pair different payloads in a collective. Applying them in the attempt’s declared order pairs the same payload on both ranks. payload the owners sent on the relay tree. In asyn- chronous RL, the same arrival check also runs before the request gate closes, andflushrepeats it afterward. Relay-tree queueing acknowledgments and the imme- diate responses to control calls confirm only queue- ing or registration, soflushalso drains the receiver queues, synchronizes devices, and collects completion acknowledgments that report the configured post- apply checks. Figure B.3 contrasts the two kinds of acknowledgments. A failedflushreturns (⊥, failed), which Algorithm B.2 treats as a failed exception. B.2. Joint commit and recovery Version푣stays committed through every failure and retry until a joint commit publishes version푣+1 after every required receiver acknowledges completion. Commit and end of the transition. Afterflush succeeds with completion acknowledgments from ev- ery required receiver,commitmay atomically replace 퐾 푣 with (푣+1,푔), provided the commit record still equals퐾 푣 . Owners start the in-place baseline up- date of § 6.3 when they read the commit record (푣+1,푔), even when the commit is confirmed late or after failover. Each owner then reports completion to the coordinator, which ends the transition only af- ter every owner has reported (Figure B.3). Receivers invalidate their KV caches before the coordinator reopens the request gate (Figure 8, bottom). Commit outcomes. Commit outcome푐issuccess, failed,unknown, orconflict. If the coordinator cannot © 2026 NVIDIA. All rights reserved.21 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale ONN queuedqueued coordinator flushACK required receivers ACK commit: K v → (v+1, g) owners update B v in place transition ends, gate reopens all owners report Figure B.3|Top: acknowledgments on the relay tree of Figure 7. Each hop only acknowledges that it queued a payload. Afterflushdrains the receiver queues and synchronizes devices, each receiver sends its completion acknowledgment, the ACK of Fig- ure 4. Bottom: the end of a transition. After ev- ery required receiver acknowledges,commitreplaces 퐾 푣 with (푣+1,푔). Owners read the new record, update their baselines in place, and report. Once all owners have reported, the transition ends and the request gate reopens. confirm the outcome,commitreturnsunknown. Be- fore the coordinator decides whether to retry after anunknownoutcome or a failover,read_commit(휏) waits until no commit of휏can change the commit record. Each commit carries a deadline set by its coordinator. The control plane accepts it only before then and never after its attempt is marked obso- lete, which bounds this wait (Figure B.4, bottom). The commit record (푣+1,푔) confirms success, un- changed퐾 푣 confirms failure, and any other value signals a commit conflict, meaning that another coor- dinator committed the transition first. A conflict that read_commitreveals after the coordinator’s own com- mit can reach only a replaced coordinator. It stops its attempt, cleans up, and returns without retrying, leaving the gate to the coordinatorthat committed. Of these outcomes, only a confirmed failure takes the retry path in Figure B.4, top. Stopping an attempt. Before recovery, the coor- dinator callsmark_obsolete, after which endpoints ignore the attempt’s later payloads andopen_apply, and the control plane refuses the attempt’scommit (Figure B.5). Thenstop_and_waitstops delivery and relay-tree forwarding, waits for pending trans- port work and receiver writes to finish, synchronizes devices, and collects acknowledgments from all surviv- ing endpoints. A terminated endpoint is retired from the failed attempt only after its process and pending writes are confirmed unable to resume. Receiver mem- bership changes take effect only between attempts, so a retired receiver is not a required receiver of the retry (Figure B.5, bottom). After stop_and_wait returns, cleanupreleases the attempt’s payloads and delta records, leaving퐵 푣 unchanged. A failure before a suc- cessful prepare leaves nothing to stop or clean up. Algorithm B.2 Refit with overwrite recovery. Re- tries stop at commit success, a conflict, an operator abort, or a rejection before admission. Require: 푊 푣+1 , 퐵 푣 , 퐾 푣 , fixed ℳ and source membership, and 휏 old on takeover Ensure: 푣+1 satisfying (3), or reject / conflict 1: 푚← mixed; admitted ← false 2: if taking over an admitted transition then 3: mark_obsolete(휏 old ); stop_and_wait(휏 old ) 4: 퐾 ← read_commit(휏 old ) 5: if 퐾 ̸= 퐾 푣 then 6:await baseline update named by 퐾 7:cleanup(휏 old ); reopen the gate under 휏 old 8:return conflict 9: end if 10: cleanup(휏 old ) 11:admitted ← true; 푚← overwrite 12: end if 13: loop 14: 휏 ←⊥; 푐← failed 15: try 16: 푔 ← new identifier; 휏 ← prepare(푣+1,ℳ,푔,푚) 17:admitted ← true 18:(ack,휎)← concurrent schedule in Algorithm B.1 19:if 휎 ̸= success then 20:raise failed 21:end if 22: 푐← commit(휏 ) 23: catch reject 24:if ¬admitted then return reject 25: catch conflict: return conflict 26: catch failed: pass 27: end try 28: if 푐 = unknown then 29: 퐾 ← read_commit(휏 ) 30: 푐← ⎧ ⎨ ⎩ success 퐾 = (푣+1,푔), failed 퐾 = 퐾 푣 , conflict otherwise. 31: end if 32: if 푐 = success then 33:await baseline update named by (푣+1,푔) 34:cleanup(휏 ); reopen the gate under 휏; return 푣+1 35: end if 36: if 휏 ̸=⊥ then 37:mark_obsolete(휏 ); stop_and_wait(휏 ) 38:cleanup(휏 ) 39:if 푐 = conflict then 40:return conflict 41:end if 42: end if 43: if admitted then 푚← overwrite 44: end loop Failover and conflicts. Commit conflicts arise only from coordinator failover (Figure B.6): the co- ordinator that committed first is either a new coor- dinator started after failover or the old coordinator whose pending commit took effect. On takeover, the new coordinator first stops and drains writes from the old attempt휏 old , then waits withread_commit until no commit of휏 old can change the record. If such a commit took effect, the new coordinator waits for the baseline update named by the record, cleans up, reopens the gate under휏 old , and returnsconflict. If the record remains퐾 푣 , the new coordinator cleans up © 2026 NVIDIA. All rights reserved.22 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale and retries in overwrite mode from the unchanged퐾 푣 , repairing any partial writes before its own commit replaces퐾 푣 . Itspreparereplaces휏 old , so endpoints then ignore whatever the old coordinator still sends. The control plane also raises aconflictexception for any laterprepareorcommitof the old coordina- tor, which then returns without retrying or touching the request gate. An old coordinator whose com- mit outcome wasunknownlearns of the conflict from read_commit, stops and cleans up its attempt, and also leaves the gate alone. Owners hold their candi- date shards and their baselines in memory, so they can finish the in-place baseline update after a co- ordinator failover.Becausethe update writes abso- lute values derived from the fixed푊 푣+1 , an owner that reads the commit record again can safely re- peat it. Failover therefore needs no durable record of which owners have finished. Joining receivers. A new or restarted receiver restores the committed version through a dense refit, runsprewarm, and joins at the nextprepareif the commit record still names that version. Otherwise, it first restores the newer committed version with another dense refit before joining. Operator intervention. Some persistent failures fall outside the automatic recovery of Algorithm B.2 and require operator intervention: conflicting canoni- cal values or destination writes, a write that bypasses the dispatch hook, a receiver that never acknowledges, or an owner that loses its candidate shards or its base- line in a crash. The operator aborts the transition: the coordinator stops and drains the attempt and published stop_and_wait, overwrite retry, newτ preparesendapply, flushcommit K v ?K coordinator control plane commit unknown read_commit deadline (v+1, g): success K v : failed other: conflict Figure B.4|Top: the commit sequence. Version푣 stays committed until success, and a failure after prepareleads tostop_and_waitand an overwrite retry under a new휏. Bottom: resolving an unknown commit outcome, which takes the retry path only after read_commitconfirms failure. The reply tocommitis lost, so the record may hold퐾 푣 , (푣+1,푔), or another value.read_commitwaits until the deadline, after which the control plane accepts no commit of휏, so the record 퐾 it returns decides the outcome. mark_obsoletestop_and_waitcleanup refuses commit(τ) ignoreτ, drain, sync coordinator endpoints control plane ACK retired τ 1 τ 2 τ 3 dense refitprewarm coordinator r 1 r 2 r 3 ACK retired stop_and_waitcommitprepare Figure B.5|Top: the steps that stop a failed at- tempt휏. Once the coordinator callsmark_obsolete, the endpoints ignore the later payloads and the open_applyof휏, and the control plane refuses its commit. Thenstop_and_waitwaits until the sur- viving endpoints drain pending work, synchronize devices, and acknowledge, and it retires a terminated endpoint that cannot resume.cleanupthen releases the attempt’s payloads and delta records, leaving퐵 푣 unchanged. Bottom: receiver membership across at- tempts. Receiver푟 3 fails during휏 1 and is retired, so it is not required in the retry휏 2 . After a dense refit and prewarm, 푟 3 joins the next attempt 휏 3 at prepare. record̸= K v record = K v new coordinator: mark_obsolete, stop_and_wait(τ old ) read_commit(τ old ) await baseline update, clean up, reopen gate clean up, prepare→ τ: endpoints ignore τ old conflict: v+1 publishedoverwrite retry, commit old coordinator fails in τ old conflicting values or writes write bypasses hook receiver never ACKs owner loses its state operator aborts: stop, drain, clean up a closed gate stays closed until a dense refit restores receivers owner reinitializes its baseline Figure B.6|Top: coordinator takeover. After drain- ing the old attempt, the new coordinator waits with read_commituntil no commit of휏 old can change the record. It then retries in overwrite mode or ends the transition with a conflict. Bottom: operator inter- vention. Each persistent failure on the left makes the operator abort the transition, and a closed gate stays closed until a dense refit restores every required receiver to the committed version. An owner that lost its state also reinitializes its baseline, drawn dashed. cleans up, and a closed request gate stays closed un- til a dense refit restores every required receiver to the committed version (Figure B.6). An owner that lost its state also reinitializes its baseline, as after a source membership change. © 2026 NVIDIA. All rights reserved.23 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale C. Integration and Measurement This appendix describes the implementation behind § 3 and gives the pipeline settings for § 7.2 and mea- surement details for § 8. The cited NeMo RL com- mit [22] contains the Megatron Bridge and vLLM integration used in the experiments. Figure C.1 sketches the integration. Each transport adapter im- plements onlysend,stop_and_wait, andcleanupof Table B.1, and Appendix A.1 lists the training-side conversion tasks and mapping classes. Rollout side. The rollout side adds three pieces to the native loader: aprewarmcall, a dispatch hook, and a staging queue. Forprewarm, receivers obtain canonical names, shapes, and dtypes from the train- ing side. The dispatch hook intercepts the loader’s final storage copies, which areaten.copy_operations. The receiver reserves 32 resident groups of at most eight decoded payloads, plus one separate group of at most eight payloads under assembly, for at most 264 decoded payload slots in all (Figure C.2, bottom). In asynchronous RL, a separate host staging area holds compressed payloads untilopen_apply. A de- coder then feeds this queue in declared order while the application drains it. Figure C.2 also shows the mem- ory that a receiver keepsbeyond its resident weights. Pipeline settings. Owners compare source shards and convert required residual tensors in chunks, with targets of 64 MiB for object storage and 256 MiB for the relay tree, and both transports use a 512 MiB bucket target. The different chunk targets shift bucket boundaries, so the payload volumes of the two transports differ slightly in Table 4. These tar- gets are measured in bytes of local source tensors for affine tasks and of converted canonical tensors for residual tasks, before unchanged elements are dropped. A bucket counts the full input size of each chunk that emits records. Tensors and chunks are never split, so a chunk or bucket can exceed its target. At 120B, the bucket target yields 482 pay- loads across 32 owners. Figure 8 illustrates both units on one owner, and Figure C.3 shows how their sizes count toward the targets. Synthetic codec benchmark. The benchmark of § 8.2 runs PyTorch 2.14.0 on four CPU threads, with random seed 42 for the BF16 inputs and 44 for the FP32 noise. The same input and noise vectors serve all three values of푎. Scaling the noise by푎and adding it to the input are separate FP32 operations, and rounding precedes the bitwise comparison that selects changed words (Figure C.4). Each stream holds only little-endian 16-bit words compressed with zstd 1.5.5 at level 1. The overwrite stream decodes to the updated BF16 words, and XORing the decoded masks into the inputs gives the same words. The reduction ratios of Table 2 use exact byte counts. Table C.1 | Checkpoints of the latency runs. Size CheckpointGB 30B Nemotron-3-Nano-30B-A3B63.2 120B Nemotron-3-Super-120B-A12B247.2 235B Qwen3-235B-A22B470.2 550B Nemotron-3-Ultra-550B-A55B1,121 1T Nemotron-3-Ultra-550B-A55B with twice the layers 2,242 Latency runs. Table C.1 lists the BF16 check- points behind the 30B–1T latency points, where the 30B point isNemotron-3-Nano-30B-A3B, not the Qwen3-30B-A3Bof § 8.3 and 8.4. Figure C.5 shows the testbed and the transport-only full-checkpoint reference of § 8.1. Timed window and intervals. Each refit’s timed window runs from푡 0 to푡 1 . At푡 0 , the first owner begins delta construction, and at푡 1 , the last owner finishes its baseline update after the joint commit. Re- fit latency is푡 1 − 푡 0 minus the post-apply check time. For the lower bounds of § 8.7, each compute interval runs from an operation’s start until its output is ready: change flags or locations for comparison, mapped in- dices and values for projection, canonical tensors for residual conversion, location and value buffers for encoding, and compressed bytes for zstd. For non- blocking operations, the interval ends when execution completes. Each relay-tree transport interval runs from payload submission until the sending endpoint, the owner’s transport endpoint, receives its queueing acknowledgment. On the relay tree, receivers stage and apply a payload only after their rollout node acknowledges its queueing, so no transport interval waits for its own payload’s staging or application. In the relay-tree latency runs, the staging queue never filled, so no interval waited for queuespace either. Finalization runs from the end of the last loader ap- plication to푡 1 . Thepreparecall precedes푡 0 , and cleanup and the gate reopening follow 푡 1 . Lower bounds. Each lower bound is the longest union of intervals on one owner or sending endpoint: the construction lower bound takes the union of each owner’s compute intervals, and the transport lower bound takes the union of each sending endpoint’s transport intervals. Before taking unions, we map Nsight Systems trace timestamps onto a common © 2026 NVIDIA. All rights reserved.24 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale # Transport adapter: three calls class ObjectStorage: def send(self, tau, p): put(key(tau, p.id), p) post_manifest(tau, p.id) def stop_and_wait(self, tau): stop(tau) return wait_until_done(tau) def cleanup(self, tau): delete(tau) class RelayTree: def send(self, tau, p): root.push(tau, p) def stop_and_wait(self, tau): stop_forwarding(tau) return collect_acks(tau) def cleanup(self, tau): release(tau) # Training side: conversion tasks for T in conversion_tasks(model): track(T) # every task if affine(T, M): # direct projection project(T) else: # residual path residual(T) # Mapping metadata gives shard # axes and offsets. # No model-specific refit code. # Rollout side: native loader prewarm(M)# scratch, skips on payload p: # either transport ingest(tau, p.id, p) def apply(p): for name, idx, vals in p.items: scratch[:] = q scratch[idx] = vals with dispatch_hook(): loader(name, scratch) # No model-specific placement code. Figure C.1|Integration sketch. The codec, receiver checks, commit, and recovery of Algorithms B.1 and B.2 form a shared core with three interfaces: a transport adapter with three calls, conversion tasks with mapping metadata on the training side, and the serving runtime’s native loader with a dispatch hook on its final storage copies. time axis and clip each interval to [푡 0 ,푡 1 ], discard- ing empty intersections. Each union lies within the refit’s timed window, so both bounds are at most 푡 1 − 푡 0 . We compute bounds per refit and aver- age them as § 8.1 averages refit latency. Mean re- fit latency, which already excludes the post-apply check time, exceeds both mean bounds at every traced size. Figure C.5 illustrates both bounds in hostGPU staged delta staging queuescratch residentP r buffers L r staged decode assembly32 groupsloader open_apply full Figure C.2|Top: receiver memory beyond the resi- dent weights푃 푟 . The host holds the decoded staging queue and the reusable scratch thatprewarmsizes for the largest canonical tensor. Dashed boxes are tem- porary: the compressed delta staged in asynchronous RL, and GPU buffers for active masks and optional post-apply checks. Bottom: in synchronous RL, arriv- ing payloads are decoded into a separate group under assembly, which then joins the queue of 32 resident groups of at most eight decoded payloads, the green cells. Thus the queue plus the assembly group holds at most 264 decoded payloads. Afteropen_apply, the receiver applies queued payloads through the native loader. A full queue makes ingestion wait and slows upstream stages. In asynchronous RL, compressed payloads wait in the dashed host staging area until open_apply. A decoder then feeds the queue while the application drains it in declared order. 256 512 tensors chunks records∅ bucketℓbucketℓ+1 Figure C.3|Chunk and bucket sizes, with heights pro- portional to input bytes and dashed lines at the relay- tree chunk target of 256 MiB and the bucket target of 512 MiB. A tensor larger than both targets forms its own chunk and bucket, since tensors and chunks are never split. Arrows lead from chunks with records to the bucket that counts their full input sizes, so the unchanged chunk (∅) adds nothing to either bucket. + noise round zstd zstd α β BF16 words XORα⊕β overwriteβ 0x3C550xBE800x3D400x4012 0x3C560xBE800x3D410x4012 0x00030x0001 0x3C560x3D41 0.33 MB 0.74 MB Figure C.4|Synthetic codec benchmark. FP32 noise and rounding turn the BF16 inputs훼into훽, and only the changed words enter the XOR and overwrite streams. Sizes are those of Table 2 at 푎 = 10 −4 . synchronous RL, and its receivers row shows the loader applications before finalization. D. Coverage and Payload Format For § 4, 5, and 7.1, this appendix details coverage, the payload format, and payload volume. © 2026 NVIDIA. All rights reserved.25 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale trainingrollout 8 nodes, 4 GB300 each 8 nodes, 8 H100 each S3 1/8 eachfull owner 0 owner 1 sending endpoint receivers construction lower bound transport lower bound t 0 t 1 finalization Figure C.5|Top: latency testbed and transport-only full-checkpoint reference of § 8.1, with three of the eight nodes on each side drawn. Every training node uploads an eighth of the checkpoint to S3 concurrently, and every rollout node then downloads all of it in par- allel across the dashed cluster boundary. Each node’s cross-cluster flow is up to 5 Gbps. Bottom: timed win- dow of one refit. Thin boxes are intervals, and thick strips are their unions, which count overlaps once and exclude gaps. The longest union over owners gives the construction lower bound, and the longest union over sending endpoints gives the transport lower bound. XOR and affine coverage. The XOR/overwrite split follows the mapping classes of Appendix A.1 and the loader paths of Appendix A.2. On the default BF16 paths, each owned shard of an affine tensor contributes one XOR item when it has changed ele- ments and its path is representation-preserving with no overlapping writes. Each required residual ten- sor emits an overwrite item when its converted bits change. Figure D.1 summarizes the rule and shows why scaled and cast values need overwrites. yesyesyes nonono affine task? representation- preserving path? no overlapping write? XOR overwrite record 0x3F020x3F010x0003 0x3F430x3F420x0001 v+1vmask source ⊕= converted ⊕= ×1.5, cast̸= Figure D.1|Top: encoding of a record in the default mixed mode. All conditions are decided in advance for each mapping class and loader path. Bottom: a source XOR mask under a conversion that scales by 1.5 and casts to BF16. The example element’s mask is 0x0003, but the mask of its converted values0x3F42 and0x3F43is0x0001, so the source mask cannot update the converted bits, which need an overwrite. Direct-projection coverage. Table D.1 shows that affine mappings cover all but 185 canonical ten- sors of the Nemotron-3-Ultra-550B-A55B checkpoint, which has no custom export hook. Routed experts use qualifying per-expert Auto mappings. Of the 185 residual tensors, 183 have mappings outside the supported classes: the Mamba input projections and convolutions, and attention Q/K/V, including those of the multi-token-prediction layer. The re- maining two residual tensors are the embedding and outputhead. For Qwen3-30B-A3B, Table D.2 counts 18,721 of the 18,867 canonical tensors as affine. Its residual tensors are attention Q/K/V and the embedding and output head. For the Qwen3-30B-A3B checkpoint of Table D.2, when every tensor and shard changes, XOR covers 18,721 of 18,867 items at tensor-parallel and expert- tensor-parallel degree one, 74,161 of 74,307 at degree four, and 148,081 of 148,227 at degree eight. Each of the 18,432 expert tensors and 48 attention output pro- jections splits into푡shards at degree푡, while the 48 router and 193 normalization tensors stay whole, giv- ing 18,480푡+ 241 XOR items (Figure D.2). The XOR share of weight bytes stays 96.3% because a higher degree only splits the same weightsinto more shards. For the Nemotron-3-Ultra-550B-A55B checkpoint of Table D.1, when every tensor and shard changes, Table D.1|Affine and residual canonical tensors in Nemotron-3-Ultra-550B-A55B. The Bytes col- umn gives shares of the 1,121 GB checkpoint, with one share for the three residual rows together. Other affine weights include shared experts, routers, latent-MoE projections, attention output projec- tions, normalization, and the output projections and state parameters of Mamba. Weight typeTensors Bytes Affine Routed expert up/down50,176 93.86% Other affine weights662 3.13% Residual3.01% Attention Q/K/V39 Mamba input projection and convolution 144 Embedding and output head2 Table D.2|Affine and residual canonical tensors in Qwen3-30B-A3B. The Bytes column gives each weight type’s share of the 61.06 GB checkpoint. Weight typeTensors Bytes Affine Expert gate/up/down18,432 94.95% Attention output projection48 1.32% Router48 0.04% Normalization193 <0.01% Residual Attention Q/K/V144 1.65% Embedding and output head2 2.04% © 2026 NVIDIA. All rights reserved.26 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale expert, attention output t shardst items router, norm whole1 item Figure D.2|XOR items of Qwen3-30B-A3B at tensor-parallel and expert-tensor-parallel degree푡, drawn for푡= 4. Each expert tensor and atten- tion output projection splits into푡shards, one item each, while each router and normalization tensor stays whole as one item. 403,869 of the 404,054 items are affine at tensor- parallel and expert-tensor-parallel degree eight. Each of the 50,176 routed expert up/down tensors splits into푡 e shards at expert-tensor-parallel degree푡 e , and 257 of the 662 other affine tensors split into푡shards at tensor-parallel degree푡, while the remaining 405 stay whole, giving 50,176푡 e + 257푡+ 405 affine items and 185 residual items.Everyaffine item takes the XOR path except those of the few Mamba state parameters that the native loader transforms, so the XOR share of weight bytes stillrounds to the 97.0% affine share. Payload format. Each payload carries attempt, owner, and payload identifiers, flat location and value buffers, and per-item metadata: tensor name, shape, dtype, encoding, location type, and offsets into both buffers. Contiguous locations use a start index, and the number of values gives the run length. Scattered locations use gaps in the small- est suitable integer type. A BLAKE2b-128 digest [1] covers each compressed payload body. Figure D.3 illustrates the payload format with two ten- sors. The changed elements of tensor A form one con- tiguous run, so its metadata row stores a start index instead of location offsets and adds nothing to the loca- tion buffer. Tensor B’s scattered positions are stored as gaps. Both rows index the shared value buffer. Payload volume. To relate payload bytes to the element change rate (Figure D.3), let Ω be the full canonical index set and푁=|Ω|. Because ownership is disjoint,푁 src = ∑︀ 푇 |퐽 푇 |counts unique source el- ements and푁 chg = ∑︀ 푇 |퐶 푇 |counts unique changes. The element change rate is푠 src =푁 chg /푁 src . The emitted destinations form퐷= ⋃︀ 푇∈풯 aff 휋 푇 (퐶 푇 )∪ ⋃︀ 푢∈풰 req 퐷 푢 . The destinations with changed canonical bits form퐷 chg =푑∈Ω| 퐼(푊 푣+1 ) 푑 ̸=퐼(푊 푣 ) 푑 , and complete change detection makes it a subset of퐷. The canonical change density is푠=|퐷 chg |/푁. The element change rate and the canonical change density are equal when a representation-preserving bijection maps source elements to canonical elements. Before zstd, the payloads of one attempt together carry metadata, run- and gap-coded locations, and payload Arunstart 3values [0, 6) Bgaps[0, 3)values [6, 9) 222 AAAAAABBB 0123456789 tensor A tensor B 222 zstd, BLAKE2b-128 metadata locations values by dtype metadatalocationsvalues sourcecanonicalpayloads s src = N chg /N src D ⊇ D chg s =|D chg |/N zstd→ V payload moved: (1+N node )V payload Figure D.3|Top: payload format. To decode gaps, start at−1 and add each gap plus one: gaps 2, 2, 2 yield positions 2, 5, 8. Values have the width of their tensor’s dtype. Bottom: quantities behind payload volume. Owners find푁 chg changes among 푁 src source elements, which gives the element change rate푠 src . The emitted destinations퐷contain the changed destinations퐷 chg , whose share of the푁 canonical elements is the canonical change density푠. When a mapping is not a representation-preserving bijection,퐷can also hold an unchanged destination, drawn outlined. Payloads carry metadata, locations, and values, and with푁 node rollout nodes, object storage moves about (1+푁 node )푉 payload per attempt in uploads and downloads. one value per record. Let퐻= ∑︀ 푑∈Ω 푏 푑 be the canon- ical checkpoint size, where푏 푑 is the width of element 푑in bytes. With one record per emitted destina- tion, the value bytes total ∑︀ 푑∈퐷 푏 푑 , which at uniform width is|퐷|퐻/푁, or푠퐻when퐷=퐷 chg . Here퐷 counts unique destinations. Each additional over- write record for a destination adds another푏 푑 bytes. More scattered changes need more location bytes, and more items need more metadata. After zstd compresses the locations and values, the attempt’s payloads total푉 payload . On the object-storage trans- port, each attempt moves about (1 +푁 node )푉 payload of upload and download traffic when푁 node rollout nodes download every payload. On the relay tree without retransmission, each payload crosses the clus- ter boundary once per attempt, so cross-cluster traffic is about푉 payload , and the root forwards each payload over local links to every rollout node. E. GRPO Experiment Settings This appendix details the GRPO settings for the element-change measurements in § 2.2 and for the training runs in § 8.4. © 2026 NVIDIA. All rights reserved.27 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale Element-change measurements. These mea- surements follow the agentic SWE RL recipe for Qwen3 [24]: BF16 weights, FP32 AdamW state, con- stant learning rate 10 −6 , and no warmup, weight decay, or KL penalty. The six measured models are Qwen3-30B-A3B, Llama-3.2-3B,Gemma-3-4B, and Qwen2.5 at 0.5B, 1.5B, and 7B. Training runs. These runs use the first stage of the same recipe, withswe1.jsonlfrom the Nemotron- RL-Super-Training-Blends dataset. This stage re- wards single-step tool calls for matching the argu- ments of the expert action. Each GRPO step uses 64 prompts with 8 responses each and a batch size of 512. Withratio_clip_min=0.2andratio_clip_ max=0.28, probability ratios are clipped to [0.8,1.28]. The loggedtrain/gen_kl_errorestimates the per- token KL divergence between rollout and training policies. Figure E.1 shows the receiver kills. F. System Comparison This appendix gives the evidence behind each mark of Table 1 under requirements G1–G5 of § 2.3, and the same analysis for the slime framework. NeMo-DCR. The design meets all five require- ments. § 6.2 and Appendix A establish G1, and the dense-refit comparisons of § 8.1 confirm it. For G2, the payload volumes of Tables 3 and 4 scale with the changed set. For G3, placement stays in the native loader with no model-specific logic, as § 6.1 and 8.1 describe. For G4, both transports deliver without a cross-cluster collective, as § 7 describes, and the latency runs of § 8.1 use them between clusters that share no InfiniBand or EFA route. Payload dedupli- cation, overwrite recovery, and joint commit meet G5, as § 6.3 and Appendix B describe, and with receivers killed mid-refit, both transports follow dense NCCL’s mean reward and KL trajectories in Figure 9. Bit-exact (G1). Approaches A and B fully meet G1, and approach D avoids arithmetic reconstruc- tion but meets G1 only partly. NCCL-Reshard instances 02550step Figure E.1|Receiver kills in one NeMo-DCR run of § 8.4, with an illustrative choice of instances. Every five steps, a randomly chosen vLLM instance is killed mid-refit at a cross and restarted five steps later, at a dot, where it restores the committed version. and checkpoint-engine send every weight rather than a delta, so they preserve the target bits [20,23]. TensorHub copies its published buffers into the serv- ing runtime’s registered tensors, which yields exact target bits because the published buffers hold the weights in the layout and dtype of the registered tensors [40]. PULSE’s Appendix H.6 proves exact reconstruction of a tensor from a correct base [19], but extending the proof through a native loader re- quires conditions on the loader’s transformations and storage writes. In verl, NaN masks in intercepted copies protect unchanged positions, and a round-trip test checks that the model’s generated outputs are bit-identical [36], but the loader’s supported trans- formations are not stated. SparseRL-Sync patches the loader’s copy with a NaN-masked sparse scatter and reports tensor-by-tensor bitwise agreement with a dense refit, but does not specify supported trans- formations or interception of every storage write [11]. NeMo-DCR specifies both in the loader conditions of § 6.1 and Appendix A.2, and Appendix A proves that under these conditions its refits produce the same bits as a dense refit. In approach C, arithmetic reconstruction of updated or pre-update values can introduce rounding errors. ROSE sends arithmetic differences [6]. AuroraRL re- ports lossless reconstruction by scatter-adding deltas but does not state when this reconstruction is bit- exact [30]. AReaL-DTE [26] reconstructs pre-update weights by inverting AdamW and compares BF16 bit patterns after aligning them to a common layout. Its authors note rounding in fused AdamW implementa- tions, so comparison against these reconstructed pre- update weights can miss a changed element. NeMo- DCR instead compares against stored values retained in the committed source baseline. Delta (G2). Approaches A and B move every weight byte, while the delta schemes in approaches C and D scale with the changed set. PULSE, SparseRL- Sync, and verl in approach D send absolute overwrites as indices and values of changed elements, with BF16 values in PULSE and verl. The verl framework de- tects changes on each training rank’s shard and maps them to full-tensor coordinates through block place- ments, and it supports conversions only when they permute elements [35,36]. AReaL-DTE sends target values of detected changes, and AuroraRL packages each step as a BF16 delta checkpoint. Native (G3). The serving runtime performs all weight placement in checkpoint-engine, SparseRL- Sync, and verl, and only part of it in NCCL-Reshard and TensorHub. In checkpoint-engine, the serving © 2026 NVIDIA. All rights reserved.28 NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale runtime also chooses which weights to copy. NCCL- Reshard copies expert feed-forward shards directly into vLLM parameters using its own placement meta- data and hooks. Only the remaining parameters pass through the native loader, so native placement is par- tial. TensorHub copies its published buffers into the serving runtime’s registered tensors without interpret- ing their layout. Yet weights are converted to the rollout layout before publication, not by the native loader, so native placement is only partial [40]. Most delta systems place updates themselves. ROSE uses shard-aware routing to receiver shards, and Auro- raRL writes directly into parameter storage. PULSE writes values at the transmitted indices into receiver tensors that share the sender’s layout. AReaL-DTE remaps updates into rollout layouts at the receiver and uses the native loader only as a fallback, so its native placement is partial. Decoupled (G4). NCCL-Reshard, SparseRL-Sync, and verl each use a collective that spans training and rollout. The checkpoint-engine system runs along- side serving runtimes, loads full weights onto GPUs from disk or the training engine, and broadcasts them among its workers. Its description mentions no cross- cluster transport [20]. PULSE, AReaL-DTE, ROSE, AuroraRL, and TensorHub transfer weights through replication, relays, or shared storage without a cross- cluster collective. TensorHub pipelines its full-weight replication, placing one replica in each remote datacen- ter before replicating locally. AuroraRL streams each delta checkpoint through one seed actor per region. Recovery (G5). No existing system in Table 1 fully meets G5. ROSE, SparseRL-Sync, and verl describe no recovery protocol, although verl does checksum each broadcast payload. For NCCL-Reshard and checkpoint-engine, repeating a dense refit can restore weights, but the cited descriptions do not specify a complete protocol for partial failures and policy com- mit. TensorHub provides transactions, invalidates in- complete replicas, and recovers transfers from another source [40], but it coordinates version selection within each model-parallel group rather than committing across all required receivers. AReaL-DTE commits a version manifest and leaves partially applied versions to its control plane, without describing their repair. AuroraRL checks the version a delta applies to, uses leases for fault handling, stages deltas before activa- tion, and lets lagging actors replay the delta chain, but does not describe repair after partial writes. PULSE receivers advance independently, without com- mitting the source baseline and the required receivers’ policy version together. PULSE’s Algorithm 5 checks the local version before decoding [19], and a receiver applies consecutive deltas directly. PULSE also keeps periodic full checkpoints called anchors. After missed versions, the receiver restores a ready anchor and re- plays subsequent deltas in order. Ready markers pre- vent use of incomplete uploads, and a full-state hash checks each reconstruction before the new weights are returned. A mismatch triggers anchor recovery. Absolute overwritescan be repeated safely, and an- chor restoration replaces a partially applied state. slime. The slime framework’s delta weight sync pub- lishes byte-level XOR or overwrite deltas of the gath- ered Hugging Face tensors to a shared filesystem [33]. Each rollout node patches a full local checkpoint, ver- ifies per-tensor checksums, and reloads the checkpoint from disk through the native loader. This design meets G1–G4, but every refit reads the full model. It does not fully meet G5: the trainer advances its source baseline before any rollout node applies the published version, and no commit ties the baseline to the version that the rollout nodes hold. © 2026 NVIDIA. All rights reserved.29