Paper deep dive
Zephon: Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines
Maximilian Böther, Josh Wills, Ties Robroek, Sonnet Xu, Paul Burstein, Daniel Zayas, Cody Blakeney, Siddharth Joshi, Haoli Yin, Rishabh Adiga, Haakon Mongstad, Luke Merrick, Pratyush Maini, Ari Morcos, Matthew Leavitt, Ana Klimovic, Bogdan Gaza
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deterministic data loading is important for foundation model development: model researchers need confidence that differences they observe across costly ablations are caused by the parameter they changed rather than non-determinism in the training data sequence. The data loader must provide elastic determinism, i.e., a deterministic sequence of global training data batches despite changes to the GPU topology across runs (e.g., due to GPU scarcity), frequent checkpoint-resume cycles, and different data processing execution backends. Achieving this is difficult because modern foundation model data pipelines tokenize, pack, and mix samples online, introducing stateful n-to-m transformations that break sample indexing. Existing data loaders largely assume indexable 1-to-1 pipelines, and the common workaround of offline materialization is expensive and, for some modalities such as video, infeasible. We present Zephon, a data loader for foundation models that supports online, stateful pipelines while providing elastic determinism and efficient resumption from checkpoints. It partitions the global stream into topology-independent lanes, serializes ordering decisions while parallelizing stateless work on interchangeable backends, and checkpoints only bounded in-flight state so recovery cost does not grow with training progress. We evaluate Zephon on text and vision-language workloads and show that it achieves competitive throughput while providing a combination of guarantees that no existing loader offers for online, stateful pipelines.
Tags
Links
- Source: https://arxiv.org/abs/2610.03087v1
- Canonical: https://arxiv.org/abs/2610.03087v1
Trouble viewing inline? Open PDF directly →
Full Text
113,415 characters extracted from source content.
Expand or collapse full text
@bx@sep @bx@sep Zephon : Elastic Determinism for Online, Stateful Foundation Model Data Loading Pipelines Maximilian Böther Affiliation: DatologyAI & ETH Zürich Note: Correspondence to mboether@datologyai.com , Josh Wills Affiliation: DatologyAI , Ties Robroek Affiliation: DatologyAI & ITU Copenhagen , Sonnet Xu Affiliation: Stanford University Note: Work done while at DatologyAI. , Paul Burstein Affiliation: DatologyAI , Daniel Zayas Affiliation: DatologyAI , Cody Blakeney Affiliation: Arcee AI , Siddharth Joshi Affiliation: DatologyAI , Haoli Yin Affiliation: DatologyAI , Rishabh Adiga Affiliation: DatologyAI , Haakon Mongstad Affiliation: DatologyAI , Luke Merrick Affiliation: DatologyAI , Pratyush Maini Affiliation: DatologyAI & CMU , Ari Morcos Affiliation: DatologyAI , Matthew Leavitt Affiliation: DatologyAI , Ana Klimovic Affiliation: ETH Zürich and Bogdan Gaza Affiliation: DatologyAI Abstract. Deterministic data loading is important for foundation model development: model researchers need confidence that differences they observe across costly ablations are caused by the parameter they changed rather than non-determinism in the training data sequence. The data loader must provide elastic determinism, i.e., a deterministic sequence of global training data batches despite changes to the GPU topology across runs (e.g., due to GPU scarcity), frequent checkpoint-resume cycles, and different data processing execution backends. Achieving this is difficult because modern foundation model data pipelines tokenize, pack, and mix samples online, introducing stateful n-to-m transformations that break sample indexing. Existing data loaders largely assume indexable 11-to-11 pipelines, and the common workaround of offline materialization is expensive and, for some modalities such as video, infeasible. We present Zephon , a data loader for foundation models that supports online, stateful pipelines while providing elastic determinism and efficient resumption from checkpoints. It partitions the global stream into topology-independent lanes, serializes ordering decisions while parallelizing stateless work on interchangeable backends, and checkpoints only bounded in-flight state so recovery cost does not grow with training progress. We evaluate Zephon on text and vision-language workloads and show that it achieves competitive throughput while providing a combination of guarantees that no existing loader offers for online, stateful pipelines. †authors: . PVLDB Reference Format: PVLDB, 20(1): X-X, 2027. doi:X.X/X.X † This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing info@vldb.org. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. 20, No. 1 ISSN 2150-8097. doi:X.X/X.X 1. Introduction [width=]svg-inkscape/fig_8a_standard_loader_small_svg-tex.pdf_tex (a) Standard data loader [width=]svg-inkscape/fig_8b_zephon_small_svg-tex.pdf_tex (b) Zephon Figure 1. The effects of elastic determinism: a 1 B text model is trained on 16 GPUs, checkpointed at step 150, and resumed on 8, 32, and 64 GPUs while the original run continues uninterrupted, causing loss variance between the runs when not using Zephon . The barcode below each plot shows the per-step batch SHA-256 hash of the 8-GPU continuation: blue shows matching batches while red shows differing batches.TODO! A fundamental principle in experimental research is isolating variables, i.e., changing one thing at a time, as stressed by early systems work such as Saltzer and Gintell (1969). If two experiments differ only in a single parameter while the rest of the system behaves deterministically, any change in outcome is attributable to that parameter alone. Deterministic infrastructure is especially important for modern machine learning (ML) systems. During foundation model development, practitioners run a series of ablations (prototype runs designed to isolate the effect of individual interventions, e.g., changing the data mix) before using the best-performing configuration for the large-scale hero run (Groeneveld et al., 2025; Hoffmann et al., 2022; Leavitt, 2022; Gadre et al., 2025). Ablations, however, are costly, even at smaller scales. For example, consider a 400 B-total/20 B-active Mixture-of-Experts hero run trained on 20 T tokens. Even an ablation 1.5 orders of magnitude smaller in total training compute requires roughly 30 k to 50 k H100 GPU-hours, costing approximately $250 k at early-2026 cloud prices. Achieving reproducible ablations requires deterministic training infrastructure (Zhuang et al., 2022). One part of this infrastructure is the last-mile data loading. The data loader transforms an encoded representation of training data to tensors that the training framework passes to the model (Mohan et al., 2021b; Zhao et al., 2022; Murray et al., 2021; Graur et al., 2022; Graur et al., 2024; Robroek et al., 2026; Kuchnik et al., 2022; Isenko et al., 2022). Without a deterministic data loader, the data ordering changes. This is problematic as the training loss, the common proxy for success of an ablation (Kaplan et al., 2020; Hoffmann et al., 2022; Gadre et al., 2025), is sensitive to such changes in data order (Wang et al., 2025a; Biderman et al., 2023). Any variation in the input data sequence confounds the results of ablations and impedes data debugging. When a loss spike occurs, engineers must be able to inspect the offending sample(s). Determinism with respect to a fixed cluster configuration is not sufficient. In ML clusters, GPU resources are scarce. It is standard practice to prototype runs intended for large GPU counts on smaller allocations, using gradient accumulation to simulate the target global batch size (Mosaic ML Team, 2022b; Wan et al., 2025a; Amazon Web Services, 2026; Li et al., 2023; Kang et al., 2025). Moreover, cluster priorities may shift throughout the life of a training job. A more urgent job may preempt part of an allocation, forcing a run to resume on fewer nodes. Conversely, when additional nodes become available, a job should be able to scale up to finish sooner. In all these cases, the data loader must produce identical global training batches regardless of the GPU topology. Mosaic Streaming calls this property elastic determinism (MosaicML, [n. d.]). Figure 1 illustrates how, without elastic determinism, the loss curve varies when a training run is resumed on a different GPU count after a fault. We use a standard 1 B training setup in Torchtitan (sequence length 4 096, AdamW, peak LR 7.908⋅10−47.908· 10^-4 with a WSD schedule, local batch size 7, global batch size 448), and use a shared initial seed checkpoint. With Zephon , every topology receives the same global batches, confirmed by identical SHA hashes, causing the loss curves to overlap. Beyond elastic determinism, the data loader must also be able to resume efficiently from a checkpoint (Mosaic ML Team, 2022a; Mohan et al., 2021a; Wan et al., 2025a; Wan et al., 2025b; Wang et al., 2023; Strati et al., 2025). Resumptions are frequent and arise from both expected causes (e.g., preemption by higher-priority jobs or planned maintenance) and unexpected ones, such as hardware failures. At scale, hardware failures are common, e.g., Kokolis et al. (2025) project a mean-time-to-failure of 1.8 h for 16 k GPU training jobs. As data loader state is part of every checkpoint (Wan et al., 2025a), if restoring it requires replaying the full pipeline from the start, recovery time grows with training progress, and expensive GPUs sit idle waiting for data (Mohan et al., 2021b). However, implementing elastic determinism and efficient resumption for modern foundation model input data pipelines is hard because they consist of stateful operations with n-to-m characteristics (An et al., 2025; Wang et al., 2025b; Zhao et al., 2026; Chen et al., 2024a). In pre-training, tokenization splits documents into multiple subsequences (Sennrich et al., 2016; Kudo and Richardson, 2018), packing maintains in-flight bins to fill fixed-length slots (Krell et al., 2021; Zhao et al., 2024b), and data mixing reorders samples (Zhao et al., 2026; Böther et al., 2026). Post-training shares these characteristics: supervised fine-tuning packs chat-templated conversations with aligned response-only loss masks, while online reinforcement learning filters and regroups variable-cardinality rollouts. Whereas in a 11-to-11 pipeline, progress is a sample index and resuming means looking up the last accessed sample, in a stateful n-to-m pipeline, logical progress is spread across in-flight operator state, partially formed outputs, and reordered records. Such pipelines are thus not indexable. Restoring the exact state without fully replaying all data and ensuring it remains identical across changes in parallelism, execution backend, or cluster topology is the core challenge. Existing data loaders such as tf.data (Murray et al., 2021) or Cedar (Zhao et al., 2024a) optimize pipelines dominated by stateless 11-to-11 mappings (e.g., decode a JPEG, flip an image). Mosaic Streaming provides elastic determinism for such indexable datasets. Grain supports deterministic resumption for stateful pipelines under fixed execution configurations and elastic determinism for indexable pipelines (Ritter et al., 2023). While stream processing engines such as Flink (Carbone et al., 2015b) or Naiad (Murray et al., 2013) support stateful pipelines, their correctness model targets properties such as exactly-once processing, not a canonical output sequence of training batches that is preserved across topology changes. To our knowledge, no existing system combines support for stateful n-to-m pipelines with elastic determinism and efficient resumption. The common workaround is to materialize datasets in fully preprocessed form, i.e., tokenized, mixed, and packed into ready-made training batches (Shoeybi et al., 2019; NVIDIA, 2026; Böther et al., 2026; An et al., 2025; Wang et al., 2025b; Chen et al., 2024b; Zhu et al., 2025; NVIDIA, 2024), using offline systems like Spark (Zaharia et al., 2016), Beam (Akidau et al., 2015), or Ray (Moritz et al., 2018), reducing the pipeline to an indexable 11-to-11 mapping. But model developers make frequent changes to the data pipeline (e.g., test a new filter), and re-materializing petabyte-scale corpora in offline systems on every change is prohibitively expensive in both storage and compute (Böther et al., 2026). For some data modalities, such as video, even materializing raw tensors is fundamentally not viable due to the massive inflation in size (Zhao et al., 2026; von Werra et al., 2026). Labs that work with such modalities resort to building proprietary loading infrastructure (Anyscale, 2024; Crosley, 2026; Perceptron AI, 2025). More broadly, the lack of a system that meets these requirements forces model teams to repeatedly reinvent data loaders, burning time and resources that could be better spent on training models and doing research. The ML community needs an online data loading system that supports stateful n-to-m operators while providing elastic determinism and efficient resumption. Since data loading pipelines often consist of compute-intensive operators, they can bottleneck training and leave expensive GPUs idle waiting for data (Mohan et al., 2021b). Decoupling data processing semantics from the execution backend is important for optimizing performance across configurations, which can range from disaggregated data preprocessing (Audibert et al., 2023; Zhang et al., 2025; Anyscale, 2024; Zhao et al., 2026) to multiprocessing or multithreading (with the removal of Python’s GIL (Hira et al., 2025)) on training nodes. The choice of execution backend must not affect the training data sequence. In this paper, we co-locate the data loader on the training node, as this is the standard industry setup that we observe in Datology’s customer deployments. We present Zephon , to our knowledge, the first data loader to offer elastic determinism and efficient resumption for online, stateful n-to-m foundation model data pipelines. Zephon carefully manages the pipeline state, rather than representing it as a sample index. Our contributions are as follows: (1) Zephon introduces accumulators which serialize stateful ordering decisions in a single thread while allowing stateless transformations to run in parallel on flexible execution backends called runners. It generalizes Mosaic Streaming’s elastic determinism to online, stateful n-to-m pipelines using a lane abstraction, a logical sub-stream that produces a slice of the training data regardless of the underlying topology. (2) Zephon checkpoints stateful data loading pipelines and restores them with bounded cost. For this, it performs per-offset contributor tracking, a mechanism that handles operators which reorder, merge, and spawn samples. (3) We release Zephon as an open-source foundation model data loading system, with a composable builder-pattern interface. It ships with a toolbox of common foundation model operators. 2. Stateful n-to-m data loading pipelines TODO! Figure 2. A non-indexable VLM preprocessing pipeline. Operators vary in cardinality (below the operator) and statefulness (blue solid/yellow dotted). Arrow color/pattern indicates the source, thick purple arrows are packed samples. TODO! Consider a VLM training corpus of image-caption pairs, interleaved documents, and text-only samples, as used for the M1 family of models (McKinzie et al., 2024). Figure 2 shows a representative preprocessing pipeline (e.g., see the Qwen-VL model family (Wang et al., 2025b; Bai et al., 2025b; Bai et al., 2025a) and Eagle 2 model (Li et al., 2025)). The operators read records, decode text and images, plus, if necessary, apply vision transforms, apply the chat template and tokenize the sequence, shuffle, pack, and construct batches. The operators in this pipeline differ in cardinality. Decoding and augmentation are 11-to-11: one input record yields one output. Tokenization with splitting is 11-to-n: a long document becomes multiple subsequences. Packing is n-to-11: several short sequences merge into one training example. Filtering is 11-to-00: records that fail a quality check are dropped. Orthogonal to cardinality, many operators are stateful. Packing maintains partially filled bins across inputs. Shuffle operators hold a buffer of pending samples. Mixture enforcement tracks per-component counters to preserve dataset ratios. Because of this state, the next output depends not only on the current input but on the history of prior inputs. We call a pipeline indexable if the source records contributing to its output at training step t can be located from t without evaluating the preceding input prefix. n-to-m cardinality alone already breaks indexability. When tokenization splits documents into a data-dependent number of subsequences, indexability is lost: the mapping from training step to source record depends on the length of all prior documents. Statefulness compounds the problem, since resuming from an arbitrary input offset requires reconstructing operator state. Any change to the execution that alters how state accumulates (worker count, backend, cluster topology) silently changes the output sequence. 3. Requirements overview A data loader for non-indexable pipelines must meet the same guarantees as for indexed pipelines: high throughput, composability, deterministic output, and efficient recovery. However, the mechanisms indexed pipelines rely on (sample offsets, static partitioning, index-based resume) no longer apply. We discuss why stateful, non-indexable pipelines make these requirements harder to satisfy: R1: High throughput and backend portability. Data loading must not bottleneck training (Mohan et al., 2021b). In an indexable pipeline, parallel processing is straightforward using partitioning. In a non-indexable pipeline, output boundaries are data-dependent, so the system must decouple parallel execution from the logical output order. A data loader must keep throughput high and support different execution substrates (e.g., processes/threads) to achieve this throughput across diverse training workloads. R2: Composability. Pipelines differ across teams and modalities. In order to accommodate this, users should be able to compose operators freely and plug in custom transformations, rather than having a fixed, hardcoded pipeline for a single modality. A data loader should be composable and should remain decoupled from any particular file format, storage backend, or training framework. R3: Elastic determinism across execution configurations. Determinism in data loading supports debugging and controlled ablations, removes one confounder when diagnosing silent data corruption (Ma et al., 2025; Gemini Team, 2025), and increasingly matters for compliance (European Parliament and Council of the European Union, 2024). If we view the loader as a stateful dataflow graph, its logical result is the ordered sequence of global batches consumed by training. Because training is order-sensitive, elastic determinism is a correctness requirement: for a fixed logical pipeline, the result must remain identical when execution choices such as operator parallelism or execution substrate change. It must also remain identical as the data-parallel degree changes, at constant global batch size. A data loader must provide elastic determinism even for non-indexable pipelines, where no static sample index exists to remap across topologies. R4: Efficient resume from checkpoint. The data loader state must be checkpointed with the model. For an indexable pipeline, a sample offset suffices. In a non-indexable pipeline, progress is entangled across operator state (bins, buffers, counters) with no compact representation. Naively restoring such state by replaying the pipeline from the start causes recovery costs to grow with training progress. A data loader must therefore support bounded-cost checkpoint recovery regardless of pipeline progress. 4. Zephon ’s architecture and abstractions Figure 3. Zephon ’s high-level architecture. The WorkSource produces chunks for each lane (colored lines), which are converted into samples by the Fetch operator. Each lane holds an independent chunk stream and independent operator states.TODO! We build Zephon , a data loader for training foundation models that addresses these requirements. Zephon separates what to train on from how to process it: a WorkSource defines the training curriculum (which datasets to use, in what proportions, in what order), and a pipeline of composable operators (R2) turns this into training batches. 4.1. User interface Zephon is available as a Python package and supports user-defined functions and arbitrary data payloads. We open-source the code,11 1 https://github.com/datologyai/zephon comprising ca. 20 k LOC, excluding tests. Pipelines are defined via a builder-pattern API: ⬇ ws = StaticMixtureWorkSource( [Dataset.from_path("demo", "s3://example-dataset")], mixture=MixtureSpec("demo": 1.0), chunk_size=1024) pipeline = (Pipeline(ws) .tokenize(tokenizer_id="gpt2", parallelism=4) .pack_sequences(target_length=2048) .ensure_mixture() .batch(microbatch_size=8)) for batch in pipeline: train_step(batch.to_training()) Lines 1–3 define the training curriculum via a WorkSource. Lines 4–8 attach a chain of operators that turn raw data into training batches. We now describe each of these abstractions. 4.2. Component overview The overall data flow is shown in Figure 3. Zephon ’s mechanism for elastic determinism (R3) is the lane. The global data stream is partitioned into a fixed number of lanes configured by the user. Each lane is a logical sub-stream that produces a deterministic, non-overlapping slice of the training data. Each lane produces an independent, deterministic data stream with its own operator state. Lanes generalize Mosaic Streaming’s canonical nodes (Mosaic ML Team, 2022b) to non-indexable, stateful pipelines. The WorkSource. The WorkSource 1 defines the training curriculum, e.g., by indexing the data 0. For each lane, it produces fixed-size chunks of C sample pointers, ordered as ⟨p0,…,pC−1⟩ p_0,…,p_C-1 2. It does not read the sample payloads. Fetch: from pointers to data. Zephon processes samples in a DAG of operators. Every pipeline begins with the fetch operator. For each pointer in the current chunk, it fetches the actual payload from the corresponding file and generates a record carrying it. The record also carries metadata that uniquely identifies it within its lane, used for checkpoint resumption (R4/Section 7). Further processing. After fetch, records flow through the user-defined operator graph 3. Operators compose freely (R2), and the chain can contain arbitrary cardinalities, i.e., records can get filtered, split, etc. After the final operator, Zephon yields the sample to the training loop 4. Operators, not workers. Zephon treats a data loading pipeline as a graph of operators that scale independently, rather than as workers that each run the whole pipeline (as, e.g., in PyTorch’s DataLoader). This matters for two reasons. First, operators have different resource profiles. For example, data fetching is IO-bound, while tokenization is CPU-bound. Per-operator scaling avoids a single performance knob (R1). Second, the degree of parallelism can change the output order in worker-based loaders, coupling a performance choice to training semantics. 4.3. Data model Zephon follows the shard-based approach of data loaders such as Mosaic Streaming (Mosaic ML Team, 2022b) or LitData (Chaton and AI, 2023). A dataset D is a named collection of data at a storage path, e.g., a local directory, an S3 bucket, or a distributed file system prefix. A dataset is split into k≥1k≥ 1 shards =S1,…,SkD=\S_1,…,S_k\, where each shard is a file containing a contiguous slice of the data. A sample is the smallest addressable unit within a shard: the j-th sample in shard SiS_i of dataset D is addressed by the sample pointer p=(,i,j)p=(D,i,j). Zephon is not tied to a single file format; above the format-specific reader layer, the abstraction is always the same (R2): a dataset has shards, and each shard has a known number of samples. 4.4. WorkSource The WorkSource is Zephon ’s abstraction for decoupling the training curriculum from preprocessing. This enables a model engineering team to iterate on downstream operators without changing the data composition, and vice versa to adjust training data without touching preprocessing code. The WorkSource protocol. A WorkSource is an object that produces a deterministic stream of chunks for each lane. Zephon ’s current implementation specifies curricula at the file system level: mixtures reference datasets by storage path. However, the abstraction is not tied to this. Any backend that can translate a curriculum specification into chunks of sample pointers satisfies the protocol. For example, one could implement a WorkSource backed by a centralized sample management server such as Mixtera (Böther et al., 2026), where curricula are expressed as SQL-like queries rather than file paths. Chunk structure. A chunk is a fixed-size sequence of sample pointers drawn from one or more mixture components (domains). Given components d1,…,dmd_1,…,d_m with target weights w1,…,wmw_1,…,w_m, each chunk contains pointers from each component proportional to its weight. Iteration order within a chunk is controlled at two levels: inter-component, where pointers are interleaved across components in a deterministic weighted round-robin, and intra-component, where the WorkSource controls the order in which samples are yielded. For intra-component ordering, the WorkSource can shuffle samples within and across shards before forming and assigning chunks to lanes. Because the curriculum is defined at chunk granularity, this approach naturally supports dynamic curricula (Chen et al., 2023; Böther et al., 2026; Jiang et al., 2024): a WorkSource can adjust weights in between chunks. 5. Execution model TODO! Figure 4. Zephon ’s runtime architecture. The engine feeds chunks into a pipeline compiled into stages, each assigned a runner. Within each stage, a single-threaded accumulator dispatches microbatches to parallel stateless workers.TODO! Figure 4 shows how Zephon executes pipelines to achieve high throughput (R1). When training starts, Zephon compiles the operator graph into a plan that groups operators into stages, each assigned a runner. We describe the engine that orchestrates the data flow, the accumulator model that enables parallel execution of stateful operators within each stage, and the runners that implement worker backends. 5.1. Engine and pipeline execution The engine drives the compiled pipeline. Between stages, execution is pull-driven: stages are chained as Python iterators, and the training loop at the tail drives execution by requesting the next batch. Within a stage, execution is push-driven: a dedicated pump per operator (Section 5.2) actively pushes records through its accumulator and workers. Prefetch buffers at multiple levels overlap computation with data movement. From WorkSource to stages. For each lane, the engine requests work chunks from the WorkSource A. The engine iterates over the pointer stream of each chunk and provides them to the first stage B. When a chunk is exhausted, the next chunk is requested. Inter-stage communication. Stages are connected by configurable bounded queues C. When a downstream stage is slow, its input queue fills up, and the upstream stage blocks on insertion. This provides backpressure that prevents unbounded memory growth. Consumer handoff. Data from all lanes (Section 4.2) flow through the system. At the tail of the pipeline, the lane multiplexer D collects fully processed data from all lanes. It buffers per-lane output and emits data in deterministic round-robin order across lanes. Lanes may produce at different rates (e.g., if one lane’s samples are more expensive to process). The lane multiplexer buffers faster lanes while waiting for the next lane in the round-robin. This is key to elastic determinism (Section 6). The training loop consumes batches through a standard Python iterator, with a prefetch buffer to overlap data loading with GPU computation. 5.2. Stage execution and the operator pump Operators like image decoding are expensive and must run in parallel, without blocking the training process. Zephon supports this, even for stateful operators, by splitting each operator into a stateful accumulator and stateless workers, orchestrated by a pump. Stage structure. A stage with k operators spawns one feeder E and k pumps, connected by bounded queues. The feeder runs in a dedicated thread, pulling records from upstream and pushing them into the first operator’s input queue. Each operator has a dedicated pump F running in its own thread. The pump processes records through the operator-specific accumulator and parallel workers G, then pushes results into the next operator’s input queue, or, for the last operator, into the stage output queue. Each stage is therefore simultaneously consumer and producer: it pulls from upstream via the feeder and exposes an iterator downstream, while internally all forwarding is push-driven. All queues are bounded, so backpressure propagates end-to-end. Between operators, a slow consumer fills its input queue, blocking the upstream pump; between a pump and its workers, bounded in-flight queues stall the pump. The pump loop. Each pump runs the loop in Algorithm 1. It pulls microbatches from its input queue and feeds them to the operator-specific accumulator. A microbatch is a small group of records processed and forwarded as a single unit, amortizing per-record overhead (queue operations, serialization, cross-process transfer). The accumulator runs within the pump and is stateful: it receives incoming microbatches and reorganizes them as needed for the operator. For example, a packing accumulator holds partially filled bins until they reach the target length, and emits a microbatch of samples that should be packed together. The accumulator may not mix records across lanes. When the accumulator emits a microbatch, the pump dispatches it to a pool of workers that apply the operator’s transformation. Workers may complete out of order; a reorder buffer reassembles results using the sequence number assigned at dispatch. Dispatch and DrainResults are operations specialized by the runner. Algorithm 1 Pump loop for a single operator. Input: operator state s, worker backend, output queue qoutq_out 1 while ←s.input_queue.get()≠Stopmb← s.input\_queue.get()≠ Stop do 2 DrainResults(s, qoutq_out); // reorder+forward 3 ←s.accumulator.push()ready← s.accumulator.push(mb); 4 foreach microbatch b∈b do 5 s.seq←s.seq+1s.seq← s.seq+1; 6 Dispatch(b, s.seqs.seq, worker backend); 7 DrainResults(s, qoutq_out); // keep workers fed 8 s.accumulator.flush()→s.accumulator.flush()→ dispatch remaining; 9 drain all in-flight results into qoutq_out; 10 qout.put(Stop)q_out.put( Stop); 5.3. The runner Each stage is assigned a runner that controls the execution of operators. A runner determines how worker functions are executed and thus provides the interchangeable execution substrate (R1). Currently, Zephon supports a process and a thread runner that co-locate data loading on the same node as the training process. Disaggregated data preprocessing that moves preprocessing operations to separate CPU nodes is mostly used in recommendation systems (Zhao et al., 2022) and proprietary generative video models (Anyscale, 2024). Mainstream training frameworks such as Torchtitan and Megatron all co-locate the data loader. Even systems built for disaggregation, such as Pecan, co-locate by default and offload only when profiling identifies a local-CPU bottleneck (Graur et al., 2024). We leave a detailed treatment of the challenges of providing elastic determinism under disaggregation to future work. 5.3.1. The process runner Python’s Global Interpreter Lock (GIL) blocks threads from running CPU-bound code in parallel. Zephon ’s process runner therefore executes workers in separate processes. Dispatch (line 1 of Algorithm 1) serializes the microbatch via pickle and sends it over a cross-process queue. DrainResults (line 1) deserializes results from a corresponding return queue. Operators and IPC. Process-based execution in an operator model is more challenging than in a monolithic-worker model as used by, e.g., PyTorch’s DataLoader. At startup, each operator’s stateless transformation must be serialized and sent to its worker processes. At runtime, every microbatch crosses a process boundary on each dispatch and return, unlike monolithic-worker systems, where only the final output leaves the process. Operator-based loaders like Cedar (Zhao et al., 2024a) address this by fusing adjacent operators to reduce cross-process transfers. Zephon minimizes transfer cost through shared-memory coalescing. IPC and shared-memory coalescing. The process runner’s queues use OS pipes, where serializing large payloads is costly and can saturate pipe buffers. PyTorch reduces this cost for tensors by placing each tensor in a POSIX shared-memory (shm) segment. However, per-tensor allocations and system calls remain expensive, many tensors can exhaust the per-process file-descriptor (FD) limit, and non-PyTorch payloads still traverse the pipe. Zephon addresses these limitations by coalescing all large payloads in a microbatch before enqueueing. It groups tensors by dtype, allocates one contiguous shm region per dtype, and copies each tensor into its slice; large byte buffers and NumPy arrays are packed by the same mechanism. For these payloads, the pipe then carries only lightweight metadata (shapes, offsets, and shm handles), allowing the receiver to reconstruct them as zero-copy views. Coalescing thus reduces the number of segments from one per payload to one per dtype per microbatch, cutting allocation overhead and FD pressure while extending shared-memory transport beyond PyTorch tensors. 5.3.2. The thread runner Thread-based execution avoids the serialization and shared-memory management complexity of the process runner entirely. Threads share the same address space and achieve true parallelism for operators implemented as C extensions or that otherwise release the GIL, as pioneered by Spdl (Hira et al., 2025). Alternatively, free-threaded Python (3.13t/3.14t) removes the GIL, making threads viable for all operators. The thread runner uses a thread pool within the current process. Dispatch (line 1) submits the microbatch to the thread pool; DrainResults (line 1) reads results from an in-process queue. For each operator in the stage, the thread runner keeps a pool of pre-initialized operator instances. A worker thread checks out an instance for the duration of a microbatch. Python’s Future callbacks run synchronously on the caller when the future is already complete; since the pump is the sole consumer of the result queue, such a callback writing to a full queue would deadlock. Zephon avoids this by stashing synchronous results in a thread-local buffer that the pump drains inline after each dispatch. 5.4. User obligations and system guarantees In most cases, the execution machinery above is almost entirely hidden from users. Zephon comes with a toolbox of operators and WorkSource implementations that are deterministic by construction, and users typically just compose them (Section 4.1). Users may also insert stateless map or filter UDFs, which Zephon wraps into operators automatically. Only advanced users who implement custom stateful n-to-m operators must reason about determinism (Section 6) and the sample lifecycle (Section 7). State. When writing a custom operator, users need to separate state from the stateless processing function. State needs to be managed per lane. For example, the shuffle-buffer operator provides additional shuffling for each lane, supplementing the WorkSource’s shuffle before lane assignment (Section 4.4). Generally, accumulator decisions must depend only on the input sequence, not on how upstream operators group records into microbatches. In addition, operators must seed randomness from the contents of each microbatch rather than use a global RNG. For example, Zephon ’s shuffle-buffer implementation does this, ensuring identical output regardless of which worker processes the microbatch. Auto-validation. As these contracts cannot be enforced by Python’s type system, Zephon has a validation harness that automatically probes each user-defined operator, both statically on its AST and by running it on synthetic samples. Zephon flags operators whose output is non-deterministic or that wrongly handle sample metadata to help users catch these bugs. Trainer consumption. For a fixed global batch size of N records and L lanes, consider an execution with data-parallel degree G, per-replica training microbatch size m, and a=N/(Gm)a=N/(Gm) gradient-accumulation steps. With TP, P, or EP, lanes map to DP coordinates rather than global ranks; the training framework decides which ranks materialize, replicate, or shard the corresponding batches. Zephon ’s elastic determinism requires G|LG L and L/G|aL/G a; the latter is equivalent to L|GaL Ga, where Ga=N/mGa=N/m. The first condition assigns equally many lanes to every data-parallel replica; the second ensures that every lane contributes a whole number of microbatches to every global batch. Users choose L as a common multiple of the anticipated data-parallel degrees while ensuring that each intended m divides N/LN/L. The fixed L caps the data-parallel degree at L; larger values support more scale points but increase per-lane bookkeeping. Other configurations with G≤LG≤ L still execute deterministically under a fixed topology, but Zephon does not guarantee identical global batches across topology changes. Subject to these obligations, Zephon guarantees an identical global batch sequence across supported execution configurations. 6. Deterministic and elastic execution This section formalizes elastic determinism as a correctness guarantee over Zephon ’s stateful dataflow graph and shows that its ordered sequence of global batches is invariant across supported physical executions (R3). 6.1. Determinism by induction We show by induction that the output of a Zephon pipeline is deterministic. Intuitively, the accumulator is the sole component that decides ordering; all others mostly forward, so the overall stream is deterministic. Base case: single-operator determinism. Consider a single operator executed by a runner (Section 5.3). It follows the rules in Section 5.4. The pump (Algorithm 1) receives records in a fixed order, makes stateful decisions (grouping, buffering, emission) in the accumulator, and then tags each outgoing microbatch with a monotonic sequence number. Workers execute only their stateless processing function and may complete out of order. The reorder buffer then reassembles results by sequence number before forwarding them downstream. Inductive step: pipeline determinism. Let O1,…,OkO_1,…,O_k be operators with deterministic output. Then Ok+1O_k+1 receives deterministic input and, analogous to the base case, produces deterministic output. Stages and operators within a stage are connected by order-preserving FIFO queues. The full pipeline is therefore deterministic by induction. 6.2. Elastic determinism Let Π=(,,N) =(I,P,N) denote a fixed semantic pipeline specification. Here, =⟨I0,…,IL−1⟩I= I_0,…,I_L-1 contains the canonical ordered input stream of every lane, including flush sentinels (Section 7.3), P is the operator graph, and N is the global batch size. This definition is semantic: an implementation or configuration change is comparable only if it preserves I and P. The trainer consumption contract in Section 5.4 ensures that N/L=ma/(L/G)N/L=ma/(L/G) is an integer; let q=N/Lq=N/L. Let ℰ(Π)E( ) denote the supported physical executions of Π . Executions in ℰ(Π)E( ) may differ in runner, operator parallelism, data-parallel degree, and lane-to-replica assignment. Training microbatch size may also differ as it only regroups the same delivered records. An execution e with data-parallel degree GeG_e and training microbatch size mem_e belongs to ℰ(Π)E( ) only if Ge|LG_e L and me|qm_e q. For e∈ℰ(Π)e ( ), let Sℓ(Π,e)=⟨sℓ,0,sℓ,1,…⟩S_ ( ,e)= s_ ,0,s_ ,1,… denote the ordered record sequence delivered for lane ℓ , flattening the output if a Batch operator is present. The contribution of lane ℓ to optimizer step t is bℓ,t(Π,e)=⟨sℓ,tq,…,sℓ,(t+1)q−1⟩b_ ,t( ,e)= s_ ,tq,…,s_ ,(t+1)q-1 . The corresponding logical global batch is Bt(Π,e)=⟨b0,t(Π,e),…,bL−1,t(Π,e)⟩B_t( ,e)= b_0,t( ,e),…,b_L-1,t( ,e) . Zephon provides elastic determinism iff f.a. e,e′∈ℰ(Π)e,e ( ) and f.a. t:Bt(Π,e)=Bt(Π,e′)t:B_t( ,e)=B_t( ,e ). We omit Π and e below. Lanes as the unit of elasticity. Each lane’s flattened output sequence is independent of physical execution parameters such as runner or operator parallelism. Lanes are therefore the topology-independent units of the output. With data-parallel degree G, each replica owns L/GL/G lanes (with G|LG L). Changing G reassigns lanes to ranks but does not change their input streams, operator state, or ordered outputs. It remains to show that the per-rank outputs are consumed such that every optimizer step receives the same contribution bℓ,tb_ ,t from every lane. Round-robin execution and output multiplexing. Samples from all lanes owned by a rank flow through the same pipeline. At the source, the engine feeds the lanes in round-robin order. However, as lanes can produce output at different speeds, the raw pipeline output arrives in unpredictable lane order. The lane multiplexer (Section 5.1) restores the source ordering by buffering outputs in per-lane queues and emitting from the owned lanes in strict round-robin order. Faster lanes wait until their turn. Let m be the number of records in a training microbatch22 2 Note that Zephon ’s internal microbatching does not need to align with the training microbatches emitted by the batch operator.. For a supported execution, m|qm q. Before each optimizer step, every rank consumes N/(Gm)N/(Gm) training microbatches from Zephon and accumulates their gradients. Since a rank owns L/GL/G lanes, this corresponds to N/(Gm)L/G=NLm=qm N/(Gm)L/G= NLm= qm complete multiplexer rounds. Each owned lane therefore contributes q/mq/m training microbatches, containing exactly the q records in bℓ,tb_ ,t, to optimizer step t. Thus every optimizer step advances every lane by exactly q records. The calculation depends only on the number of lanes owned by each rank, not on which lanes it owns, so the lane-to-rank assignment does not affect BtB_t. Lane reassignment preserves global batches. The lane multiplexer and the trainer’s accumulation schedule ensure that each global batch contains exactly one contribution bℓ,tb_ ,t from every lane. Because each lane’s stream is deterministic and lane-local, bℓ,tb_ ,t is the same regardless of which replica owns lane ℓ . Changing G therefore changes only where each contribution is produced, not BtB_t; together with Section 6.1 and deterministic replay in Section 7, this establishes elastic determinism across all executions in ℰ(Π)E( ). Note that Zephon ’s guarantees end at BtB_t: different GPU topologies may change the training framework’s float accumulation and reduction order, so identical global batches do not imply bit-identical losses or model states. Zephon may execute configurations outside ℰ(Π)E( ) (e.g., G∤LG L or m∤qm q) deterministically under a fixed topology. Example. With L=4L=4 lanes and q=mq=m, two GPUs own lanes 0,1\0,1\ and 2,3\2,3\ and emit two microbatches per step, round-robin over their lanes. With four GPUs, each owns one lane and emits one. Either way, Bt=⟨b0,t,b1,t,b2,t,b3,t⟩B_t= b_0,t,b_1,t,b_2,t,b_3,t . 7. Checkpointing and resumption Zephon checkpoints and resumes the pipeline without replaying from the start (R4). Our key insight is that only in-flight chunks, i.e., chunks whose samples have entered the pipeline but have not all been delivered to the training loop, need to be saved alongside the WorkSource state and replayed on resumption. A checkpoint stores their sample pointers, not payloads, so checkpoint size is bounded by the number of in-flight chunks, which backpressure caps regardless of training progress (Section 5.1). The important question is when a chunk can be evicted, i.e., how the engine knows that all its samples have been delivered. 7.1. Cursors and contributors A record’s cursor uniquely identifies it. Orthogonally, contributor references track which source records a record contains. Cursors. Each record has a cursor (,,)( chunk,\, offset,\, lineage) that uniquely identifies it within its lane. The first two fields are the base offset of the cursor, i.e., the source chunk position the cursor is derived from. lineage is a path of integer indices that grows as fan-out operators derive new records, distinguishing records that share a base offset. For example, splitting a document into three sequences yields lineages (0,)(0,), (1,)(1,), (2,)(2,); further splitting child 1 produces (1,0)(1,0), (1,1)(1,1), etc. Fan-in operators like packing choose one primary cursor for the output, derived from one source record. This primary cursor serves as the packed record’s replay identity, while all source records, including the primary one, are tracked for eviction through contributor references (below). A checkpoint, in addition to the in-flight chunks and WorkSource state, stores one replay cursor per lane. For a batched pipeline, the replay cursor is batch-aligned: it is the cursor of the last record, in pipeline order, in the last batch delivered from that lane. For an unbatched pipeline, it is the cursor of the last delivered record. Contributors. A record has one cursor as its replay identity, but a fan-in operator like packing may assemble its payload from multiple source records. Each record therefore carries one contributor reference per source record, each storing that source record’s cursor and an is_last_child flag. A 11-to-11 record has one contributor: its own cursor. The base offset (,)( chunk,\, offset) of a cursor is the granularity at which we track completion. A base offset can be closed when a delivered contributor for it carries is_last_child. A single packed record can close several base offsets at once. 7.2. Replay On resumption, the engine restores from the checkpoint in two phases. In Phase 1, it replays the in-flight chunks through the pipeline. Because the pipeline is deterministic, replay produces identical output, including an identical accumulator state. A replay filter operator, automatically inserted during pipeline compilation, drops the already delivered prefix. It compares each record’s cursor against the stored replay cursor and suppresses everything up to and including the match. In Phase 2, the engine fetches new chunks from the restored WorkSource and normal execution continues. Elastic resumption. In distributed training, each rank’s engine owns a subset of lanes. A complete checkpoint must contain state for all L lanes so that any rank assignment can restore from it. Per-rank state grows sublinearly with the lanes a rank owns. Unlike data loaders where each node independently saves and restores its own state, Zephon must aggregate per-rank state into a global, self-contained checkpoint because on elastic resume, a rank may receive lanes it did not own previously and needs their state. When resuming on a different number of GPUs, each rank loads the merged checkpoint, takes ownership of its assigned lanes (Section 6.2), and replays its in-flight chunks independently, without further coordination. Figure 5. Per-chunk completion bitmaps for in-flight chunks. Each cell is one base offset; it closes when its last-child contributor is delivered. Non-monotone operators reorder samples across chunks, so chunks close out of order, and a chunk is evicted only once its bitmap is full. 7.3. Eviction We now discuss when a chunk can be evicted. Monotone eviction. We call a pipeline monotone if every operator both preserves cursor order (within each lane, records are emitted in nondecreasing cursor order) and carries no history-dependent state beyond its current buffer. In such a pipeline, eviction reduces to a simple watermark: if a record from chunk c reaches the training loop, all chunks before c are complete and can be evicted, because no operator’s future output depends on them. Non-monotone eviction. Operators like packing reorder records across chunk boundaries, breaking the watermark invariant. For example, a packed record may combine fragments from chunks 5 and 7, and chunk 5’s last sample may be delivered after a record from chunk 7. Evicting chunk 5 based on seeing a sample from chunk 7 would lose data. Hence, in the general non-monotone case, the engine tracks per-offset progress. It maintains a completion bitmap (Figure 5), marking offset j of chunk c as closed when a contributor with base offset (c,j)(c,j) and the last child flag set is delivered to the training loop. A chunk can be evicted only once all its offsets are closed. Completion bitmaps are not persisted in checkpoints; rather, they are rebuilt during replay. When a filter drops a record that is the last child for some base offset, the offset would never close. Hence, operators emit a tombstone, i.e., a non-payload sentinel carrying only a closing contributor reference. The problem of non-monotone pipelines is deeper than reordering. A history-dependent accumulator state, e.g., the packing accumulator’s bin layout, depends on the full history of inputs, not just the records currently buffered. After a chunk’s samples have all left the bins, their influence on bin placement persists. Crucially, two accumulator instances that start from different states may never converge, even on identical subsequent input. For example, consider a bin packer with 2 bins of capacity 10 and an input stream of elements of size 3. Starting from remaining-capacity states (10,10)(10,10) vs. (10,5)(10,5), both enter periodic emission cycles of the same period but permanently phase-shifted. This means that completion bitmaps alone are not sufficient. Replaying from a checkpoint would start the accumulator from an empty state instead of its true state during the checkpoint, producing different output and breaking deterministic replay. Flush sentinels and segments. Zephon solves this by injecting a flush sentinel in the stream every K chunks per lane in non-monotone pipelines. When a flush sentinel reaches an accumulator, it emits all buffered data and additionally resets internal state. Conceptually, the flush sentinels divide the source stream into segments of K chunks each, where segment SiS_i spans the chunks [iK,(i+1)K)[iK,\;(i+1)K). Flush sentinel positions are stored in the checkpoint and re-injected at the same positions on replay. Within a segment, the accumulator builds up state normally. At the segment boundary, the flush resets it to a clean state, making the next segment independent of all prior segments. For monotone pipelines, no flushes are needed. The resulting stream is deterministic for a fixed K. The segment-size tradeoff. A smaller K allows chunks to be evicted more frequently. However, each flush forces operators such as the packer to emit partially filled bins, which may increase padding waste. Changing K has little effect on checkpoint state because backpressure limits how far the pipeline reads ahead (Section 8.2). Flush discipline. Non-monotone operators must flush immediately when the flush sentinel arrives. Stalling would let post-boundary records mix with pre-boundary buffered state, creating an accumulator state that replay from empty cannot reconstruct. Furthermore, a stalled flush sentinel delays the reset for downstream operators, causing their flushes to occur in divergent state. The exception is the Batch operator, which stalls the flush sentinel, allowing post-boundary records to complete a partial batch. This is safe because Batch preserves cursor order and has no history-dependent state beyond its buffer. The replay filter is placed immediately before Batch during compilation; on resume, the consumed prefix is dropped before batching, so Batch rebuilds from an empty buffer at the correct batch-aligned cut. Lemma 7.1 (Segment eviction safety). Segment SiS_i can be safely evicted when these two conditions hold: (1) All offsets in all chunks of SiS_i are closed (per-chunk completion bitmaps full). (2) The replay cursor does not reference a chunk in SiS_i. Proof sketch. The flush sentinel injected after SiS_i drains and resets all accumulators, so no future output depends on SiS_i. Condition 1 implies any Batch-stalled flush sentinel has already released: offsets close only when last-child contributors are delivered in complete batches, and Batch preserves cursor order, so all SiS_i records have been emitted before condition 1 holds. The batch-aligned replay cursor (Section 7.1) identifies exactly this consumed prefix. There is no incorrect early eviction due to stalling. Since segments are evicted in order, condition 2 ensures the replay cursor is past SiS_i, so replay begins from the next segment’s clean state. ∎ For non-monotone pipelines, eviction is atomic per segment as chunks within a segment share accumulator history. Example. Consider a non-monotone pipeline with K=2K=2. Lane 0 has in-flight chunks 4–7 spanning segments S2=4,5S_2=\4,5\ and S3=6,7S_3=\6,7\, with a flush sentinel between them. The replay cursor references chunk 6. At checkpoint time, all offsets in chunks 4 and 5 are closed (condition 1) and the replay cursor does not reference a chunk in S2S_2 (condition 2), so the engine evicts S2S_2. The checkpoint saves only the pointers of chunks 6–7, the WorkSource position past chunk 7, the replay cursor, and the stored boundary position 6 (marking the start of S3S_3). 8. Evaluation We evaluate Zephon to answer the following questions: (1) How competitive is Zephon ’s throughput, how do its knobs affect scaling, and what do its guarantees cost? (R1) (2) How do checkpoint size and recovery cost scale with training progress and lane count? (R4) (3) How does elastic determinism affect training loss and downstream accuracy across GPU counts? (R3) Setup. All performance benchmarks use an AWS r8id.48xlarge instance (Intel Xeon 6975P @ 2.70 GHz, 96 physical cores, 1.5 TiB RAM, 3×3.53× 3.5 TB NVMe in RAID-0) running Ubuntu 24.04. The instance is a single-tenant (dedicated) CPU host running the data loader and all its workers. The checkpointing experiments use eight parallel processes on this host. Training experiments use 8–64 H100 GPUs, with the data loader co-located with each training rank. To minimize variance, we disable SMT, C-state transitions, ASLR, and the NMI watchdog. 8.1. Throughput We compare Zephon to other data loaders and evaluate how its tuning knobs affect performance. Rather than end-to-end model training throughput, we use raw data loader throughput as the evaluation metric, as this is model-independent. For a given training scenario, to avoid data stalls, the data loader must sustain a throughput higher than the model’s peak data consumption rate (Mohan et al., 2021b; Zhao et al., 2024a). For each scenario, we warm up the data loader until it reaches steady state, and then measure throughput over a fixed number of steps. We run every benchmark with three random seeds and report the mean. We store all data in the LitData data format (Chaton and AI, 2023), in shards of roughly 64 MiB size. We store all data on the local NVMe raid to benchmark the performance of the data loader instead of cloud storage throughput. Cloud storage throughput has high variance, and the intention of these benchmarks is to show the limits of the data loaders and their processing pipeline. Fetching data from the cloud would mostly show bandwidth limitations, not the scalability of the implementation. We consider two benchmarks, fetch-and-batch (Section 8.1.1), which loads data like pretokenized strings from disk, and a VLM pipeline (Section 8.1.2). 8.1.1. Fetch-and-batch [width=]svg-inkscape/fig_5_overview_text_svg-tex.pdf_tex [width=]svg-inkscape/fig_5_overview_vlm_svg-tex.pdf_tex (a) Comparison across loaders and parallelism [width=]svg-inkscape/fig_5_multithread_text_svg-tex.pdf_tex [width=]svg-inkscape/fig_5_multithread_vlm_svg-tex.pdf_tex (b) Effect of GIL removal [width=]svg-inkscape/fig_5_shm_text_svg-tex.pdf_tex [width=]svg-inkscape/fig_5_shm_vlm_svg-tex.pdf_tex (c) Impact of SHM coalescing Figure 6. Fetch-and-batch throughput for text (top) and VLM (bottom) data with Zephon ’s process (Z-Proc) and thread (Z-Thread) runners. On text data, Zephon is sufficiently fast for stall-free training; on VLM data, it outperforms all baselines. In this benchmark, we load raw sample payloads and create batches. While this scenario involves minimal processing, it stress-tests the overhead of Zephon ’s operator machinery in a setting where workers have a structural advantage, yet highlights the flexibility of the operator-based architecture. Setup details. Besides Zephon , we test the LitData data loader, a fork of Mosaic Streaming (short Streaming) with support for LitData shards, and Google Grain (short Grain). The text samples are sequences of 4 096 tokens. The VLM samples are multimodal documents, i.e., PyTree-serialized dicts containing metadata, text (as strings), and a variable number of images (as packed bytes), from the MAmmoTH-VL dataset (Guo et al., 2025). We shuffle the sample order in the WorkSource. For Zephon , we vary the degree of parallelism (DOP) of the fetch operator, and for the other worker-based loaders, we vary the number of workers. We report results for batch size 32. Overview. The results are displayed in Figure 6. For text data, the worker-based baselines perform very well. Grain peaks at 201 k samples/s at 16 workers, benefiting from two-stage prefetch both within the underlying dataset and its workers. LitData and Streaming reach 120 k and 100 k, respectively. All three show diminishing returns or regression beyond their optimal worker count. Zephon ’s process runner saturates at 59 k samples/s (DOP≥8). Its thread runner, constrained by the GIL, plateaus at just 34 k samples/s despite avoiding IPC entirely. For small text payloads, workers that combine fetching and batching in a single process have a structural advantage over Zephon ’s finer-grained operator model. VLM data, where samples are variable-size dicts containing text and multiple images, favors Zephon . LitData and Streaming fail to scale beyond ∼ 2 k samples/s regardless of the number of workers; Grain scales further to 5.8 k at 64 workers. Zephon ’s process runner reaches 16 k samples/s, a 2.7×2.7× improvement over the best baseline, and its (GIL-limited) thread runner reaches 10 k. Cost of the guarantees. Disabling deterministic result reordering raises process-runner peaks from 59 k and 16 k to 62.1 k and 20.5 k samples/s for text/VLM. Further removing engine bookkeeping, lane execution, contributor tracking, and replay-control mechanisms brings them to 70.6 k/20.6 k samples/s. The guarantees thus cost 16 %/22 % relative to a stripped pipeline that can neither order its output nor checkpoint. Key Message: Zephon ’s operator overhead is amortized by larger payloads. On VLM data, it delivers 2.7×2.7× higher throughput than the best baseline, while remaining sufficiently high for text data. Multithreading. Zephon ’s thread runner on Python 3.13 is severely GIL-limited, plateauing below the process runner despite avoiding IPC entirely. The GIL serializes the fetch threads, preventing the runner from exploiting additional parallelism beyond the first few threads. We evaluate Zephon on Python 3.14t, which removes the GIL (Figure 6). For text, the thread runner improves from 34 k to 140 k samples/s (4.1×4.1×), now scaling up to DOP==16 rather than plateauing at DOP==4. This brings it within reach of the best baseline (Grain/201 k) while retaining Zephon ’s operator-graph flexibility. The process runner also benefits modestly (59 k→71 k, 1.2×1.2×), as the pumps run concurrently with the Python main thread. For VLM data, the effect is even more pronounced. The thread runner jumps from 10 k to 59 k samples/s (5.9×5.9×), surpassing even the Python 3.14t process runner (27 k) by 2.2×2.2×. Where the process runner was the stronger VLM configuration on 3.13 (16 k vs. 10 k), GIL removal inverts this. We do not report the worker-based data loaders, since their throughput only regresses due to the per-thread overhead in GIL-free Python. So far, besides Zephon , only the Spdl project (Hira et al., 2025) is built for GIL-free data loading. However, Spdl supports only threads, whereas Zephon allows choosing between threads and processes as execution substrate while maintaining determinism. Key Message: Removing the GIL unlocks thread-based scaling. On Python 3.14t, the thread runner closes the gap with workers on text and surpasses the process runner by 2.2×2.2× on VLM data. SHM coalescing. We evaluate shared-memory coalescing in Zephon ’s process runner (Section 5.3.1) by disabling it (OS Pipe in Figure 6). For text data, this improves throughput by 45 %, from 59 k to 86 k samples/s. These samples are small, fixed-size NumPy arrays (∼ 16 KiB each), so the overhead of shm allocation, copying, and metadata transfer exceeds the cost of direct serialization. For VLM data, the opposite holds: disabling coalescing collapses throughput from 16 k to 2.4 k samples/s (−-85 %), and the process runner no longer scales. VLM samples contain large, variable-size image buffers that quickly saturate the OS pipe buffer when serialized directly. Thus, SHM coalescing amortizes IPC cost for large payloads, but its fixed overhead dominates when samples are small. Key Message: SHM coalescing enables Zephon to achieve a 6.7×6.7× throughput gain on VLM data over standard IPC. 8.1.2. VLM pipeline We benchmark a non-monotone, standard VLM preprocessing pipeline (Section 2). After reading data from storage (fetch), decode parses text into conversations and decodes images to Pillow. Preprocess validates image-text alignment, expands placeholders into vision-token sequences, applies the chat template, and runs image transforms like resize. Tokenize converts the decorated text into tokens. Afterwards, shuffle+pack performs first-fit bin packing. For fairness, we share the same packing implementation across all loaders. Last, extract prepares each packed sample for batching, and construct assembles a flat training batch compatible with FlashAttention-2 (Dao, 2024) and validates batch integrity. Flat batches concatenate all documents into a single tensor without a batch dimension, so raw token throughput does not reflect capacity wasted by suboptimal packing. We therefore report the normalized usable tokens per second (utps), i.e., the total number of non-padding tokens across all GPUs divided by wall-clock time and GPU count. We normalize per-GPU to emphasize the scaling behavior of each loader. [width=0.95]svg-inkscape/fig_6_vlm_scaling_svg-tex.pdf_tex Figure 7. Per-GPU normalized VLM pipeline throughput depending on degree of parallelism. Loaders. We compare Zephon to a Grain pipeline and to InternalDs, a solution built around Streaming (Mosaic ML Team, 2022b) by (non-systems) ML researchers that runs all preprocessing in GIL-contending background threads in the main Python process. We use Zephon with a fused implementation which merges decode, preprocess, and tokenize into a single operator, and use the process runner since the torchvision transforms are not thread-safe. For Grain and InternalDs, the degree of parallelism (DOP) equals the number of worker processes. In Zephon , aggregate DOP is the sum of per-operator parallelism across pipeline operators. We sweep a few distributions of a fixed parallelism budget and report the best at each DOP. Zephon ’s deterministic mode is enabled throughout. Results. Figure 7 shows the normalized usable token throughput as the DOP increases. InternalDs’s throughput plateaus as DOP increases because its GIL-contending threads cannot exploit additional parallelism. Its throughput also stays below the 16.9 k utps per GPU required to avoid data stalls. Grain and Zephon both scale near-linearly. At DOP 32 their per-GPU throughput is effectively tied (29.8 k vs. 29.9 k). At DOP 64, Grain leads by ca. 8 % (26.6 k vs. 28.7 k). That Zephon matches Grain and is able to scale despite incurring IPC at each operator boundary is a direct consequence of our optimized SHM-coalesced transport (Section 5.3.1). Bulk image and token data is placed in shared memory with only lightweight metadata crossing the pipe, and the flat payload layout minimizes serialization depth. At DOP 64, both systems deliver over 23×23× the throughput of InternalDs. Crucially, Zephon provides elastically deterministic output ordering across parallelism changes (Section 8.3), which no other baseline provides. Key Message: Zephon matches Grain’s near-linear scaling on a VLM pipeline despite IPC and maintaining elastic determinism. 8.1.3. Time-to-first-batch Beyond steady-state throughput, startup time matters when iterating rapidly on short training runs. We measure time-to-first-batch (TTFB) from loader construction to the first batch being available. We use the scenarios from Sections 8.1.1 and 8.1.2 at DOP 64. Process overhead. With the GIL, Zephon ’s thread runner has the lowest fetch-and-batch TTFB: 0.24/0.46 s for text/VLM payloads, versus 2.87/3.03 s with processes. Spawning 64 worker processes dominates startup, whereas thread creation is near-instant. Unlike for steady-state throughput, removing the GIL has little effect: TTFB falls to 0.22/0.43 s with threads and 2.78/2.97 s with processes. Streaming (0.97/1.68 s) and Grain (2.24/2.84 s) fall between the two runners. LitData is a striking outlier on text (17.92 s). Its TTFB scales superlinearly with worker count, as each worker independently initializes shard metadata. For VLM payloads, fewer samples per shard reduce this overhead (2.27 s). VLM pipeline. The full VLM pipeline amplifies TTFB differences. InternalDs requires 101.17 s, as its GIL-contending background threads serialize all preprocessing during startup. With the GIL, Zephon ’s process runner achieves 7.61 s versus Grain’s 9.21 s, reducing TTFB by 17 % relative to Grain and by a factor of 1313 relative to InternalDs. Key Message: Zephon ’s thread runner delivers subsecond TTFB by avoiding process-spawn overhead. 8.2. Checkpointing analysis We evaluate the non-monotone VLM pipeline on eight parallel processes. S3 stores all data, aggregated states, and checkpoints; each run starts with an empty NVMe dataset cache. We repeat each configuration three times per seed and report medians. In-flight state. We examine how flush sentinel cadence K and chunk size C affect the retained state. We sweep K∈1,2,8,16,32K∈\1,2,8,16,32\ and C∈1 024,2 048,4 096C∈\1\,024,2\,048,4\,096\ with three seeds, checkpointing after 100 simulated training steps. Each step consumes four batches per rank without model computation. With increasing C, each rank holds 2626/1313/77 in-flight chunks, corresponding to 26.626.6/26.626.6/28.728.7 k in-flight records. The nearly constant record count is caused by backpressure limiting read-ahead. Changing the flush cadence K has a negligible effect on the in-flight state. To confirm how the state scales with training progress, we fix K=8K=8, C=1 024C=1\,024, and checkpoint after 1 000/10 000 steps. The checkpoints retain an almost-constant amount of 29.8/30.3 k in-flight records per rank (26.6 k at step 100). Frequent flushes may have an impact on stream quality, such as packing efficiency. However, we find that even in the worst case (K=1K=1, C=1 024C=1\,024) flushing increases token waste by less than 0.370.37 p relative to no flushing. Save cost. In the K/CK/C sweep (step 100), saving to S3, including aggregation, takes 3.03 s (IQR 2.37–3.54 s). Checkpoint size grows only modestly with training step (2.61/2.92/2.97 MiB for 100/1 k/10 k steps). Saving takes 3.56 s at both later checkpoints. Restore cost and replay overhead. We measure TTFB from checkpoint reading to the first batch, including pipeline startup and replay. At step 100, the median TTFB across the K/CK/C sweep is 48.5 s, compared to 44.8 s for a cold start (+8 %). Restoring checkpoints taken at steps 1 k/10 k takes 70.2/73.8 s. Lane scaling. We test recovery with more than one lane per rank, fixing K=8K=8, C=1 024C=1\,024, and step 100. We increase L to 64 and 512 (8/64 lanes per rank). In-flight records count per rank rises from 26.6 k to 35.1/131.1 k; checkpoint sizes increase to 3.46/13.10 MiB. Saving takes 3.47/8.76 s, and TTFB after restore is 62.6/403 s. Key Message: Checkpoint state grows only marginally with training progress; more lanes per rank increase in-flight state and recovery cost. [width=]svg-inkscape/fig_9a_core_v1_svg-tex.pdf_tex (a) Core v1 [width=]svg-inkscape/fig_9b_fineweb_svg-tex.pdf_tex (b) FineWeb Spread across GPU counts (p) Figure 8. Score spread (max - min) across GPU counts (8, 16, 32, 64) for 20 B-token training. The dashed line is the reference level above which differences are meaningful.TODO! 8.3. Elastic determinism In Figure 1 we demonstrate how changing the GPU count leads to different loss curves. We now quantify the effect of this on downstream model accuracy. Then, we evaluate Zephon ’s elastic determinism on a non-monotone VLM pipeline (R3). We reuse the architecture, hyperparameters, and seed checkpoint from Figure 1 and train a 1 B model for 20 B tokens (Chinchilla-optimal (Hoffmann et al., 2022)) on 8, 16, 32, and 64 GPUs, both with and without elastic determinism. We then evaluate the final model of each run on the DCLM Core v1 (Li et al., 2024) and FineWeb (Penedo et al., 2024a) evaluation suites. We repeat this for two seeds. Figure 8 shows the max. spread of scores across GPU counts for both loaders, averaged across the two seeds. DCLM describes differences within 0.20.2 Core v1 points as comparable. For FineWeb, ∼0.15 0.15 p is the smallest uplift reconstructed from the reported results for the selected filters (Li et al., 2024; Penedo et al., 2024a). Without elastic determinism, the spread is 0.9220.922 p on Core v1 and 0.5550.555 p on FineWeb, about four times the reference level. With Zephon , however, the spread is just 0.0610.061 p and 0.0070.007 p, landing comfortably below the threshold. Using Zephon , downstream performance is thus determined by the run instead of by the GPU topology it happened to be trained on. VLM. To test Zephon in a second setting, we train a 2.1 B-parameter VLM consisting of a frozen SigLIP2-SO400M vision encoder (Tschannen et al., 2025), a two-layer MLP projector, and a Qwen3-1.7B language backbone (Yang et al., 2025), for a total of ca. 1.7 B trainable parameters. We train for 300 steps with a global batch size of 256 using AdamW and a warmup-stable-decay learning rate schedule (Zhai et al., 2022; Hu et al., 2024; Hägele et al., 2024; Wen et al., 2025) (20-step warmup, peak LR 5⋅10−45· 10^-4). We do not train the model to convergence because this experiment is designed to demonstrate the effect of elastic determinism rather than final model quality. The shorter run makes the loss curves easier to inspect in detail. To isolate the effect of the input sequence, we minimize numerical variation in the training framework. We train in fp32, use SDPA attention, enable all deterministic CUDA options, and initialize every run from the same seed checkpoint. With lower precision, loss curves can differ more across GPU configurations even when the global batches are identical. We compare Zephon and the InternalDs solution based on Streaming and train the model using 8, 16, 32, and 64 H100 GPUs. The results are shown in Figure 9. Without elastic determinism, each GPU count produces a distinct loss trajectory. Zephon produces identical global batches for every GPU count; the corresponding loss curves are closely aligned, with Zephon ’s mean absolute pairwise difference being 5.84×5.84× lower. Key Message: Zephon produces identical global batches across 8–64 GPUs on a stateful, non-monotone VLM pipeline. [width=]svg-inkscape/fig_7a_non_det_svg-tex.pdf_tex (a) InternalDs [width=]svg-inkscape/fig_7b_bernina_svg-tex.pdf_tex (b) Zephon Figure 9. VLM training loss across 8–64 GPUs. 9. Related work Foundation model data. The ML community is exploring how to best filter (Gadre et al., 2024; Li et al., 2024; Penedo et al., 2024a), mix (Xie et al., 2023; Chen et al., 2023; Ye et al., 2025; Liu et al., 2025; Chen et al., 2026), and curate and process (Penedo et al., 2024b; Soldaini et al., 2024; Huang et al., 2025; Maini et al., 2024; Maini et al., 2025; Niklaus et al., 2026; NVIDIA, 2025) training data for foundation models. Data loaders. Data stalls have motivated data loader research since the late 2010s (Yang and Cong, 2019; Zhu et al., 2018; Mohan et al., 2021b; Isenko et al., 2022; Kuchnik et al., 2022; Bachkaniwala et al., 2024). Operator-centric loaders organize pipelines as operator graphs (Murray et al., 2021; Zhao et al., 2024a). Format-centric loaders optimize specific data formats (Mosaic ML Team, 2022b; Aizman et al., 2019; Chaton and AI, 2023; Gruener et al., 2018). There is work on disaggregated data loading (Audibert et al., 2023; Graur et al., 2022; Graur et al., 2024; Um et al., 2023; Zhang et al., 2025) and sharing loaded data across co-located jobs (Kumar and Sivathanu, 2020; Xie et al., 2022; Robroek et al., 2026). MinatoLoader (Nouaji et al., 2026) improves throughput by reordering samples, explicitly trading determinism for throughput. Foundation model-specific systems include Mixtera (Böther et al., 2026) and MegaScale-Data (Zhao et al., 2026). Mosaic Streaming introduces elastic determinism, i.e., the preservation of the global sample order across changes in the number of GPUs, nodes, or CPU workers (MosaicML, [n. d.]). It implements this for indexable datasets by partitioning the sample space into canonical nodes, which are remapped to the physical topology. Grain (Ritter et al., 2023) similarly provides elastic determinism for indexable pipelines and deterministic resumption for stateful pipelines under fixed execution configurations. Autotuning (Graur et al., 2022; Graur et al., 2024; Gavriilidis et al., 2025; Didrich et al., 2025) can adjust data loader operator parallelism. Elastic training and recovery. EasyScale (Li et al., 2023) preserves bitwise model state across GPU counts using logical workers and saved index/RNG state; Tenplex (Wagenländer et al., 2024) preserves sample order by repartitioning immutable, indexed data. Both assume indexable datasets. Complementarily, Gemini (Wang et al., 2023) accelerates checkpoint recovery under fixed resources, but leaves ordering out of scope. Stream processing and dataflow systems. Checkpointing stateful pipelines over unbounded streams is a classic problem in stream processing (Akidau et al., 2013; Carbone et al., 2015b). For example, Flink adapts Chandy–Lamport distributed snapshots (Chandy and Lamport, 1985; Carbone et al., 2015a). Naiad (Murray et al., 2013) introduces timely dataflow and pointstamp-based progress tracking for cyclic, iterative dataflow graphs. Data loading pipelines are structurally simple: each lane is a single-input operator chain with no joins or triggers. While Zephon ’s operator toolbox is tailored for foundation model training, the system could be used for any stateful n-to-m dataflow graph requiring reproducible ordered output. Deterministic parallelism. Prior work explores deterministic parallelism at the language (DPJ (Jr. et al., 2009)), compiler (CoreDet (Bergan et al., 2010), Kendo (Olszewski et al., 2009)), or OS level (DThreads (Liu et al., 2011), Determinator (Aviram et al., 2010)). 10. Conclusion We present Zephon , to our knowledge, the first data loader to provide elastic determinism and efficient resumption for online, stateful n-to-m foundation model data pipelines by serializing ordering decisions in accumulators, while parallelizing stateless work on interchangeable backends. With Zephon , we can run more reliable ablations while reducing our compute-spend thanks to Zephon ’s higher and more stable throughput for VLM training and low startup time on trillion-token-scale text corpora. References Aizman et al. (2019) Alex Aizman, Gavin Maltby, and Thomas Breuel. 2019. High Performance I/O For Large Scale Deep Learning. In Proceedings of the IEEE International Conference on Big Data (BigData). doi:10.1109/BigData47090.2019.9005703 Akidau et al. (2013) Tyler Akidau, Alex Balikov, Kaya Bekiroğlu, Slava Chernyak, Josh Haberman, Reuven Lax, Sam McVeety, Daniel Mills, Paul Nordstrom, and Sam Whittle. 2013. MillWheel: Fault-Tolerant Stream Processing at Internet Scale. Proceedings of the VLDB Endowment 6, 11 (2013), 1033–1044. doi:10.14778/2536222.2536229 Akidau et al. (2015) Tyler Akidau, Robert Bradshaw, Craig Chambers, Slava Chernyak, Rafael Fernández-Moctezuma, Reuven Lax, Sam McVeety, Daniel Mills, Frances Perry, Eric Schmidt, and Sam Whittle. 2015. The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing. Proceedings of the VLDB Endowment 8, 12 (2015). doi:10.14778/2824032.2824076 Amazon Web Services (2026) Amazon Web Services. 2026. Using elastic training in Amazon SageMaker HyperPod. https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-eks-elastic-training.html. An et al. (2025) Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, Chunsheng Wu, Huajie Tan, Chunyuan Li, Jing Yang, Jie Yu, Xiyao Wang, Bin Qin, Yumeng Wang, Zizhen Yan, Ziyong Feng, Ziwei Liu, Bo Li, and Jiankang Deng. 2025. LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training. arXiv preprint (2025). doi:10.48550/arXiv.2509.23661 Anyscale (2024) Anyscale. 2024. How Runway Powers AI Media Creation with Anyscale. https://w.anyscale.com/resources/case-study/runway. Audibert et al. (2023) Andrew Audibert, Yang Chen, Dan Graur, Ana Klimovic, Jiří Šimša, and Chandramohan A. Thekkath. 2023. tf.data service: A Case for Disaggregating ML Input Data Processing. In Proceedings of the Symposium on Cloud Computing (SoCC). doi:10.1145/3620678.3624666 Aviram et al. (2010) Amittai Aviram, Shu-Chun Weng, Sen Hu, and Bryan Ford. 2010. Efficient System-Enforced Deterministic Parallelism. In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). doi:10.48550/arXiv.1005.3450 Bachkaniwala et al. (2024) Rajveer Bachkaniwala, Harshith Lanka, Kexin Rong, and Ada Gavrilovska. 2024. Lotus: Characterization of Machine Learning Preprocessing Pipelines via Framework and Hardware Profiling. In Proceedings of the IEEE International Symposium on Workload Characterization (IISWC). doi:10.1109/iiswc63097.2024.00013 Bai et al. (2025a) Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. 2025a. Qwen3-VL Technical Report. arXiv preprint (2025). doi:10.48550/arXiv.2511.21631 Bai et al. (2025b) Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025b. Qwen2.5-VL Technical Report. arXiv preprint (2025). doi:10.48550/arXiv.2502.13923 Bergan et al. (2010) Tom Bergan, Owen Anderson, Joseph Devietti, Luis Ceze, and Dan Grossman. 2010. CoreDet: A Compiler and Runtime System for Deterministic Multithreaded Execution. In Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). doi:10.1145/1736020.1736029 Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. arXiv preprint (2023). doi:10.48550/arXiv.2304.01373 Böther et al. (2026) Maximilian Böther, Xiaozhe Yao, Tolga Kerimoglu, Dan Graur, Viktor Gsteiger, and Ana Klimovic. 2026. Mixtera: A Data Plane for Foundation Model Training. In Proceedings of the International Conference on Management of Data (SIGMOD). doi:10.1145/3786668 Carbone et al. (2015a) Paris Carbone, Gyula Fóra, Stephan Ewen, Seif Haridi, and Kostas Tzoumas. 2015a. Lightweight Asynchronous Snapshots for Distributed Dataflows. arXiv preprint arXiv:1506.08603 (2015). doi:10.48550/arXiv.1506.08603 Carbone et al. (2015b) Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015b. Apache Flink™: Stream and Batch Processing in a Single Engine. Bulletin of the Technical Committee on Data Engineering 38, 4 (2015). Chandy and Lamport (1985) K. Mani Chandy and Leslie Lamport. 1985. Distributed Snapshots: Determining Global States of Distributed Systems. ACM Transactions on Computer Systems 3, 1 (1985), 63–75. doi:10.1145/214451.214456 Chaton and AI (2023) Thomas Chaton and Lightning AI. 2023. LitData: Transform datasets at scale. Optimize datasets for fast AI model training. Chen et al. (2024a) Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Ge, Dawei Gao, Yuexiang Xie, Zhaoyang Liu, Jinyang Gao, Yaliang Li, Bolin Ding, and Jingren Zhou. 2024a. Data-Juicer: A One-Stop Data Processing System for Large Language Models. In Proceedings of the International Conference on Management of Data (SIGMOD). doi:10.1145/3626246.3653385 Chen et al. (2026) Mayee F. Chen, Tyler Murray, David Heineman, Matt Jordan, Hannaneh Hajishirzi, Christopher Re, Luca Soldaini, and Kyle Lo. 2026. OLMix: A Framework for Data Mixing Throughout LM Development. arXiv preprint (2026). doi:10.48550/arXiv.2602.12237 Chen et al. (2023) Mayee F. Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Re. 2023. Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models. In Advances in Neural Information Processing Systems (NeurIPS). doi:10.48550/arXiv.2307.14430 Chen et al. (2024b) Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yiming Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. 2024b. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. arXiv preprint (2024). doi:10.48550/arXiv.2412.05271 Crosley (2026) Blake Crosley. 2026. Video Generation AI Infrastructure: Building for Sora-Scale Models. https://introl.com/blog/video-generation-ai-infrastructure-sora-scale-models-guide-2025. Introl Blog, March 30, 2026. Dao (2024) Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. In Proceedings of the International Conference on Learning Representations (ICLR). doi:10.48550/arXiv.2307.08691 Didrich et al. (2025) Benedikt Didrich, Haralampos Gavriilidis, Vasilis Gkolemis, Matthias Boehm, and Volker Markl. 2025. Learning to Accelerate: Tuning Data Transfer Parameters. In Proceedings of the International Workshop on Applied AI for Database Systems and Applications (AIDB). https://w.vldb.org/2025/Workshops/VLDB-Workshops-2025/AIDB/AIDB25_11.pdf European Parliament and Council of the European Union (2024) European Parliament and Council of the European Union. 2024. Regulation (EU) 2024/1689, Art. 11: Technical Documentation. https://eur-lex.europa.eu/eli/reg/2024/1689/oj. Official Journal of the European Union, L Series, 12 July 2024. Gadre et al. (2024) Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. 2024. DataComp: In Search of the Next Generation of Multimodal Datasets. In Advances in Neural Information Processing Systems (NeurIPS). doi:10.5555/3666122.3667301 Gadre et al. (2025) Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Luca Soldaini, Alexandros G. Dimakis, Gabriel Ilharco, Pang Wei Koh, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt. 2025. Language models scale reliably with over-training and on downstream tasks. In Proceedings of the International Conference on Learning Representations (ICLR). doi:10.48550/arXiv.2403.08540 Gavriilidis et al. (2025) Haralampos Gavriilidis, Kaustubh Beedkar, Matthias Boehm, and Volker Markl. 2025. Fast and Scalable Data Transfer Across Data Systems. In Proceedings of the International Conference on Management of Data (SIGMOD). doi:10.1145/3725294 Gemini Team (2025) Gemini Team. 2025. Gemini: A Family of Highly Capable Multimodal Models. arXiv preprint (2025). doi:10.48550/arXiv.2312.11805 Graur et al. (2022) Dan Graur, Damien Aymon, Dan Kluser, Tanguy Albrici, Chandramohan A Thekkath, and Ana Klimovic. 2022. Cachew: Machine Learning Input Data Processing as a Service. In Proceedings of the Annual Technical Conference (ATC). Graur et al. (2024) Dan Graur, Oto Mraz, Muyu Li, Sepehr Pourghannad, Chandramohan A Thekkath, and Ana Klimovic. 2024. Pecan: Cost-Efficient ML Data Preprocessing with Automatic Transformation Ordering and Hybrid Placement. In Proceedings of the Annual Technical Conference (ATC). Groeneveld et al. (2025) Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. 2025. OLMo: Accelerating the Science of Language Models. arXiv preprint (2025). doi:10.48550/arXiv.2402.00838 Gruener et al. (2018) Robbie Gruener, Owen Cheng, and Yevgeni Litvin. 2018. Petastorm: Uber ATG’s Data Access Library for Deep Learning. https://w.uber.com/blog/petastorm/. Guo et al. (2025) Jiawei Guo, Tianyu Zheng, Yizhi Li, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Graham Neubig, Wenhu Chen, and Xiang Yue. 2025. MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). doi:10.18653/v1/2025.acl-long.680 Hägele et al. (2024) Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. 2024. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. In Advances in Neural Information Processing Systems (NeurIPS). doi:10.5555/3737916.3740343 Hira et al. (2025) Moto Hira, Christian Puhrsch, Valentin Andrei, Roman Malinovskyy, Gael Le Lan, Abhinandan Krishnan, Joseph Cummings, Miguel Martin, Gokul Gunasekaran, Yuta Inoue, Alex Turner, and Raghuraman Krishnamoorthi. 2025. Scalable and Performant Data Loading. arXiv (2025). doi:10.48550/arXiv.2504.20067 Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training Compute-Optimal Large Language Models. arXiv preprint (2022). doi:10.48550/arXiv.2203.15556 Hu et al. (2024) Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. In Proceedings of the Conference on Language Modeling (COLM). doi:10.48550/arXiv.2404.06395 Huang et al. (2025) Yue Huang, Siyuan Wu, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Chaowei Xiao, Jianfeng Gao, Lichao Sun, and Xiangliang Zhang. 2025. DataGen: Unified Synthetic Dataset Generation via Large Language Models. In Proceedings of the International Conference on Learning Representations (ICLR). doi:10.48550/arXiv.2406.18966 Isenko et al. (2022) Alexander Isenko, Ruben Mayer, Jeffrey Jedele, and Hans-Arno Jacobsen. 2022. Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines. In Proceedings of the International Conference on Management of Data (SIGMOD). doi:10.1145/3514221.3517848 Jiang et al. (2024) Yiding Jiang, Allan Zhou, Zhili Feng, Sadhika Malladi, and J. Zico Kolter. 2024. Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws. In Proceedings of the International Conference on Learning Representations (ICLR). doi:10.48550/arXiv.2410.11820 Jr. et al. (2009) Robert L. Bocchino Jr., Vikram S. Adve, Danny Dig, Sarita V. Adve, Stephen Heumann, Rakesh Komuravelli, Jeffrey Overbey, Patrick Simmons, Hyojin Sung, and Robert Tartler. 2009. A Type and Effect System for Deterministic Parallel Java. In Proceedings of the Conference on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA). doi:10.1145/1640089.1640097 Kang et al. (2025) Xueze Kang, Guangyu Xiang, Yuxin Wang, Hao Zhang, Yuchu Fang, Yuhang Zhou, Zhenheng Tang, Youhui Lv, Eliran Maman, Mark Wasserman, Alon Zameret, Zhipeng Bian, Shushu Chen, Zhiyou Yu, Jin Wang, Xiaoyu Wu, Yang Zheng, Chen Tian, and Xiaowen Chu. 2025. ElasWave: An Elastic-Native System for Scalable Hybrid-Parallel Training. arXiv preprint (2025). doi:10.48550/arXiv.2510.00606 Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. arXiv preprint (2020). doi:10.48550/arXiv.2001.08361 Kokolis et al. (2025) Apostolos Kokolis, Michael Kuchnik, John Hoffman, Adithya Kumar, Parth Malani, Faye Ma, Zachary DeVito, Shubho Sengupta, Kalyan Saladi, and Carole-Jean Wu. 2025. Revisiting Reliability in Large-Scale Machine Learning Research Clusters. In Proceedings of the IEEE International Symposium on High-Performance Computer Architecture (HPCA). doi:10.1109/hpca61900.2025.00096 Krell et al. (2021) Mario Michael Krell, Matej Kosec, Sergio P. Perez, and Andrew Fitzgibbon. 2021. Efficient Sequence Packing without Cross-contamination: Accelerating Large Language Models without Impacting Performance. arXiv preprint (2021). doi:10.48550/arXiv.2107.02027 Kuchnik et al. (2022) Michael Kuchnik, Ana Klimovic, Jiri Simsa, Virginia Smith, and George Amvrosiadis. 2022. Plumber: Diagnosing and Removing Performance Bottlenecks in Machine Learning Data Pipelines. In Proceedings of the Conference on Machine Learning and Systems (MLSys). doi:10.48550/arXiv.2111.04131 Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018. SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). doi:10.18653/V1/D18-2012 Kumar and Sivathanu (2020) Abhishek Vijaya Kumar and Muthian Sivathanu. 2020. Quiver: An Informed Storage Cache for Deep Learning. In Proceedings of the Conference on File and Storage Technologies (FAST). Leavitt (2022) Matthew Leavitt. 2022. Mosaic ResNet Deep Dive. https://w.databricks.com/blog/mosaic-resnet-deep-dive. Databricks Blog. Li et al. (2024) Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. 2024. DataComp-LM: In Search of the Next Generation of Training Sets for Language Models. In Advances in Neural Information Processing Systems (NeurIPS). doi:10.5555/3737916.3738371 Li et al. (2023) Mingzhen Li, Wencong Xiao, Hailong Yang, Biao Sun, Hanyu Zhao, Shiru Ren, Zhongzhi Luan, Xianyan Jia, Yi Liu, Yong Li, Wei Lin, and Depei Qian. 2023. EasyScale: Elastic Training with Consistent Accuracy and Improved Utilization on GPUs. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (SC ’23). doi:10.1145/3581784.3607054 Li et al. (2025) Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, Nadine Chang, Karan Sapra, Amala Sanjay Deshmukh, Tuomas Rintamaki, Matthieu Le, Ilia Karmanov, Lukas Voegtle, Philipp Fischer, De-An Huang, Timo Roman, Tong Lu, Jose M. Alvarez, Bryan Catanzaro, Jan Kautz, Andrew Tao, Guilin Liu, and Zhiding Yu. 2025. Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models. doi:10.48550/arXiv.2501.14818 Liu et al. (2025) Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. 2025. RegMix: Data Mixture as Regression for Language Model Pre-training. In Proceedings of the International Conference on Learning Representations (ICLR). doi:10.48550/arXiv.2407.01492 Liu et al. (2011) Tongping Liu, Charlie Curtsinger, and Emery D. Berger. 2011. Dthreads: Efficient Deterministic Multithreading. In Proceedings of the Symposium on Operating Systems Principles (SOSP). doi:10.1145/2043556.2043587 Ma et al. (2025) Jeffrey Jian Ma, Hengzhi Pei, Leonard Lausen, and George Karypis. 2025. Understanding Silent Data Corruption in LLM Training. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). doi:10.18653/v1/2025.acl-long.996 Maini et al. (2025) Pratyush Maini, Vineeth Dorna, Parth Doshi, Aldo Carranza, Fan Pan, Jack Urbanek, Paul Burstein, Alex Fang, Alvin Deng, Amro Abbas, Brett Larsen, Cody Blakeney, Charvi Bannur, Christina Baek, Darren Teh, David Schwab, Haakon Mongstad, Haoli Yin, Josh Wills, Kaleigh Mentzer, Luke Merrick, Ricardo Monti, Rishabh Adiga, Siddharth Joshi, Spandan Das, Zhengping Wang, Bogdan Gaza, Ari Morcos, and Matthew Leavitt. 2025. BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining. arXiv preprint (2025). doi:10.48550/arXiv.2508.10975 Maini et al. (2024) Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. 2024. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). doi:10.18653/v1/2024.acl-long.757 McKinzie et al. (2024) Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier Biard, Sam Dodge, Philipp Dufter, Bowen Zhang, Dhruti Shah, Xianzhi Du, Futang Peng, Haotian Zhang, Floris Weers, Anton Belyi, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu He, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, and Yinfei Yang. 2024. M1: Methods, Analysis & Insights from Multimodal LLM Pre-training. arXiv preprint (2024). doi:10.48550/arXiv.2403.09611 Mohan et al. (2021a) Jayashree Mohan, Amar Phanishayee, and Vijay Chidambaram. 2021a. CheckFreq: Frequent, Fine-Grained DNN Checkpointing. In Proceedings of the Conference on File and Storage Technologies (FAST). Mohan et al. (2021b) Jayashree Mohan, Amar Phanishayee, Ashish Raniwala, and Vijay Chidambaram. 2021b. Analyzing and mitigating data stalls in DNN training. Proceedings of the VLDB Endowment 14, 5 (2021). doi:10.14778/3446095.3446100 Moritz et al. (2018) Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. 2018. Ray: A Distributed Framework for Emerging AI Applications. In Proceedings of the Symposium on Operating Systems Design and Implementation (OSDI). doi:10.48550/arXiv.1712.05889 Mosaic ML Team (2022a) Mosaic ML Team. 2022a. Fast Resumption — Streaming Documentation. https://docs.mosaicml.com/projects/streaming/en/latest/distributed_training/fast_resumption.html. Mosaic ML Team (2022b) Mosaic ML Team. 2022b. Streaming: A Data Streaming Library for Efficient Neural Network Training. MosaicML ([n. d.]) MosaicML. [n. d.]. Elastic Determinism — Streaming Documentation. https://docs.mosaicml.com/projects/streaming/en/latest/distributed_training/elastic_determinism.html. Accessed: 2026-09-03. Murray et al. (2021) Derek Murray, Jiří Šimša, Ana Klimovic, and Ihor Indyk. 2021. tf.data: a machine learning data processing framework. Proceedings of the VLDB Endowment 14, 12 (2021). doi:10.14778/3476311.3476374 Murray et al. (2013) Derek G. Murray, Frank McSherry, Rebecca Isaacs, Michael Isard, Paul Barham, and Martín Abadi. 2013. Naiad: a timely dataflow system. In Proceedings of the Symposium on Operating Systems Principles (SOSP). doi:10.1145/2517349.2522738 Niklaus et al. (2026) Joel Niklaus, Guilherme Penedo, Hynek Kydlicek, Elie Bakouch, Lewis Tunstall, Ed Beeching, Thibaud Frere, Colin Raffel, Leandro von Werra, and Thomas Wolf. 2026. The Synthetic Data Playbook: Generating Trillions of the Finest Tokens. https://huggingface.co/spaces/HuggingFaceFW/finephrase. Nouaji et al. (2026) Rahma Nouaji, Stella Bitchebe, Ricardo Macedo, and Oana Balmau. 2026. MinatoLoader: Accelerating Machine Learning Training Through Efficient Data Preprocessing. In Proceedings of the European Conference on Computer Systems (EuroSys). doi:10.1145/3767295.3769376 NVIDIA (2024) NVIDIA. 2024. Sequence Packing for NeVA — NVIDIA NeMo Framework User Guide. https://docs.nvidia.com/nemo-framework/user-guide/24.09/nemotoolkit/multimodal/mllm/sequence_packing.html. Accessed: 2026-03-20. NVIDIA (2025) NVIDIA. 2025. Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv preprint (2025). doi:10.48550/arXiv.2512.20848 NVIDIA (2026) NVIDIA. 2026. Megatron-LM Data Pipeline. https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/datasets/readme.md. Olszewski et al. (2009) Marek Olszewski, Jason Ansel, and Saman Amarasinghe. 2009. Kendo: Efficient Deterministic Multithreading in Software. In Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). doi:10.1145/1508244.1508256 Penedo et al. (2024a) Guilherme Penedo, Hynek Kydlicek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024a. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Advances in Neural Information Processing Systems (NeurIPS). doi:10.5555/3737916.3738886 Penedo et al. (2024b) Guilherme Penedo, Hynek Kydlicek, Alessandro Cappelli, Thomas Wolf, and Mario Sasko. 2024b. DataTrove: Large Scale Data Processing. doi:10.5281/zenodo.11038903 Perceptron AI (2025) Perceptron AI. 2025. Isaac-0.1: A Simple Path to the Efficient Frontier of Perception. Ritter et al. (2023) Marvin Ritter, Ihor Indyk, Aayush Singh, Andrew Audibert, Anoosha Seelam, Camelia Hanes, Eric Lau, Jacek Olesiak, Jiyang Kang, and Xihui Wu. 2023. Grain - Feeding JAX Models. Robroek et al. (2026) Ties Robroek, Neil Kim Nielsen, and Pınar Tözün. 2026. TensorSocket: Shared Data Loading for Deep Learning Training. In Proceedings of the Conference on Management of Data (SIGMOD). doi:10.1145/3749185 Saltzer and Gintell (1969) Jerome H. Saltzer and John W. Gintell. 1969. The Instrumentation of Multics. In Proceedings of the Symposium on Operating Systems Principles (SOSP). doi:10.1145/961053.961104 Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). doi:10.18653/V1/P16-1162 Shoeybi et al. (2019) Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv preprint (2019). doi:10.48550/arXiv.1909.08053 Soldaini et al. (2024) Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. 2024. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). doi:10.18653/v1/2024.acl-long.840 Strati et al. (2025) Foteini Strati, Michal Friedman, and Ana Klimovic. 2025. PCcheck: Persistent Concurrent Checkpointing for ML. In Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). doi:10.1145/3669940.3707255 Tschannen et al. (2025) Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. 2025. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. arXiv preprint (2025). doi:10.48550/arXiv.2502.14786 Um et al. (2023) Taegeon Um, Byungsoo Oh, Byeongchan Seo, Minhyeok Kweun, Goeun Kim, and Woo-Yeon Lee. 2023. FastFlow: Accelerating Deep Learning Model Training with Smart Offloading of Input Data Pipeline. Proceedings of the VLDB Endowment 16, 5 (2023). doi:10.14778/3579075.3579083 von Werra et al. (2026) Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2026. TRL: Transformers Reinforcement Learning - DataCollatorForVisionLanguageModeling. Wagenländer et al. (2024) Marcel Wagenländer, Guo Li, Bo Zhao, Luo Mai, and Peter Pietzuch. 2024. Tenplex: Dynamic Parallelism for Deep Learning Using Parallelizable Tensor Collections. In Proceedings of the Symposium on Operating Systems Principles (SOSP). doi:10.1145/3694715.3695975 Wan et al. (2025a) Borui Wan, Mingji Han, Yiyao Sheng, Yanghua Peng, Haibin Lin, Mofan Zhang, Zhichao Lai, Menghan Yu, Junda Zhang, Zuquan Song, Xin Liu, and Chuan Wu. 2025a. ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model Development. In Proceedings of the Symposium on Networked Systems Design and Implementation (NSDI). doi:10.48550/arXiv.2407.20143 Wan et al. (2025b) Borui Wan, Gaohong Liu, Zuquan Song, Jun Wang, Yun Zhang, Guangming Sheng, Shuguang Wang, Houmin Wei, Chenyuan Wang, Weiqiang Lou, Xi Yang, Mofan Zhang, Kaihua Jiang, Cheng Ren, Xiaoyun Zhi, Menghan Yu, Zhe Nan, Zhuolin Zheng, Baoquan Zhong, Qinlong Wang, Huan Yu, Jinxin Chi, Wang Zhang, Yuhan Li, Zixian Du, Sida Zhao, Yongqiang Zhang, Jingzhe Tang, Zherui Liu, Chuan Wu, Yanghua Peng, Haibin Lin, Wencong Xiao, Xin Liu, and Liang Xiang. 2025b. Robust LLM Training Infrastructure at ByteDance. In Proceedings of the Symposium on Operating Systems Principles (SOSP). doi:10.1145/3731569.3764838 Wang et al. (2025a) Jiachen T. Wang, Dawn Song, James Zou, Prateek Mittal, and Ruoxi Jia. 2025a. Capturing the Temporal Dependence of Training Data Influence. In Proceedings of the International Conference on Learning Representations (ICLR). doi:10.48550/arXiv.2412.09538 Wang et al. (2025b) Weizhi Wang, Yu Tian, Linjie Yang, Heng Wang, and Xifeng Yan. 2025b. Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resources. arXiv preprint (2025). doi:10.48550/arXiv.2504.00595 Wang et al. (2023) Zhuang Wang, Zhen Jia, Shuai Zheng, Zhen Zhang, Xinwei Fu, T. S. Eugene Ng, and Yida Wang. 2023. GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints. In Proceedings of the Symposium on Operating Systems Principles (SOSP). doi:10.1145/3600006.3613145 Wen et al. (2025) Kaiyue Wen, Zhiyuan Li, Jason S. Wang, David Leo Wright Hall, Percy Liang, and Tengyu Ma. 2025. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective. In Proceedings of the International Conference on Learning Representations (ICLR). doi:10.48550/arXiv.2410.05192 Xie et al. (2022) Jian Xie, Jingwei Xu, Guochang Wang, Yuan Yao, Zenan Li, Chun Cao, and Hanghang Tong. 2022. A Deep Learning Dataloader with Shared Data Preparation. In Advances in Neural Information Processing Systems (NeurIPS). Xie et al. (2023) Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. 2023. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. In Advances in Neural Information Processing Systems (NeurIPS). doi:10.48550/arXiv.2305.10429 Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. 2025. Qwen3 Technical Report. arXiv preprint (2025). doi:10.48550/arXiv.2505.09388 Yang and Cong (2019) Chih-Chieh Yang and Guojing Cong. 2019. Accelerating Data Loading in Deep Neural Network Training. In Proceedings of the IEEE International Conference on High Performance Computing, Data, and Analytics (HiPC). doi:10.1109/hipc.2019.00037 Ye et al. (2025) Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. 2025. Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance. In Proceedings of the International Conference on Learning Representations (ICLR). doi:10.48550/arXiv.2403.16952 Zaharia et al. (2016) Matei Zaharia, Reynold S. Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J. Franklin, Ali Ghodsi, Joseph Gonzalez, Scott Shenker, and Ion Stoica. 2016. Apache Spark: a unified engine for big data processing. Commun. ACM 59, 11 (2016), 56–65. doi:10.1145/2934664 Zhai et al. (2022) Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. 2022. LiT: Zero-Shot Transfer with Locked-Image Text Tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/CVPR52688.2022.01759 Zhang et al. (2025) Zili Zhang, Yinmin Zhong, Yimin Jiang, Hanpeng Hu, Jianjian Sun, Zheng Ge, Yibo Zhu, Daxin Jiang, and Xin Jin. 2025. DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Models. In Proceedings of the SIGCOMM Conference. doi:10.1145/3718958.3750472 Zhao et al. (2026) Juntao Zhao, Qi Lu, Wei Jia, Borui Wan, Lei Zuo, Junda Feng, Jianyu Jiang, Yangrui Chen, Shuaishuai Cao, Jialing He, Kaihua Jiang, Yuanzhe Hu, Shibiao Nong, Yanghua Peng, Haibin Lin, Xin Liu, and Chuan Wu. 2026. MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Training. In Proceedings of the European Conference on Computer Systems (EuroSys). doi:10.48550/arXiv.2504.09844 Zhao et al. (2024a) Mark Zhao, Emanuel Adamiak, and Christos Kozyrakis. 2024a. cedar: Optimized and Unified Machine Learning Input Data Pipelines. Proceedings of the VLDB Endowment 18, 2 (2024). doi:10.14778/3705829.3705861 Zhao et al. (2022) Mark Zhao, Niket Agarwal, Aarti Basant, Buğra Gedik, Satadru Pan, Mustafa Ozdal, Rakesh Komuravelli, Jerry Pan, Tianshu Bao, Haowei Lu, Sundaram Narayanan, Jack Langman, Kevin Wilfong, Harsha Rastogi, Carole-Jean Wu, Christos Kozyrakis, and Parik Pol. 2022. Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training. In Proceedings of the Annual International Symposium on Computer Architecture (ISCA). doi:10.1145/3470496.3533044 Zhao et al. (2024b) Yu Zhao, Yuanbin Qu, Konrad Staniszewski, Szymon Tworkowski, Wei Liu, Piotr Miłoś, Yuxiang Wu, and Pasquale Minervini. 2024b. Analysing The Impact of Sequence Composition on Language Model Pre-Training. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). doi:10.18653/v1/2024.acl-long.427 Zhu et al. (2025) Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. 2025. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. arXiv preprint (2025). doi:10.48550/arXiv.2504.10479 Zhu et al. (2018) Yue Zhu, Fahim Chowdhury, Huansong Fu, Adam Moody, Kathryn Mohror, Kento Sato, and Weikuan Yu. 2018. Entropy-Aware I/O Pipelining for Large-Scale Deep Learning on HPC Systems. In Proceedings of the International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS). doi:10.1109/mascots.2018.00023 Zhuang et al. (2022) Donglin Zhuang, Xingyao Zhang, Shuaiwen Song, and Sara Hooker. 2022. Randomness in Neural Network Training: Characterizing the Impact of Tooling. In Proceedings of the Conference on Machine Learning and Systems (MLSys). doi:10.48550/arXiv.2106.11872