Paper deep dive
Training Language Models via Neural Cellular Automata
Dan Lee, Seungwook Han, Akarsh Kumar, Pulkit Agrawal
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/13/2026, 12:55:31 AM
Summary
The paper introduces Neural Cellular Automata (NCA) as a synthetic, non-linguistic data source for 'pre-pre-training' large language models. By training on NCA dynamics, models acquire transferable computational priors that improve downstream language modeling performance and convergence speed across web text, math, and code, while also enhancing reasoning capabilities on benchmarks like GSM8K and HumanEval.
Entities (6)
Relation Signals (3)
Neural Cellular Automata → enhancesbenchmarkscore → GSM8K
confidence 95% · On GSM8K, NCA pre-pre-training improves accuracy from 3.8% to 4.4% at pass@1
Neural Cellular Automata → enhancesbenchmarkscore → HumanEval
confidence 95% · On HumanEval, NCA pre-pre-training improves pass@1
Neural Cellular Automata → improvesperformanceof → Large Language Models
confidence 95% · NCA pre-pre-training improves downstream language modeling by up to 6% and accelerates convergence by up to 1.6x.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Pre-training is crucial for large language models (LLMs), as it is when most representations and capabilities are acquired. However, natural language pre-training has problems: high-quality text is finite, it contains human biases, and it entangles knowledge with reasoning. This raises a fundamental question: is natural language the only path to intelligence? We propose using neural cellular automata (NCA) to generate synthetic, non-linguistic data for pre-pre-training LLMs--training on synthetic-then-natural language. NCA data exhibits rich spatiotemporal structure and statistics resembling natural language while being controllable and cheap to generate at scale. We find that pre-pre-training on only 164M NCA tokens improves downstream language modeling by up to 6% and accelerates convergence by up to 1.6x. Surprisingly, this even outperforms pre-pre-training on 1.6B tokens of natural language from Common Crawl with more compute. These gains also transfer to reasoning benchmarks, including GSM8K, HumanEval, and BigBench-Lite. Investigating what drives transfer, we find that attention layers are the most transferable, and that optimal NCA complexity varies by domain: code benefits from simpler dynamics, while math and web text favor more complex ones. These results enable systematic tuning of the synthetic distribution to target domains. More broadly, our work opens a path toward more efficient models with fully synthetic pre-training.
Tags
Links
- Source: https://arxiv.org/abs/2603.10055v1
- Canonical: https://arxiv.org/abs/2603.10055v1
Trouble viewing inline? Open PDF directly →
Full Text
66,458 characters extracted from source content.
Expand or collapse full text
Training Language Models via Neural Cellular Automata Dan Lee * 1 Seungwook Han * 2 3 Akarsh Kumar 2 Pulkit Agrawal 2 3 1 Independent Contributor 2 MIT 3 Improbable AI Lab Abstract Pre-training is crucial for large language models (LLMs), as it is when most representations and capabilities are acquired. However, natural language pre-training has problems: high-quality text is finite, it contains human biases, and it entangles knowledge with reasoning. This raises a fundamental question: is natural language the only path to intelligence? We propose using neural cellular automata (NCA) to generate synthetic, non-linguistic data for pre-pre-training LLMs–training on synthetic-then-natural language. NCA data exhibits rich spatiotemporal structure and statistics resembling natural language while being controllable and cheap to generate at scale. We find that pre-pre-training on only 164M NCA tokens improves downstream language modeling by up to 6% and accelerates convergence by up to 1.6×. Surprisingly, this even outperforms pre-pre-training on 1.6B tokens of natural language from Common Crawl with more compute. These gains also transfer to reasoning benchmarks, including GSM8K, HumanEval, and BigBench-Lite. Investigating what drives transfer, we find that attention layers are the most transferable, and that optimal NCA complexity varies by domain: code benefits from simpler dynamics, while math and web text favor more complex ones. These results enable systematic tuning of the synthetic distribution to target domains. More broadly, our work opens a path toward more efficient models with fully synthetic pre-training. Website: https://hanseungwook.github.io/blog/ nca-pre-pre-training/ Code: https://github.com/danihyunlee/ nca-pre-pretraining * Equal contribution. Correspondence to: Dan Lee <dhl2134@columbia.edu>, Seungwook Han <swhan@mit.edu>. 1. Introduction Baseline NCA PPT Accuracy WebText Code NCA Complexity Pre-training Tokens Loss Pre-Pre-TrainingPre-Training Time Natural Language Data LLM f θ i θ i Space of NCAs LowMedium Complexity High Complexity Band X i ∈ℤ H×W×T <k WebTextMathCode By the end of 2017, every smartphone manufacturer has embraced the 18:9 aspect ratio screen model... Whenever I start on a large refactoring, I need to justify that the exercise... from typing import List def has_close_elements( numbers: List[float], threshold: float ) -> bool: """ check if in given list of numbers, are any two are closer to each other than the threshold """ ... Let be a sequence of positive integers such that Prove that there exist indices for which is a perfect square. Solution: Consider the partial sums a 1 ,a 2 ,...,a n a 1 +a 2 +⋯+a n =n 2 . i<j a i +a i+1 +⋯+a j S k =a 1 +a 2 +⋯+a k LLM ...... Figure 1.Overview of NCA Pre-pre-training to Language Pre- training. We pre-pre-train a transformer with next-token pre- diction on the dynamics of neural cellular automata (NCA) sam- pled from selected complexity regions. We then conduct standard pre-training on natural language corpora. NCA pre-pre-training improves both validation perplexity and convergence speed on language pre-training. Interestingly, the optimal NCA distribution varies by downstream domain. Scale has transformed neural networks, enabling emergent abilities like reasoning (Jaech et al., 2024; Jiang, 2023; Austin et al., 2021) and in-context learning (Brown et al., 2020; Wei et al., 2022; Zhao et al., 2024) in large lan- guage models (LLMs). However, neural scaling laws predict that continued improvements require exponentially more data (Kaplan et al., 2020), which is nearing exhaustion by 2028 (Villalobos et al., 2022). Furthermore, natural lan- guage inherits many undesirable human biases and needs tedious data curation and cleaning before it is used for train- ing foundation models (Han et al., 2025a; An et al., 2024). This raises a fundamental question: Is natural language the only path to learning useful representations? In this paper, we explore an alternative path to using synthetic data from cellular automata. Our core hypothesis is that the emergence of reasoning and other abilities in LLMs relies on the underlying structure 1 arXiv:2603.10055v1 [cs.LG] 9 Mar 2026 Training Language Models via Neural Cellular Automata of natural language, rather than its semantics. Text is a lossy record of human cognition and the world it describes, containing diverse kinds of structure, from reasoning traces to procedural instructions (Ribeiro et al., 2023; Ruis et al., 2024; Cheng et al., 2025; Del ́ etang et al., 2024). Next-token prediction on such data pressures models to internalize the latent computational processes that support coherent contin- uations, fostering key capabilities of intelligence (Del ́ etang et al., 2023; Jiang, 2023). If the key ingredient is exposure to various structures rather than language semantics, then richly structured non- linguistic data could also be effective for teaching models to reason. To investigate this hypothesis, we employ algo- rithmically generated synthetic data from neural cellular automata (NCA) (Mordvintsev et al., 2020) as a synthetic training substrate. NCA generalize systems like Conway’s Game of Life (Gardner, 1970) by replacing fixed dynam- ics rules with neural networks and can be used to generate diverse data distributions with spatially local rules. This produces long-range spatio-temporal patterns (see Figure 1) of arbitrary sizes that exhibit heavy-tailed, Zipfian token distributions (see Figure 8 in Appendix A) reminiscent of natural data. Crucially, we propose a method to explicitly control the complexity of NCA, enabling systematic tun- ing of the synthetic data distribution for optimal transfer to downstream domains. Prior work on synthetic pre-training has explored ap- proaches like generating random strings with a recurrent network (Bloem, 2025) and simple algorithmic tasks (Wu et al., 2022; Shinnick et al., 2025a), but they have yet to match or outperform language training under matched token budgets. We hypothesize this is because such synthetic dis- tributions are narrow and homogeneous, lacking certain key properties that characterize natural language. NCAs address this gap. The parametric structure of NCA yields diverse dynamics and allows systematic control over complexity. This enables us to ask not only whether synthetic data can transfer, but what structural properties make it effective. We adopt a pre-pre-training framework: an initial phase of training on NCA dynamics that precedes standard pre- training on natural language (Hu et al., 2025b). Our ultimate vision is to pre-train entirely on clean synthetic data, fol- lowed by fine-tuning on a limited and curated corpora of natural language to acquire semantics (Han et al., 2025a). The pre-pre-training framework serves as an early prototype of this paradigm, allowing us to measure how computational primitives learned from synthetic NCA transfer to language tasks. Our contributions are as follows: 1. A synthetic pre-pre-training substrate that trans- fers to language and reasoning. We propose neural cellular automata (NCA) as a fully algorithmic, non- linguistic data source for pre-pre-training. NCA pre- pre-training improves downstream language modeling by up to 6% and converges up to 1.6×faster across web text, math, and code. These perplexity gains trans- fer to reasoning across benchmarks including GSM8K, HumanEval, and BigBench-Lite. Surprisingly, it out- performs pre-pre-training on natural language (C4), even with more data and compute. 2. Synthetic pre-training enables domain-targeted data design. We find that the optimal NCA complexity regime varies by downstream task: code benefits from lower-complexity rules while math and web text ben- efit from higher-complexity ones. NCAs’ parametric structure offers a new lever for efficient training: tun- ing the complexity of training distributions to match the computational character of target domains. 3.Attention captures the most transferable priors. The attention layers capture the most useful com- putational primitives, accounting for the majority of the transfer gains.Attention appears to be a universal carrier of transferable capabilities such as long-range dependency tracking and in-context learn- ing, whereas MLPs encode more domain-specific knowledge–making MLP transfer conditional on align- ment between the synthetic and target domains. 2. Related Works Synthetic data is a broad umbrella term encompassing a wide spectrum of artificially generated data, ranging from using LLMs (Nad ˇ as ̧et al., 2025; Wang et al., 2023; Xu et al., 2025; Mukherjee et al., 2023; Li et al., 2023; Lu et al., 2024; Wei & Zou, 2019) to simple algorithms to generate data. In this work, we pursue the latter, a non-linguistic approach. Algorithm-Based Synthetic DataSome works have gone beyond natural data altogether, using simple algorithmic pro- cedures (e.g., OpenGL shader images) to generate synthetic training data (Baradad et al., 2022). Past works have trained vision models on data generated by simple processes like fractals, dead leaves, and wavelet models (Kataoka et al., 2020; Baradad Jurjo et al., 2021; Baradad et al., 2022). De- spite lacking semantic content, these models learn represen- tations that transfer well to real images. Baradad Jurjo et al. (2021) argue that what matters is not natural data per se, but naturalistic data, i.e. data that reproduces the statistical structure of the natural world, such as the approximate scale- invariance (Field, 1987) or the Zipfian distribution (Zipf, 1949; Chan et al., 2022). In the language domain, using algorithmically generated data is less common, as language is thought to be uniquely complex. Nevertheless, some works have explored this 2 Training Language Models via Neural Cellular Automata approach (Saxton et al., 2019; Desai et al., 2015; Papadim- itriou & Jurafsky, 2023). Chiang & yi Lee (2020); Hu et al. (2025a) showed that pre-training LLMs on synthetic data generated by context-free grammars can be useful for nat- ural language modeling. Berkovich & Buehler (2025) and Berkovich et al. (2025) trained LLMs on cellular automata, but did not study the usefulness of the learned representa- tions for language. Shared Underlying Computation A growing body of work suggests that neural networks learn general compu- tations that transfer across domains, raising the possibility that synthetic algorithmic data could instill such primitives directly. Lu et al. (2022) show that LLMs trained on natu- ral language can transfer to seemingly unrelated domains like vision and protein folding, and Huh et al. (2024) il- lustrate that foundation models across different modalities are converging in representation, hinting that they are learn- ing a common structure. Going further, Mirchandani et al. (2023) show that, even without fine-tuning, LLMs already have in-context learning capabilities for symbolic reasoning, numeric pattern continuation, and robotic control. These works cast LLMs as universal computation engines (Lu et al., 2022). Other works have shown transfer from natural language to robotic RL environments (Reid et al., 2022). Zhang et al. (2024) showed that training LLMs on elemen- tary cellular automata allows them to better transfer to chess. Emergent Complexity A central puzzle for algorithmic synthetic data is how simple procedures can give rise to data with rich structure, resembling the complexity of the real world. This echoes a deeper observation about nature itself: despite its diversity, the universe appears governed by simple underlying laws (Wigner et al., 1990; Tegmark, 2008), and may even be fundamentally computational (Wol- fram, 2020). Researchers have developed various measures to quantify such complexity (Lloyd, 2001; Mitchell, 2009), including Kolmogorov complexity (Kolmogorov, 1963), so- phistication (Mota et al., 2013), and assembly index (Sharma et al., 2023). More recently, epiplexity was introduced as a complexity metric for computationally bounded observers (Finzi et al., 2026), demonstrating that simple deterministic rules can produce data useful for learning useful represen- tations. These works suggest that NCAs, despite having simple local rules, can generate arbitrarily complex struc- tures when rolled out over long time horizons, making them a promising source of synthetic training data. 3. Method We study whether neural cellular automata (NCA) can create training data that teaches transferable computational priors useful for language modeling. 3.1. Neural Cellular Automata (NCA) NCA is a generalization of classical cellular automata (Wol- fram, 1984), where the update rule is parametrized as a neural network, allowing the dynamics to be diversely sam- pled rather than hand-designed. Random Discrete NCA.We use 2D discrete neural cellu- lar automata on a12×12grid with periodic boundaries and an = 10state alphabet, where each cell is represented as a 10-dimensional one-hot vector. The transition dynamics are governed by a neural networkf θ that maps each cell’s3×3 neighborhood to logits over next-cell states: c (t+1) i ∼ softmax f θ c (t) N(i) /τ ,(1) wherec (t) i is the state of celliat timet,N(i)denotes its neighborhood, andτ = 10 −3 introduces mild stochasticity. We parameterizef θ as a3×3convolution (4 channels) fol- lowed by a cell-wise MLP with hidden size 16 and ReLU activation, producing 10 logits per cell. Complexity-based sampling. To generate diverse train- ing data, we sample both the transition rules and initial conditions. For each sequence, we randomly initialize the parametersθof the transition network and sample the ini- tial gridc (0) i.i.d. uniform over0,..., 9. This proce- dure yields a distribution over NCA dynamics ranging from trivially predictable (fixed points or short cylce) to highly chaotic and unpredictable. To sample NCA dynamics with controlled structural com- plexity, we sample rules based on the gzip compression ratios of generated sequences. For rollouts, we serial- ize all timesteps into a byte stream and computer = compressed bytes/raw bytes ∗ 100. We retain NCAs with trajectories of r > 50%. gzip under the hood combines Lempel-Ziv compression (Ziv & Lempel, 1977) with Huffman coding. Since Lempel- Ziv compression provides a computable upper bound on Kolmogorov complexity (Li & Vit ́ anyi, 2019), gzip com- pression ratio serves as a practical measure of intrinsic se- quence complexity. Compressible sequences exhibit simpler and more predictable structure, whereas incompressible se- quences are more chaotic, as seen in Figure 1. 3.2. Tokenization Patch vocabulary. We tokenize each grid using non- overlapping2×2patches, following the patch-based tok- enization for vision transformers (Dosovitskiy et al., 2021). Each patch contains four cells in0,..., 9and is mapped bijectively to an integer token, yielding a fixed vocabulary of 10 4 patch tokens. We serialize each timestep in row-major 3 Training Language Models via Neural Cellular Automata order with<grid>and</grid>delimiters, and concate- nate timesteps to form sequences of up to 1024 tokens. 3.3. Training Objective and Interpretation We train a transformer autoregressively on the tokenized trajectoryx = (x 1 ,...,x N ) using cross-entropy loss: L =− N X i=1 logp φ (x i | x <i ).(2) Since each sampledθdefines a distinct dynamics rule, next- token prediction requires inferring the latent rule in context and applying it within the same sequence. This aligns NCA training with the Bayesian view of in-context learning (Han et al., 2025b; Xie et al., 2022): 4. Experimental Setup 4.1. Training Paradigm We adopt a three-stage training paradigm (Shinnick et al., 2025a; Hu et al., 2025b; Bloem, 2025): 1.Pre-pre-training: An initial training phase designed to instill transferable computational priors before the main pre-training stage. In this work, we propose using synthetic, non-linguistic data (NCA trajectories) for pre-pre-training. 2.Pre-training: Standard large-scale training on natural language corpora (web text, code, or math) to acquire linguistic knowledge. 3.Fine-tuning: Task-specific adaptation (e.g., instruc- tion tuning). This work studies the transfer from stage 1 to stage 2 and 3: whether computational structure learned from synthetic data can accelerate and improve language model pre-training, and how it manifests in downstream reasoning benchmarks. 4.2. Setup We generate NCA data by randomly sampling neural net- work weights that define the transition rule. Each trajectory is thus produced by a unique rule, ensuring diversity across the training distribution. We pre-pre-train a Llama-based transformer (Touvron et al., 2023) (1.6B parameters, 24 layers, 32 heads, 2048 hidden dimension, weight-tying) on 164M NCA tokens sampled at the50%+gzip compressibil- ity band, unless otherwise noted. We measure transfer by conducting pre-training on three downstream corpora: Open- WebText (Gokaslan et al., 2019), OpenWebMath (Paster et al., 2023), and CodeParrot (Tunstall et al., 2022). We transfer all model weights except the embedding layers, which are re-initialized for the natural language vocabulary. All parameters are updated during pre-training. For downstream reasoning benchmarks, we evaluate on GSM8K (Cobbe et al., 2021), HumanEval (Chen, 2021), and BigBench-Lite (Srivastava et al., 2023). We fine-tune the models on the train set for instruction following for GSM8K and BigBench-Lite. We provide the details on our stage 3 pipeline in Appendix C. 4.3. Baselines We compare against two baselines: (i) No pre-pre-training (scratch): the model is randomly initialized and trained directly on the pre-training corpora, establishing whether pre-pre-training provides any benefit; (i) Dyck pre-pre- training: pre-pre-training on K-Dyck, a synthetic formal language studied in Hu et al. (2025b), testing how NCA pre-pre-training compares against an alternative synthetic data approach. We generate our pre-pre-training data using k = 128and infinite potential depth; (i) C4 pre-pre- training: pre-pre-training on natural language data (C4; Raffel et al. (2020)) with matched token budgets, testing how NCA pre-pre-training compares against natural lan- guage pre-pre-training. 4.4. Hyperparameters For both our pre-pre-training and baseline runs, we perform a grid search over learning rate and weight decay and use the best hyperparameters for each method. For pre-training, we train three separate models for a single epoch on each dataset: OpenWebText (9B tokens), OpenWebMath (4B to- kens), and CodeParrot (13B tokens). We report the detailed hyperparameters in Table 2 in Appendix B. For downstream reasoning benchmarks, we use models pre- trained on OpenWebText, OpenWebMath, and CodeParrot for Big-Bench-Lite, GSM8K, and HumanEval, respectively: matching the benchmark domain approximately to the pre- training domain. We report details on fine-tuning in Ap- pendix C. 4.5. Evaluation Metrics for Transfer We measure the transfer between our pre-pre-training and pre-training by mainly studying validation perplexity on a held-out set and convergence speed (Bloem, 2025; Hu et al., 2025b; Kaplan et al., 2020). We quantify convergence speed by comparing the number of tokens to reach the final perplexity of the scratch model. On downstream reasoning tasks, our primary evaluation metric is pass accuracy with multiple decodings or pass@k(Chen et al., 2021). For BigBench tasks, we compute accuracy on a normalized basis to adjust for random guessing associated with multiple- choice questions (Srivastava et al., 2023). 4 Training Language Models via Neural Cellular Automata 5. Results We present results on the impact of NCA pre-pre-training on downstream language modeling, and analyze how transfer varies across scale and data complexity. 5.1. NCA Pre-Pre-Training Improves Language Modeling We compare the language modeling performance against three baselines: (i) no pre-pre-training (“scratch”), (i) pre- pre-training on another natural language dataset C4, and (i) pre-pre-training on another synthetic language Dyck. As seen in Figure 3, across model scales (400M, 600M, 1.6B) and multiple random seeds, NCA pre-pre-trained models consistently outperform all three baselines (scratch, Dyck, and natural language). On OpenWebText, the best- performing NCA pre-pre-trained 400M model improves downstream perplexity upon the scratch baseline by 8.6%, and the 1.6B model improves by 5.7%. The relative gain decreases with scale, which is expected as larger models provide a stronger baseline and incremental perplexity im- provements become progressively harder to obtain. The improvements nonetheless remain consistent across seeds, indicating that the effect is robust rather than a fragile arti- fact of initialization. The most surprising observation is that pre-pre-training on NCA outperforms pre-pre-training on natural language (C4) under matched token and compute budgets. To further study this, we compared NCA pre-pre-training (160M to- kens) against C4 with significantly more data (1.6B tokens), with and without transferring the pre-trained embedding layers in Figure 4. NCA pre-pre-training improves upon this baseline by 5% on perplexity and converges 1.4×faster. We hypothesize this reflects what each data source teaches at each scale: C4 may emphasize shallow syntactic patterns, while NCA directly trains long-range dependency tracking and latent rule inference. We return to this discussion in Section 6. This transfer to natural language holds across the training and generalizes across different domains of math and code, as shown in Figure 2. These training curves show that NCA pre-pre-training lowers validation perplexity on OpenWeb- Math and CodeParrot by4− 5%by the end of convergence and with the NCA pre-pre-training we can achieve up to 1.6×faster convergence. These results demonstrate that the transfer is not specific to a single downstream domain but generalizes across different natural language distribu- tions. This is also not a short-lived initialization effect. The perplexity advantage persists and often grows throughout training, indicating that NCA pre-pre-training genuinely increases token efficiency. 5.2. Language Modeling Gains Translate to Downstream Reasoning Perplexity measures language modeling quality, but it is a proxy for the capabilities we ultimately care about. To assess whether these gains translate into task-level improve- ments, we evaluate on downstream reasoning benchmarks in Table 1. On GSM8K, NCA pre-pre-training improves accuracy from 3.8% to 4.4% at pass@1 and from 36.6% to 37.9% at pass@32, with gains growing slightly at higher pass@k. On HumanEval, NCA pre-pre-training improves pass@1 but the advantage diminishes at higher k. Interest- ingly, Dyck pre-pre-training is competitive with NCA on HumanEval at higher pass@k, and slightly outperforms it at pass@16 and pass@32. This likely reflects the structural overlap between Dyck languages and code, both of which require tracking nesting logic and delimiter patterns. On BigBench-Lite, pass@1 is comparable across all methods, but NCA pre-pre-training outperforms markedly at higherk, reaching 36.5% at pass@4 compared to 29.7% for C4 and 25.9% for the scratch baseline. Overall, these results demon- strate that NCA pre-pre-training transfers to downstream reasoning across math, logic, and code. Math (GSM8K) pass@kScratchC4NCADyck 13.8%± 0.1% 3.8%± 0.2% 4.4%± 0.3% 4.1%± 0.4% 817.9%± 0.3% 17.8%± 0.5% 19.2%± 0.3% 18.6%± 0.9% 1626.5%± 0.5% 26.3%± 0.6% 27.8%± 0.3% 27.3%± 0.9% 3236.6%± 0.6% 36.2%± 0.9% 37.9%± 0.3% 37.4%± 0.7% Coding (HumanEval) pass@kScratchC4NCADyck 16.8%± 0.6% 6.3%± 0.3% 7.5%± 0.4% 6.9%± 0.1% 811.2%± 0.6% 10.5%± 0.5% 11.4%± 0.8% 11.3%± 0.2% 1612.6%± 0.6% 11.6%± 0.6% 12.6%± 0.9% 12.8%± 0.2% 3213.9%± 0.5% 12.6%± 0.7% 13.8%± 1.0% 14.3%± 0.4% Reasoning (BigBench-Lite) pass@kScratchC4NCADyck 115.4%± 1.1% 15.9%± 0.9% 15.0%± 1.2% 13.4%± 2.8% 220.9%± 2.5% 22.8%± 1.2% 26.5%± 1.0% 18.1%± 2.3% 425.9%± 3.9% 29.7%± 1.3% 36.5%± 2.1% 22.7%± 2.1% Table 1.NCA pre-pre-training improves performance on down- stream reasoning benchmarks. We report the mean pass@k± std over 4 training seeds. 5.3. What Drives Transfer? The preceding results establish that NCA pre-pre-training improves both language modeling and reasoning. We now investigate the mechanism: which model components cap- ture the transferable structure, and what properties of the synthetic data control transfer effectiveness? 5 Training Language Models via Neural Cellular Automata 2.0B4.0B6.0B8.0B Tokens Observed 20 50 Perplexity a 1.5× faster 6% better OpenWebText 1.0B2.0B3.0B4.0B Tokens Observed 10 20 b 1.4× faster 5% better OpenWebMath 2.0B4.0B6.0B8.0B10.0B12.0B Tokens Observed 1.9 2.0 2.1 2.2 2.3 2.4 2.5 c 1.6× faster 4% better CodeParrot 6.0B7.0B8.0B 13.5 14.0 14.5 15.0 15.5 16.0 Zoom 3.0B3.5B4.0B 7.4 7.6 7.8 8.0 8.2 8.4 8.6 8.8 9.0 Zoom 9.0B10.0B11.0B12.0B 1.75 1.80 1.85 1.90 1.95 2.00 2.05 Zoom ScratchC4 Pre-pre-trainNCA Pre-pre-trainDyck Pre-pre-train Figure 2.NCA pre-pre-training improves and accelerates language model pre-training across diverse domains. We show the validation perplexity during pre-training on (a) OpenWebText, (b) OpenWebMath, and (c) CodeParrot for 1.6B parameter models. Models pre-pre-trained on NCA trajectories consistently outperform the scratch, Dyck pre-pre-training, and surprisingly even C4 pre-pre-training baselines. NCA pre-pre-training achieves 1.4–1.6×faster convergence to the scratch baseline’s final perplexity while also reaching up to 6% lower final perplexity. We provide a zoomed-in training curve of the last third of training for clarity. 400M600M1.6B Model Size 13 14 15 16 17 18 19 Perplexity Scratch C4 Pre-pre-train NCA Pre-pre-train Dyck Pre-pre-train How does NCA/Dyck pre-pre-training scale across model sizes? Figure 3.NCA pre-pre-training improves language model train- ing performance across model sizes (Section 5.1). We report the final validation perplexity after pre-training on OpenWebText across (400M, 600M, and 1.6B parameter models). At 164M to- kens, C4 pre-pre-training likely acquires shallow syntactic patterns that interfere with downstream learning rather than transferable structure. We investigate this further in Figure 4. 5.3.1. ATTENTION CAPTURES THE MOST TRANSFERABLE PRIMITIVES. To isolate which model components carry the transfer sig- nal, we selectively re-initialize subsets of weights after NCA pre-pre-training and measure the impact on language model- ing. As shown in Figure 5, re-initializing attention weights causes the largest degradation in transfer across all configu- rations, indicating that attention captures the most transfer- able computational primitives. The role of other components is more nuanced. On Open- WebText, retaining MLP and LayerNorm weights degrades transfer, suggesting these layers encode NCA-specific statis- tics that interfere with language learning. On CodeParrot, these components have negligible effect. This asymmetry suggests that attention is more transferable across domains, whereas MLP layers are contingent on whether the domain- specific priors align with the target task. These findings align with concurrent work identifying at- tention as the primary locus of transferable structure in synthetic pre-training (Shinnick et al., 2025a;b). They also resonate with recent analyses of Mixture-of-Expert architec- tures, which demonstrate that scaling MLP parameters pri- marily enhance memorization rather than reasoning (Jelassi et al., 2025). Together, these results suggest a functional division: attention layers learn general-purpose mechanisms for tracking dependencies and inferring latent rules, while MLP layers specialize in storing domain-specific patterns and statistics. This division may explain why attention transfers universally from NCA to language, whereas MLP weights can introduce interference when the source and target domains differ substantially. 5.3.2. DATA COMPLEXITY MODULATES TRANSFER AND THE OPTIMUM IS DOMAIN-DEPENDENT. Having established that attention carries the most transfer- able signal, we next ask: what properties of NCA data affect the transfer? We analyze complexity along two comple- mentary axes: gzip compressibility (as an upper bound to Kolmogorov complexity) and alphabet sizen(size of the state space). Complexity via gzip. We generate NCA trajectories of varying gzip compressibility bands (20–30%, 30–40%, 40–50%, 50%+). Smaller compression ratios imply regular, 6 Training Language Models via Neural Cellular Automata 2.0B4.0B6.0B8.0B Tokens Observed 20 50 Perplexity 1.4× faster 6% better OpenWebText Scratch NCA Pre-pre-train C4 Pre-pre-train (1.6B tokens) C4 Pre-pre-train (1.6B tokens without Reinit Embeddings) Dyck Pre-pre-train Figure 4.Pre-pre-training on 160M tokens of NCA is better than pre-pre-training on 1.6B tokens of natural language (C4). We report the validation perplexity during pre-training on Open- WebText. Perplexity improvement is calculated relative to the C4 pre-pre-trained model. We add a version where we also pre- serve the embedding layers from pre-pre-training to pre-training (1.6B tokens w/o embedding reinit). Surprisingly even with the embedding layers, NCA pre-pre-training is better. low-entropy dynamics that are more predictable, whereas larger compression ratios generate more diverse and unpre- dictable trajectories with richer spatiotemporal structure, as illustrated in Figure 1. As shown in Figure 6, the optimal complexity band varies by downstream domain. OpenWebText benefits from less compressible (more complex) NCA rules in the 50%+ band, while CodeParrot shows a sweet spot at intermediate com- plexity (30–40% gzip). Strikingly, this aligns with the intrin- sic complexity of the target corpora themselves in Figure 8 of Appendix A: OpenWebText and OpenWebMath have gzip ratios of 60–70%, whereas CodeParrot is substantially more compressible at 32%. The correlation is somewhat direct. Domains with higher intrinsic complexity benefit from higher-complexity synthetic data, and vice versa. This suggests a plausible and practical principle: matching the complexity of synthetic pre-training data to the target domain maximizes transfer. Rule space expressiveness via alphabet size. We vary the NCA state alphabetn∈2, 10, 15, which controls the di- versity of possible local interactions. As shown in Figure 7, larger alphabets (n = 10, 15) exhibit diminishing returns: performance improves the most at an intermediate NCA token budget, then the improvement gap narrows. Surpris- ingly, the smallest alphabet (n = 2) scales most favorably, continuing to improve where larger alphabets plateau. MLPLayerNormAttention 2 1 0 1 2 3 Perplexity (%) OpenWebText MLPLayerNormAttention CodeParrot Re-initialized Component Which weights matter most for transfer? Figure 5. Attention weights are most crucial for positive trans- fer. We report the change in validation perplexity when selectively re-initializing model components after NCA pre-pre-training, rela- tive to full transfer. Higher means the component is more important for transfer. Re-initializing attention causes the largest degrada- tion across both OpenWebText and CodeParrot, while MLP and LayerNorm effects are domain-dependent. As seen in Figure 9 in Appendix A, when increasingn, the resulting NCA data naturally becomes more complex. This result suggests that although larger rule spaces can express more complex dynamics, better guidance is necessary to sample a diverse set of NCAs that optimally transfer to language. Thus, constraining the space tok=2may para- doxically help by concentrating samples on dynamics with more consistent, transferable structure. Together, these results indicate that transfer is not simply “more NCA data is better.” The complexity of synthetic data, both gzip and alphabet size, must be calibrated. This of- fers a lever unavailable in natural language pre-training: the ability to tune the training distribution to match the compu- tational character of target domains. We further discuss the implications for domain-targeted pre-training in Section 6. 6. Discussion Why should we expect transfer?NCA data are substan- tially different from natural language and generated by de- terministic processes, prompting the question of why one should expect transfer at all? We argue that NCAs may pro- vide a purer training signal for in-context rule inference. In natural language, models may rely on semantic “shortcuts” or co-occurrence priors (Abbas et al., 2023; Geirhos et al., 2020). In contrast, every NCA sequence is generated by a hidden transition rule – parameterized by a random neural network. With no semantic knowledge to fall back on, every NCA token guides the model to in-context rule inference (Kirsch et al., 2022). This mirrors a core capability required for language mod- eling (Brown et al., 2020; Wei et al., 2022; Dong et al., 2024). Xie et al. (2022) show that training on natural text 7 Training Language Models via Neural Cellular Automata 20-30%30-40%40-50%50%+ 5 4 3 2 1 0 Perplexity (%) OpenWebText 20-30%30-40%40-50%50%+ CodeParrot Gzip Complexity Band How does NCA complexity affect transfer? Figure 6.Optimal NCA complexity is domain-dependent. We report the validation perplexity change of models trained with dif- ferent NCA complexities from the scratch model. OpenWebText benefits from higher-complexity data (50%+), while CodeParrot peaks at intermediate complexity (30–40%). This suggests that matching synthetic data complexity to the target domain is neces- sary to maximize transfer. teaches models to perform implicit Bayesian inference over latent concepts: each sequence draws from a latent concept, and predicting the next token means conditioning on the inferred concept. The same mechanism appears in math and code as well (Garg et al., 2023; Cook et al., 2025). Prior work on formal languages and algorithmic tasks such as Dyck and string copying (Hu et al., 2025b; Wu et al., 2022; Shinnick et al., 2025b) also train for this kind of in-context inference. Unlike these tasks, NCAs encompass a broad, universal class of computable functions (Copeland, 2012), some of which realize Turing-complete systems (Rendell, 2002; Wolfram & Gad-el Hak, 2003). The breadth and scale of this distribution makes memorization infeasible, forcing models to learn a general mechanism for rule inference (Li et al., 2024) that applies across the function class. This framing is supported by our mechanistic finding from Section 5.3.1: attention layers, not the MLPs or LayerNorms carry the most transferable structure. (Olsson et al., 2022) showed that ICL ability emerges with the formation of in- duction heads – attention circuits that help copy information from previous tokens to future ones. Because NCA pre-pre- training exclusively rewards this behavior, it may induce earlier and more robust formation than language-only pre- training. The transferred attention weights are, in effect, the in-context learning circuits, which are later adapted for downstream tasks and domains. A secondary motivation for transfer is epiplexity (Finzi et al., 2026). Classical information theory suggests determin- istic transformations cannot increase information content (Polyanskiy & Wu, 2025), thus questioning whether LLMs can learn meaningful structure from NCAs. However, this view assumes a computationally unbounded observer. For computationally bounded observers, Finzi et al. (2026) show 200M300M400M500M600M700M800M Pre-pre-training Tokens -7.0% -6.0% -5.0% -4.0% -3.0% Perplexity (%) 2 Color NCA 10 Color NCA 15 Color NCA How does NCA pre-pre-training scale with universe size? Figure 7.NCA alphabet size changes scaling behavior. We re- port the perplexity change relative to the scratch model (higher is better) on OpenWebText across different alphabet sizesnand NCA pre-pre-training token budgets. NCA improves perplexity at all token budgets. The smaller alphabet scales better, and the improvements degrade with scale for larger alphabets. that deterministic processes can generate useful structural information–coined epiplexity–that models must internal- ize to learn useful representations of the data. Their key insight is that simple local rules, like CA, can produce emer- gent structures (e.g., gliders, collisions) that a finite-capacity model cannot brute-force simulate. Instead, the model must learn a representation that allows it to predict the simulation at a coarser-grained abstraction. Learning these represen- tations over a diverse and universal class of functions like NCA may help with learning representations of natural lan- guage as well. Why is 160M tokens of automata better than 1.6B tokens of text?Surprisingly, with a significantly lower token bud- get, pre-pre-training on NCA data improves language mod- eling more than pre-pre-training on natural language (C4), as shown in Figure 2. How can abstract dynamical systems’ data transfer better to language than language itself? Even at 1.6B tokens, natural language pre-pre-training re- mains in an early training regime. Compute-optimal scaling laws suggest (Hoffmann et al., 2022) that a 1.6B parameter model requires roughly 32B tokens. At this early stage, language models primarily acquire shallow, local patterns and only learn more complex structures later on (Evanson et al., 2023; Chen et al., 2023). With limited tokens, C4 pre-pre-training likely spends most of its capacity on these surface-level regularities rather than the long-range dependencies and in-context learning that transfer broadly. 8 Training Language Models via Neural Cellular Automata In contrast, we hypothesize that NCA sequences provide a purer training signal for in-context learning. Each sequence is generated by a single latent rule that the model must infer from context and then apply consistently. Once identified, next-token prediction becomes nearly deterministic. Moreover, NCA pre-pre-training introduces a form of diver- sity orthogonal to what additional language tokens would provide. Despite their scale, many natural language datasets exhibit substantial redundancy (Abbas et al., 2023) in lin- guistic patterns and topic coverage. Since each of our NCA sequences represents a unique function to model, this di- versity may be more efficient per token at building general- purpose representations. Beyond one-size-fits-all pre-trainingOur complexity ab- lations reveal a nuanced picture that the optimal distribu- tion for training varies by downstream domain. In Fig- ure 6, we observed that code benefits from lower-complexity NCA rules, while web text and math benefit from higher- complexity ones, suggesting these domains encode com- putations of measurably different character. This opens a new axis of control. Rather than treating training data as fixed, we can tune the structures of synthetic data to match the target domain. Unlike grammar-based synthetic tasks, where each formal grammar defines a task with fixed struc- tural complexity, NCAs provide a continuous and tunable spectrum of complexity within a single generator family. If researchers can craft distributions that embody the primi- tives a domain requires (e.g., rigid state-tracking for code (Li et al., 2025), richer long-range dependencies for genomic sequences (Wu et al., 2025)), they can instill these capabili- ties directly, without scaling to trillions of general-purpose tokens. The result could aid the development of specialized, small language models (Belcak et al., 2025) that are more efficient to train and deploy—trained not on more data, but on better-matched data. Limitations and open problems A key question is whether NCA data can serve not only as a pre-pre-training signal, but as a scalable substitute for natural language pre- training. For larger alphabet sizes (n = 10, 15), we observe a reverse U-shaped trend: downstream improvement is opti- mal up to an intermediate token budget but plateaus beyond it. This behavior nonetheless reinforces our central the- sis: effective synthetic pre-training depends critically on structural choices in the data generator, not merely on scale. This points to a key open problem for future work: devel- oping principled methods to guide synthetic generators to sample structures that match those of target domains. Our complexity results demonstrate that such matching matters, but gzip compressibility and alphabet size are only two lens on complexity. Complexity is multifaceted: a sequence can be compressible yet be rich in long-range dependencies, or vice versa. Characterizing which axes of complexity (e.g., size of NCA network, grid size, or epiplexity) matter for which domains and learning to sample synthetic data ac- cordingly could unlock fully synthetic pre-training at scale. NCA represents one point in the vast space of possible synthetic data generators. The key insight from our work is not that NCA specifically is optimal, but that structured synthetic data with appropriate complexity characteristics can provide meaningful pre-training signal even without any linguistic content. The question is no longer whether synthetic pre-training can work, but how to design synthetic data distributions that maximize what models learn. Author Contributions Dan Lee co-lead the project and contributed to all aspects of experiments and writing. Seungwook Han co-lead the project and contributed to all aspects of experiments and writing. Akarsh Kumar supported this project, contributed to the design of the experiments and significantly to the writing. Pulkit Agrawal advised the development of the project idea from inception and contributed significantly to the writing. Acknowledgment We want to express our gratitude to Zachary Schinnick, Phillip Isola, Yoon Kim, Ryan Bahlous-Boldi, Idan Shen- feld, Nitish Dashora, and members of the Improbable AI lab for the helpful discussion on the paper. We are grateful to MIT Supercloud and the Lincoln Laboratory Supercomput- ing Center for providing HPC resources. The research was supported in part by NSF CSGrad4US Fellowship, Google, and Amazon. Also, the research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF-23-1-0277. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office, Naval Research Office, Air Force, or the U.S. Government References Abbas, A., Tirumala, K., Simig, D., Ganguli, S., and Morcos, A. S. Semdedup: Data-efficient learning at web-scale through semantic deduplication, 2023. URL https://arxiv.org/abs/2303.09540. An, J., Huang, D., Lin, C., and Tai, M. Measuring gender and racial biases in large language models, 2024. URL https://arxiv.org/abs/2403.15281. 9 Training Language Models via Neural Cellular Automata Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Baradad, M., Chen, R., Wulff, J., Wang, T., Feris, R., Tor- ralba, A., and Isola, P. Procedural image programs for representation learning. Advances in Neural Information Processing Systems, 35:6450–6462, 2022. Baradad Jurjo, M., Wulff, J., Wang, T., Isola, P., and Tor- ralba, A. Learning to see by looking at noise. Advances in Neural Information Processing Systems, 34:2556–2569, 2021. Belcak, P., Heinrich, G., Diao, S., Fu, Y., Dong, X., Mu- ralidharan, S., Lin, Y. C., and Molchanov, P. Small lan- guage models are the future of agentic ai, 2025. URL https://arxiv.org/abs/2506.02153. Berkovich, J. A. and Buehler, M. J. Lifegpt: Topology- agnostic generative pretrained transformer model for cel- lular automata. npj Artificial Intelligence, 1(1):23, 2025. Berkovich, J. A., David, N. S., and Buehler, M. J. Automatagpt: Forecasting and ruleset inference for two-dimensional cellular automata.arXiv preprint arXiv:2506.17333, 2025. Bloem, P. Universal pre-training by iterated random com- putation, 2025. URLhttps://arxiv.org/abs/ 2506.20057. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020. URLhttps:// arxiv.org/abs/2005.14165. Chan, S. C. Y., Santoro, A., Lampinen, A. K., Wang, J. X., Singh, A., Richemond, P. H., McClelland, J., and Hill, F. Data distributional properties drive emergent in-context learning in transformers, 2022. URLhttps://arxiv. org/abs/2205.05055. Chen, A., Shwartz-Ziv, R., Cho, K., Leavitt, M. L., and Saphra, N. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms. arXiv preprint arXiv:2309.07311, 2023. Chen, M. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saun- ders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., Mc- Grew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code, 2021. URLhttps://arxiv.org/abs/ 2107.03374. Cheng, Z., Cao, M., Pishdad, L., Cao, Y., and Cheung, J. C. Can llms reason abstractly over math word prob- lems without cot? disentangling abstract formulation from arithmetic computation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 14317–14344, 2025. Chiang, C.-H. and yi Lee, H. Pre-training a language model without human language, 2020. URLhttps: //arxiv.org/abs/2012.11995. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Cook, J., Sapora, S., Ahmadian, A., Khan, A., Rocktaschel, T., Foerster, J., and Ruis, L. Programming by backprop: Llms acquire reusable algorithmic abstractions during code training. arXiv preprint arXiv:2506.18777, 2025. Copeland, B. J. The church-turing thesis. In Zalta, E. (ed.), Stanford Encyclopedia of Philosophy. Stanford Encyclo- pedia of Philosophy, 2012. Del ́ etang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Ge- newein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., et al. Language modeling is compression. arXiv preprint arXiv:2309.10668, 2023. Del ́ etang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Ge- newein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., Hutter, M., and Veness, J. Lan- guage modeling is compression, 2024. URLhttps: //arxiv.org/abs/2309.10668. Desai, A., Gulwani, S., Hingorani, V., Jain, N., Karkare, A., Marron, M., R, S., and Roy, S. Program synthesis using natural language, 2015. URLhttps://arxiv.org/ abs/1509.00413. 10 Training Language Models via Neural Cellular Automata Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., et al. A survey on in- context learning. In Proceedings of the 2024 conference on empirical methods in natural language processing, p. 1107–1128, 2024. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URLhttps: //arxiv.org/abs/2010.11929. Evanson, L., Lakretz, Y., and King, J.-R. Language acqui- sition: do children and language models follow similar learning stages? arXiv preprint arXiv:2306.03586, 2023. Field, D. J. Relations between the statistics of natural images and the response properties of cortical cells. Journal of the Optical Society of America A, 4(12):2379–2394, 1987. Finzi, M., Qiu, S., Jiang, Y., Izmailov, P., Kolter, J. Z., and Wilson, A. G. From entropy to epiplexity: Rethinking information for computationally bounded intelligence. arXiv preprint arXiv:2601.03220, 2026. Gardner, M. Mathematical games. Scientific american, 222 (6):132–140, 1970. Garg, S., Tsipras, D., Liang, P., and Valiant, G. What can transformers learn in-context? a case study of simple function classes, 2023. URLhttps://arxiv.org/ abs/2208.01066. Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Bren- del, W., Bethge, M., and Wichmann, F. A. Shortcut learn- ing in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. Gokaslan, A., Cohen, V., Pavlick, E., and Tellex, S. Open- webtext corpus.http://Skylion007.github. io/OpenWebTextCorpus, 2019. Han, S., Pari, J., Gershman, S. J., and Agrawal, P. General in- telligence requires reward-based pretraining, 2025a. URL https://arxiv.org/abs/2502.19402. Han, S., Song, J., Gore, J., and Agrawal, P. Emergence and effectiveness of task vectors in in-context learning: An encoder decoder perspective, 2025b. URLhttps: //arxiv.org/abs/2412.12276. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. Training compute-optimal large language models, 2022. URLhttps://arxiv.org/ abs/2203.15556. Hu, M. Y., Petty, J., Shi, C., Merrill, W., and Linzen, T. Between circuits and chomsky: Pre-pretraining on for- mal languages imparts linguistic biases. arXiv preprint arXiv:2502.19249, 2025a. Hu, M. Y., Petty, J., Shi, C., Merrill, W., and Linzen, T. Between circuits and chomsky: Pre-pretraining on formal languages imparts linguistic biases, 2025b. URLhttps: //arxiv.org/abs/2502.19249. Huh, M., Cheung, B., Wang, T., and Isola, P. The pla- tonic representation hypothesis, 2024. URLhttps: //arxiv.org/abs/2405.07987. Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. Jelassi, S., Mohri, C., Brandfonbrener, D., Gu, A., Vyas, N., Anand, N., Alvarez-Melis, D., Li, Y., Kakade, S. M., and Malach, E. Mixture of parrots: Experts improve memorization more than reasoning, 2025. URLhttps: //arxiv.org/abs/2410.19034. Jiang, H. A latent space theory for emergent abilities in large language models. arXiv preprint arXiv:2304.09960, 2023. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language mod- els, 2020. URLhttps://arxiv.org/abs/2001. 08361. Kataoka, H., Okayasu, K., Matsumoto, A., Yamagata, E., Yamada, R., Inoue, N., Nakamura, A., and Satoh, Y. Pre- training without natural images. In Proceedings of the Asian Conference on Computer Vision, 2020. Kirsch, L., Harrison, J., Sohl-Dickstein, J., and Metz, L. General-purpose in-context learning by meta-learning transformers. arXiv preprint arXiv:2212.04458, 2022. Kolmogorov, A. N. On tables of random numbers. Sankhy ̄ a: The Indian Journal of Statistics, Series A, p. 369–376, 1963. Li, J., Hou, Y., Sachan, M., and Cotterell, R. What do language models learn in context? the structured task hypothesis, 2024. URLhttps://arxiv.org/abs/ 2406.04216. 11 Training Language Models via Neural Cellular Automata Li, J., Guo, D., Yang, D., Xu, R., Wu, Y., and He, J. Codei/o: Condensing reasoning patterns via code input- output prediction, 2025. URLhttps://arxiv.org/ abs/2502.07316. Li, M. and Vit ́ anyi, P. An Introduction to Kolmogorov Com- plexity and Its Applications. Texts in Computer Science. Springer, Cham, 4 edition, 2019. ISBN 978-3-030-11297- 4. doi: 10.1007/978-3-030-11298-1. URLhttps: //doi.org/10.1007/978-3-030-11298-1. Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., and Lee, Y. T. Textbooks are all you need i: phi-1.5 technical report, 2023. URLhttps://arxiv.org/ abs/2309.05463. Lloyd, S. Measures of complexity: a nonexhaustive list. IEEE Control Systems Magazine, 21(4):7–8, 2001. Lu, K., Grover, A., Abbeel, P., and Mordatch, I. Frozen pretrained transformers as universal computation engines. In Proceedings of the AAAI conference on artificial intel- ligence, volume 36, p. 7628–7636, 2022. Lu, Z., Zhou, A., Ren, H., Wang, K., Shi, W., Pan, J., Zhan, M., and Li, H. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms, 2024. URLhttps://arxiv.org/ abs/2402.16352. Mirchandani, S., Xia, F., Florence, P., Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, D., and Zeng, A. Large language models as general pattern machines. arXiv preprint arXiv:2307.04721, 2023. Mitchell, M. Complexity: A guided tour. Oxford University Press, 2009. Mordvintsev, A., Randazzo, E., Niklasson, E., and Levin, M. Growing neural cellular automata. Distill, 2020. doi: 10.23915/distill.00023. https://distill.pub/2020/growing- ca. Mota, F., Aaronson, S., Antunes, L., and Souto, A. So- phistication as randomness deficiency. In Descriptional Complexity of Formal Systems: 15th International Work- shop, DCFS 2013, London, ON, Canada, July 22-25, 2013. Proceedings 15, p. 172–181. Springer, 2013. Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., and Awadallah, A. Orca: Progressive learning from complex explanation traces of gpt-4, 2023. URLhttps: //arxiv.org/abs/2306.02707. Nad ˇ as ̧, M., Dios ̧an, L., and Tomescu, A.Synthetic data generation using large language models: Advances in text and code. IEEE Access, 13:134615–134633, 2025. ISSN 2169-3536. doi: 10.1109/access.2025. 3589503. URLhttp://dx.doi.org/10.1109/ ACCESS.2025.3589503. Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022. Papadimitriou, I. and Jurafsky, D. Injecting structural hints: Using language models to study inductive biases in lan- guage learning. arXiv preprint arXiv:2304.13060, 2023. Paster, K., Santos, M. D., Azerbayev, Z., and Ba, J. Open- webmath: An open dataset of high-quality mathematical web text, 2023. Polyanskiy, Y. and Wu, Y. Information Theory: From Cod- ing to Learning. Cambridge University Press, 2025. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Ex- ploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. URLhttp://jmlr. org/papers/v21/20-074.html. Reid, M., Yamada, Y., and Gu, S. S.Can wikipedia help offline reinforcement learning?arXiv preprint arXiv:2201.12122, 2022. Rendell, P. Turing Universality of the Game of Life, p. 513–539.Springer London, London, 2002.ISBN 978-1-4471-0129-1. doi: 10.1007/978-1-4471-0129- 118. URLhttps://doi.org/10.1007/978-1- 4471-0129-1_18. Ribeiro, L. C., Bernardes, A. T., and Mello, H.On the fractal patterns of language structures.PLOS ONE, 18(5):1–20, 05 2023.doi: 10.1371/journal. pone.0285630. URLhttps://doi.org/10.1371/ journal.pone.0285630. Ruis, L., Mozes, M., Bae, J., Kamalakara, S. R., Talupuru, D., Locatelli, A., Kirk, R., Rockt ̈ aschel, T., Grefenstette, E., and Bartolo, M. Procedural knowledge in pretraining drives reasoning in large language models. arXiv preprint arXiv:2411.12580, 2024. Saxton, D., Grefenstette, E., Hill, F., and Kohli, P. Analysing mathematical reasoning abilities of neural models, 2019. URL https://arxiv.org/abs/1904.01557. Sharma, A., Cz ́ egel, D., Lachmann, M., Kempes, C. P., Walker, S. I., and Cronin, L. Assembly theory explains and quantifies selection and evolution. Nature, 622(7982): 321–328, 2023. 12 Training Language Models via Neural Cellular Automata Shinnick, Z., Jiang, L., Saratchandran, H., Hengel, A. v. d., and Teney, D. Transformers pretrained on procedural data contain modular structures for algorithmic reasoning. arXiv preprint arXiv:2505.22308, 2025a. Shinnick, Z., Jiang, L., Saratchandran, H., Teney, D., and van den Hengel, A. Can you learn to see without images? procedural warm-up for vision transformers, 2025b. URL https://arxiv.org/abs/2511.13945. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on machine learning research, 2023. Tegmark, M. The mathematical universe. Foundations of physics, 38(2):101–150, 2008. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi ` ere, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation lan- guage models, 2023. URLhttps://arxiv.org/ abs/2302.13971. Tunstall, L., Von Werra, L., and Wolf, T. Natural language processing with transformers. ” O’Reilly Media, Inc.”, 2022. Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., and Hobbhahn, M. Will we run out of data? limits of llm scaling based on human-generated data. arXiv preprint arXiv:2211.04325, 2022. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions, 2023. URL https://arxiv.org/abs/2212.10560. Wei, J. and Zou, K.Eda: Easy data augmentation techniques for boosting performance on text classifica- tion tasks, 2019. URLhttps://arxiv.org/abs/ 1901.11196. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abili- ties of large language models, 2022. URLhttps: //arxiv.org/abs/2206.07682. Wigner, E. P. et al. The unreasonable effectiveness of math- ematics in the natural sciences. Mathematics and science, 13:1–14, 1990. Wolfram, S.Universality and complexity in cel- lular automata.Physica D: Nonlinear Phenom- ena, 10(1):1–35, 1984.ISSN 0167-2789.doi: https://doi.org/10.1016/0167-2789(84)90245-8. URLhttps://w.sciencedirect.com/ science/article/pii/0167278984902458. Wolfram, S. A class of models with the potential to represent fundamental physics. arXiv preprint arXiv:2004.08210, 2020. Wolfram, S. and Gad-el Hak, M. A new kind of science. Appl. Mech. Rev., 56(2):B18–B19, 2003. Wu, W., Li, Q., Li, M., Fu, K., Feng, F., Ye, J., Xiong, H., and Wang, Z. Generator: A long-context genera- tive genomic foundation model, 2025. URLhttps: //arxiv.org/abs/2502.07272. Wu, Y., Li, F., and Liang, P. Insights into pre-training via simpler synthetic tasks, 2022. URLhttps://arxiv. org/abs/2206.10139. Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An explanation of in-context learning as implicit bayesian inference, 2022. URLhttps://arxiv.org/abs/ 2111.02080. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. Wizardlm: Empowering large pre-trained language models to follow complex in- structions, 2025. URLhttps://arxiv.org/abs/ 2304.12244. Zhang, S., Patel, A., Rizvi, S. A., Liu, N., He, S., Karbasi, A., Zappala, E., and van Dijk, D. Intelligence at the edge of chaos. arXiv preprint arXiv:2410.02536, 2024. Zhao, R., Saphra, N., and Kakade, S. M. Distributional scaling laws for emergent capabilities. In NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning, 2024. Zipf, G. K. Human behavior and the principle of least effort. Addison-Wesley Press, 1949. Ziv, J. and Lempel, A. A universal algorithm for sequential data compression. IEEE Transactions on Information Theory, 23(3):337–343, 1977. doi: 10.1109/TIT.1977. 1055714. 13 Training Language Models via Neural Cellular Automata A. Analysis on Natural and Synthetic Data Distributions 10 0 10 1 10 2 10 3 10 4 Rank 10 6 10 5 10 4 10 3 10 2 10 1 Frequency (normalized) OpenWebText (gzip 70%) OpenWebMath (gzip 58%) CodeParrot (gzip 32%) NCA (gzip 50%+) Figure 8.NCA data exhibits a similar Zipfian or power-law structure to natural language. We compare the relative token frequency distribution for each of the natural language corpora and NCA data. Natural language from different domains has different average complexity as measured by gzip compressibility (see legend). In this section, we examine the distributions of natural-language and NCA-generated synthetic data with respect to two primary high-order heuristics: (1) token frequency distribution and (2) gzip compressibility. A.1. NCA data exhibits similar token distributions to natural language To compare the token frequency across different data distributions, we sample and tokenize random text sequences from the natural language datasets (OpenWebText, OpenWebMath, and CodeParrot) and NCA generated data (n = 15). Figure 8 shows the distribution of relative token frequencies. Data generated from the NCAs follows a heavy-tailed, Zipfian token distribution that is structured similarly to natural language. Another interesting observation is that natural language depending on the domain varies quite drastically from 32% in code and 60-70% in math and web text. A.2. Increasing the vocabulary size n leads to more complex generated trajectories Figure 9 compares the distribution of gzip complexity across trajectories generated by randomly sampled NCAs across different alphabet sizesn = 2, 10, 15. Asnincreases, the distribution skews towards less compressible, more complex data. This implies that with higher n, the universe of rules expands and naturally the dynamics become more complex. B. Detailed Pre-pre-training and Pre-training Setup Table 2 summarizes the hyperparameters used for both pre-pre-training on NCA data and subsequent pretraining on natural language datasets. We sweep various batch sizes (32 to 512), learning rates (1× 10 −3 to1× 10 −5 ), and weight decays (1× 10 −4 to1× 10 −6 ). To ensure reproducibility, we train our pipeline on 4 randomness seeds for each main pipeline (NCA Pre-pre-training, Scratch, and C4 Pre-pre-training) and at least 2 seeds for each ablation run. HyperparameterPre-pre-trainingPre-training Effective batch size16512 Sequence length1024 tokens1024 tokens Learning rate1× 10 −4 5× 10 −4 (Math/Text), 2× 10 −4 (Code) LR scheduleCosine w/ warmupCosine w/ warmup Warmup steps (% total)10%10% Weight decayNone1× 10 −4 Gradient clippingNone1.0 Table 2. Hyperparameters for pre-pre-training and pre-training experiments. 14 Training Language Models via Neural Cellular Automata 0%20%40%60% 0% 25% 50% 75% Relative Frequency = 15.1% 2-Color NCA 0%20%40%60% 0% 10% 20% 30% = 34.6% 10-Color NCA 0%20%40%60% 0% 10% 20% 30% = 41.9% 15-Color NCA Gzip Complexity Distribution Across NCA Universes Gzip Complexity Figure 9.Different NCA alphabet sizes,n = 2, 10, 15naturally yield different complexity distributions of the data. Increasingn inherently increases the complexity of the data. C. Detailed Fine-Tuning Setup For GSM8K and HumanEval, we evaluate on all tasks provided by the benchmarks. For BigBench-Lite, given the quantity and imbalance of samples and tasks, we randomly sample at most 300 tasks for each major category of english language problem where there are at least 100 examples available for training. For GSM8K and BigBench-Lite, we fine-tune the OpenWebMath and OpenWebText pre-trained models on the respective training sets. For GSM8K, we train for 10 epochs using a learning rate of 1e-5 to enable the models to follow the question answering format for evaluation. For GSM8k, we also fine-tune on the Chain-of-Thought reasoning trace provided by the dataset. For BigBench-Lite, we train for a single epoch at a learning rate of 5e-6 to enable models to follow the answer format. Across both we sweep hyperparameters including learning and choose the best performing models for comparison for each baseline and NCA pre-trained model. For HumanEval, we do not fine-tune the models since it is a code completion task. For reproducibility, we train 4 seeds for each model and baseline and report the averages across runs. We evaluate the models’ performances across different Pass@kwithkvarying from 1, 8, 16, and 32. For Big-Bench, because of the multiple-choice nature of some tasks, we opt to demonstrate up to 4 passes. We use the unbiased estimator from Chen (2021), computing the metric from 64 total decodings per run. For evaluation, we sampled with a temperature of0.4and top-p of0.95across GSM8k, HumanEval, and BigBench. We evaluated with higher temperatures and use0.4 temperature as higher temperatures led to overall worse and highly variable performance. We use 4 training pipeline seeds per task for each baseline and NCA pre-pretrained models and 5 decoding seeds per pipeline seed. D. NCA Pre-Pre-Training is more token efficient than natural language In this section, we compare convergence speed across pipelines by computing a different the token efficiency metric used in Hu et al. (2025b). Token efficiency gain is defined as:Token Efficiency Gain = 1− T NCA PPT +T NCA PT T base PPT +T base PT . WhereT PPT represents the number of pre-pre-training tokens andT PT represents the number of pre-training tokens to achieve the scratch model’s final loss. Note that T base PPT = 0 for the no pre-pre-training baseline. On average, NCA pre-pre-trained models exhibit token efficiency gains of 31% on OpenWebText, 27% on OpenWebMath, and 49% on CodeParrot to reach equivalent performance to the scratch baseline. 15