Paper deep dive
Structural priors for data-efficient language learning
Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager, Lisa Beinborn
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/11/2026, 5:24:36 AM
Summary
This paper investigates 'structural transfer' as a method for data-efficient language learning by pre-training GPT-2 models on non-language symbolic data (music, probabilistic grammars, cellular automata, protein sequences) before multilingual language modeling. Results show that structural pre-training reduces next-token prediction loss and weight shifts compared to random initialization, indicating a favorable parameter space initialization. However, these gains do not consistently translate to improved performance on downstream linguistic benchmarks, and structural transfer is less efficient than simply adding more language data.
Entities (10)
Relation Signals (8)
GPT-2 → isusedin → BabyLM
confidence 95% · We first train GPT-2-style models... and then continue training on the multilingual BabyLM corpus
Structural Transfer → uses → Non-language Data
confidence 95% · We investigate structural transfer: First training models on non-language data to induce useful priors for natural language.
Structural Transfer → islessefficientthan → Additional Language Data
confidence 93% · transfer from non-language data is less efficient than additional language data.
Structural Transfer → reduces → Weight Shifts
confidence 92% · These gains coincide with smaller weight shifts during subsequent language training
Probabilistic Context-Free Grammars → yields → Lower Language-Modeling Loss
confidence 90% · Several symbolic data types - notably... probabilistic grammars... yield lower language-modeling loss than random initialization.
Cellular Automata → yields → Lower Language-Modeling Loss
confidence 90% · Several symbolic data types - notably... cellular automata - yield lower language-modeling loss than random initialization.
Music → yields → Lower Language-Modeling Loss
confidence 90% · Several symbolic data types - notably music... yield lower language-modeling loss than random initialization.
→ →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Efficient language learning requires methods to reduce the reliance on large data and computational resources. We investigate structural transfer: First training models on non-language data to induce useful priors for natural language. This approach is a form of weight initialization for multilingual language modeling. We evaluate transfer via next-token-prediction loss, weight shifts in the model, and downstream linguistic benchmarks. Several symbolic data types - notably music, probabilistic grammars, and cellular automata - yield lower language-modeling loss than random initialization. These gains coincide with smaller weight shifts during subsequent language training, suggesting that structural transfer positions models in a more favorable region of the parameter space. However, a lower loss does not translate consistently into better downstream linguistic performance, and transfer from non-language data is less efficient than additional language data. We conclude that non-language data can serve as a partial substitute for language data for the training objective of next-token prediction but does not reliably support broader linguistic generalization.
Tags
Links
- Source: https://arxiv.org/abs/2609.11505v1
- Canonical: https://arxiv.org/abs/2609.11505v1
Trouble viewing inline? Open PDF directly →
Full Text
76,713 characters extracted from source content.
Expand or collapse full text
Structural priors for data-efficient language learning Yana Veitsman* Jonas Mayer Martins* Jonathan Lautenschlager Lisa Beinborn Affiliation: University of Göttingen, Germany Email: firstname.lastname@uni-goettingen.de Abstract Efficient language learning requires methods to reduce the reliance on large data and computational resources. We investigate structural transfer: First training models on non-language data to induce useful priors for natural language. This approach is a form of weight initialization for multilingual language modeling. We evaluate transfer via next-token-prediction loss, weight shifts in the model, and downstream linguistic benchmarks. Several symbolic data types—notably music, probabilistic grammars, and cellular automata—yield lower language-modeling loss than random initialization. These gains coincide with smaller weight shifts during subsequent language training, suggesting that structural transfer positions models in a more favorable region of the parameter space. However, a lower loss does not translate consistently into better downstream linguistic performance, and transfer from non-language data is less efficient than additional language data. We conclude that non-language data can serve as a partial substitute for language data for the training objective of next-token prediction but does not reliably support broader linguistic generalization. † ∗ These authors contributed equally to this work. Models and data | Code repository 1 Introduction The performance of language models scales strongly with data and compute, yet both resources are fundamentally limited. Both modeling low-resource languages and cognitively plausible modeling, for instance, require learning from little input, making data efficiency an engineering problem and a scientific challenge Hoffmann et al. (2022). The BabyLM shared task addresses this constraint directly by evaluating systems under low-data conditions and encouraging cognitively inspired approaches Choshen et al. (2026). A natural source of inspiration for data-efficient learning is human language acquisition. Despite exposure to relatively limited linguistic input, infants detect and exploit statistical regularities in the language they perceive Saffran and Kirkham (2018). We examine whether it is possible to introduce structural priors into language models that facilitate more data-efficient language learning. We therefore aim to understand to what extent non-language data give rise to representations that facilitate subsequent language learning. Such priors can be instilled in language models through pre-pretraining: an additional stage in which models are trained on non-language data before natural-language pretraining Papadimitriou and Jurafsky (2020); Hu et al. (2025); Lee et al. (2026). Pre-pretraining constitutes a form of structural transfer, in which a model first learns to predict a structured symbolic signal and then reuses the resulting parameter state during language training. Previous work suggests that this approach can improve language-modeling convergence and downstream performance in domains such as coding, mathematics, and reasoning Lee et al. (2026); Shinnick et al. (2025). However, the relationship between the parameter changes induced by pre-pretraining and performance on the target objective remains underexplored. Data efficiency is particularly important in multilingual settings, where models must generalize to typologically and distributionally diverse languages with varying amounts of data. At the same time, multilingual training introduces challenges absent from monolingual settings, including cross-lingual interference from data mixtures Ye et al. (2025) and conflicting optimization signals Wang et al. (2023). The multilingual setting thus offers the opportunity to test not only whether gains from structural transfer in monolingual or task-specific settings persist, but also whether structural transfer is language-specific. Approach and contributions We investigate whether structural transfer can accelerate multilingual language learning. We first train GPT-2-style models on structural data—including probabilistic context-free grammars, cellular automata, piano music, and protein sequences—and then continue training on the multilingual BabyLM corpus Jumelet et al. (2026a). We evaluate transfer along three axes: 1) efficiency on the natural-language next-token prediction objective, 2) model-internal weight shifts, and 3) downstream performance under the BabyLM evaluation framework Choshen et al. (2026). This setup allows us to test whether certain types of structural data yield a weight initialization that benefits multilingual modeling. Figure 1: Schematic of the experimental setup. To investigate how training on structure transfers to learning language, we first train language models on one of four symbolic data types: 1) probabilistic context-free grammars; 2) cellular automata; 3) piano music; and 4) protein sequences. The example tree could refer to the sentence Alice paints the cat. The piece shown is the beginning of Reger (1900). The protein depicted is green fluorescent protein (GFP) of jellyfish adapted from the Uniprot entry Boutet et al. (2007). Afterwards, we train these models on a multilingual corpus comprising Chinese, Dutch, and English texts. 2 Related work Small-scale language models are motivated by both practical constraints, such as limited data and compute, and cognitively plausible learning scenarios. Work in this setting has shown that models can benefit from structural biases introduced during training Papadimitriou and Jurafsky (2020); Papadimitriou and Jurafsky (2023); Hu et al. (2025); Lee et al. (2026). We view such biases through the lens of initialization: prior exposure to structural data can move a model away from an uninformed random starting point and toward a parameter region that supports more efficient language learning. 2.1 Sample-efficient modeling Previous entries in the BabyLM challenge Choshen et al. (2026) have explored a range of mechanisms for improving sample efficiency, including learning from interaction Charpentier et al. (2025); Mayer Martins et al. (2025), curriculum learning Warstadt et al. (2023); Diehl Martinez et al. (2023), and multilingual learning contexts Jumelet et al. (2026a). These approaches share the goal of improving learning when additional data or compute cannot be scaled up. Multilingual modeling Studying how models generalize across diverse linguistic distributions and data scarcity is especially relevant in multilingual settings. An open question is whether performance disparities across languages reflect model or data biases, or instead arise from specific linguistic properties Mielke et al. (2019); Poelman et al. (2025); Shani et al. (2026). Multilingual training requires careful design decisions about data mixtures Ye et al. (2025) and optimization across languages Wang et al. (2023). One line of work suggests that transfer is most effective between languages with similar linguistic properties Pires et al. (2019); K et al. (2020); Snæbjarnarson et al. (2023); another suggests that similarity may matter less than matching the informational complexity of the training signal Lee et al. (2026). The BabyLM challenge offers a controlled setting for examining whether structural transfer can support learning of heterogeneous distributions. 2.2 Weight initialization and inductive biases In neural language models, inductive biases can be introduced not only through architecture, but also through the model parameter state. Classical initialization schemes use random weights with carefully chosen scaling coefficients to break symmetry and stabilize activations and gradients Rumelhart et al. (1986); Glorot and Bengio (2010); He et al. (2015). Although these schemes do not encode task-specific structure, they shape the starting conditions for optimization. Structural pretraining extends this idea: rather than relying only on random initialization, a model can first be trained on a controlled signal that may induce representations useful for later language learning Papadimitriou and Jurafsky (2023); Hu et al. (2025); Lee et al. (2026). 2.3 Structural transfer We understand structural transfer as an umbrella term, encompassing the closely related concepts of curriculum learning, transfer learning, meta-learning, and pre-pretraining. In meta-learning, prior training across related tasks supports rapid adaptation; in curriculum learning, examples are ordered so that simpler data prepares the model for harder cases; and in transfer learning, knowledge from source tasks or languages provides a useful starting point for zero-shot performance or target-task fine-tuning Wang et al. (2021); Lee et al. (2022); Soviany et al. (2022). Pre-pretraining on structural data Papadimitriou and Jurafsky (2020); Hu et al. (2025); Lee et al. (2026); Jiang et al. (2026) combines aspects of these approaches. It introduces an earlier training stage, often on structural non-language data to induce a useful initialization for subsequent language training. The relevant question is therefore not only whether structural data can be learned, but whether the induced parameter state is beneficial for natural language training. Empirical findings. Structural data can vary in entropy, distributional shape, compositionality, hierarchy, and other regularities, which may induce different biases. Previous studies yield mixed conclusions about their benefits. Training on abstract grammars such as PCFGs can give rise to functional hierarchical representations Jumelet and Zuidema (2023); Rohweder et al. (2026), but downstream tasks are not always comparable: Hu et al. (2025) report limited gains on BLIMP, while Lee et al. (2026) focus more on reasoning benchmarks. It remains an unanswered question which properties of the structural source interact with the target language data and how they influence the performance on the evaluation task. We therefore examine a range of structural data types with varying degrees of abstractness and local and global dependencies. In our setup, cellular automata provide local rule-based dynamics, synthetic grammars contribute hierarchical and compositional structure, music offers sequential structure with long-range dependencies, and protein biological complexity. 3 Data The experimental setup consists of training a language model in two stages. In stage I, we train the model from scratch on a structural data type. In stage I, we switch to training on the target corpus. 3.1 Structural data In stage I, we explore four structural data types: synthetic grammars, cellular automata, music, and protein sequences. As a baseline, we use unstructured sequences of random numbers. Synthetic grammars A probabilistic context-free grammar (PCFG) is a formal grammar in which production rules for non-terminal symbols are insensitive to the context Manning and Schütze (2000). Every production rule is applied with a certain probability, making these synthetic grammars suitable for randomly sampling a wide variety of synthetic data. Created PCFGs mimic natural language closely by synthesizing grammar rules from the Penn Treebank Marcus et al. (1993), see Appendix A for details. We test two synthetic datasets generated by the same grammar, differing only in the lexicalization: For PCFGZipf, tokens follow a Zipfian distribution; for PCFGuni, tokens are distributed uniformly. Cellular automata (CA) A cellular automaton is a row of cells, with each cell being in a discrete state. This row evolves over time under a local transition rule f defining a set of update patterns. At each step in time, every cell is updated based on itself and its two neighbors. Cellular automata are a canonical example of self-organized complexity emergent from simple, deterministic rules von Neumann and Burks (1966); Gardner (1970); Wolfram (1983). The training objective requires the model to infer these rules from diverse patterns, making cellular automata a compelling source for learning functional capabilities. Appendix B provides a visualization and technical details. To assess the impact of structural complexity, we generate two dataset variants: CA16 (medium-complexity dynamics with K=16K=16 states) and CA256 (noisy dynamics with K=256K=256 states), each generated from about 1,8201,820 rules with multiple initial conditions per rule. The language model receives these space-time trajectories as concatenated rows. In training, the language model must infer the transition rule based on several trajectories generated from different initial conditions to succeed in next-token prediction. Music Music is a highly structured non-language signal produced by humans. We use piano music from the Aria-MIDI dataset Bradshaw and Colton (2025). For training, the language model receives a transcription of note events, timing and pedal state as a flat sequence of integers, see Appendix C. Musical data is characterized by long-range temporal dependencies with repetition and variations and rich hierarchical structure in the form of melody, harmony, and rhythm without encoding the semantics of natural language Berezovsky (2019). Protein sequences Proteins lie at the heart of vital biological functions and their amino-acid sequences encode highly complex molecular structures. We choose Swiss-Prot as a list of proteins Boutet et al. (2007) and normalize the dataset to the 20 canonical amino acids, each a single token, by replacing any rare amino acids or unknowns with an additional symbol X. Random As a control condition, we generate sequences with uniformly distributed length between 1 and 2,0482,048 from an alphabet of uniformly independently sampled integers k∈0,…,512k∈\0,…,512\. 3.2 Language data We use the dataset provided for the multilingual track of the BabyLM challenge. The dataset comprises cognitively plausible multilingual data according to the criteria outlined in Jumelet et al. (2026a). The exact composition of the dataset is listed in Appendix D. For stage I training, we sample English, Dutch, and Mandarin Chinese in proportion of 1/3 for each language. Baselines Our primary baseline model is trained directly in stage I from random initialization, without any prior exposure to structural data. We additionally train a secondary baseline on English Wikipedia Wikimedia Foundation (2023) during stage I, in order to assess whether structural transfer provides benefits beyond those obtained from additional natural-language data. 4 Experimental setup We run experiments on 80/20 data splits with a GPT-2 architecture Radford et al. (2019). Early stopping (with patience of 3 epochs and a threshold of 0.01) is used to identify the optimal amount of structural and natural-language data, respectively. For hyperparameters and implementation details, refer to Appendix E. We evaluate the gains from structural transfer using the validation loss achieved by the trilingual model on a held-out portion of the natural-language corpus, together with performance on a combination of zero-shot and fine-tuning tasks from the BabyLM framework Choshen et al. (2026), see Appendix F. Tokenization We train a trilingual byte-pair-encoding (BPE) tokenizer with a fixed vocabulary of 16,89716,897 tokens using the same approach to data sampling as outlined in Section 3.2. The tokenizer also includes 512 non-overlapping tokens used exclusively in stage I. At the start of stage I, we reinitialize the embedding layer to prevent spurious lexical transfer from the structural datasets. Loss metrics We measure the performance of the model on the next-token prediction through the cross-entropy loss ℒ(t)L(t) over the training tokens t. The average loss change relative to the baseline is defined by the loss ratio ρ as the ratio of the areas below the loss curves up to N tokens of a condition ℒiL_i and the primary baseline ℒ0L_0 for training on natural language, ρi=∫tcutoffNℒi(t)t∫tcutoffNℒ0(t)t−1, _i= _t_cutoff^NL_i(t)\,dt _t_cutoff^NL_0(t)\,dt-1\,, (1) with a minimal cutoff at tcutoff=2Mt_cutoff=2\,M, to avoid inflating the metric due to large loss differences early in training. A complementary metric, as used by Hu et al. (2025), is the token efficiency τ, i.e., the ratio of tokens necessary to achieve the final baseline loss with structural transfer, τi=1N(Ni+argmin0≤n≤Nℒi(n)≤ℒ0(N)), _i= 1N (N_i+ _0≤ n≤ N\L_i(n) _0(N)\ ), (2) where NiN_i is the number of steps we train on structural data in stage I for condition i. Intuitively, this captures how many tokens are necessary to train on structural plus language data to beat the baseline. 5 Results We examine the effect of structural transfer from each condition along three dimensions: 1) next-token prediction loss on a multilingual corpus, 2) model-internal weight shifts during training, and 3) linguistic benchmarks. Together, these analyses allow us to assess to which extent structural transfer can aid language-model training. 5.1 Transfer on next-token prediction Figure 2: Cross-entropy loss ℒL by training step of language models after having trained on structural data compared with the primary baseline of not training on any structure. The lines show the median over five seeds, except few runs that triggered early stopping. Shaded regions delineate minimum and maximum seed at each token count. Note that PCFGuni, PCFGZipf, and Music overlap. Training on symbolic data can robustly improve the loss ℒL on next-token prediction of natural language. Figure 2 shows that every condition matches or outperforms the baseline of not training on any structure (dashed black) in terms of achieving a lower loss. The clearest gains come from Music, both synthetic grammars PCFGuni and PCFGZipf, and the cellular automata CA16. Note that the two PCFG loss curves overlap, which implies that the Zipfian distribution does not benefit the structural transfer more than a uniform distribution. Although CA256 yields an early advantage over the baseline, this condition converges to the baseline at 220 M tokens, suggesting that the noisier dynamics do not induce an equally useful initialization. Proteins and random lead to more modest but consistent improvements. Overall, these results indicate that next-token-prediction loss can be robustly improved with a variety of symbolic data. However, the Wikien baseline puts these gains into perspective, showing that training on more natural language data for just one of the three languages from the multilingual corpus outperforms any of these symbolic data by a wide margin. 5.2 Model-internal representations Figure 3: Weight shift δ¯ δ per layer in the self-attention matrices when training on (a) structure and subsequently (b) language. Median seed with shaded regions for minimum and maximum seed. To understand how structural transfer impacts the model, we examine how much the weights shift in the two stages of training on structure and language, respectively. The relative weight shift from a time t before to after the respective training stage for layer l, component c, and seed s is δl,c,s(t)=‖Wl,c,safter−Wl,c,sbefore‖F‖Wl,c,sbefore‖F,δ^(t)_l,c,s= \|W^after_l,c,s-W^before_l,c,s\|_F\|W^before_l,c,s\|_F\,, (3) where ∥⋅∥F\|·\|_F is the Frobenius norm. We plot the mean across components c of the self-attention mechanism in the transformer architecture (query, key, value, and output matrices) per layer, δ¯l,s(t)=14∑cδl,c,s(t). δ^(t)_l,s= 14 _cδ^(t)_l,c,s\,. (4) Shifting weights In Fig. 3 (a), showing stage I, we observe that proteins and random induce much smaller weight shifts than the other better-performing conditions of stage I, which indicates that a random initialization is close to a local optimum for predicting these signals. Panel (b) shows how much the weights shift during stage I (language modeling). We see that training on sequences of proteins and random numbers in stage I does not reduce the required weight shifts for modeling language compared to the baseline of no structural training. The weights of models trained on synthetic grammars, music, and CA16 change less than the baseline with an average weight shift of around δ¯=0.34 δ=0.34 across layers. Notably, CA256 is an exception. The weights change strongly in the structural-learning phase, but the resulting attention pattern is not beneficial for language modeling, that is, the required weight shift is as large as for the randomly initialized baseline. Relating weights and loss Figure 4: Relation of loss ratio on language data and weight shift, averaged over layers, for training on (a) structural data and (b) language data. Solid dots show the median, small transparent dots show each seed. PCFG and music perform best because the representations from learning on structure can be partially transferred to language, CA16 to a lesser extent, too. Dotted black line and black circle (b): baseline of training on language only. Dashed gray line in (b): ordinary-least-squares regression through condition-level medians; shaded band: 95%95\,\% confidence interval. In Fig. 4, we visualize the loss reduction relative to the weight shift. The respective weight shift correlates with loss improvement on language (panel (b) with Pearson correlation r=0.96r=0.96, p<0.01p<0.01) but less so on structural data (panel (a) with r=−0.60r=-0.60, p=0.11p=0.11). The smallest weight shift achieved in stage I is associated with the biggest loss improvement for the English Wikipedia baseline. These results indicate that the weights optimized for predicting structural data facilitate learning language by requiring a smaller weight shift. Moreover, the best conditions achieve a token efficiency of approximately 60%60\,\% of the original token budget, meaning they require only about 60%60\,\% of the baseline token budget to reach equivalent language-modeling performance. The loss ratio ρ measures how much the training on structural data improves the cross-entropy loss (lower is better) on average over training steps and this ratio is closely related to token efficiency, see Appendix G. 5.3 Linguistic benchmarks Figure 5: Average accuracy on linguistic zero-shot and fine-tuning benchmarks for English, Dutch, Chinese, and averaged over these three languages, see Table 8. The dashed line denotes the baseline of not training on structural data. There are no significant changes relative to the baseline for the total, except Wikien and CA16. Dots show the median and error bars indicate the minimum and maximum seed. We employ the framework of the BabyLM challenge, which combines zero-shot and fine-tuning evaluation in English, Dutch, and Chinese Choshen et al. (2026), covering testing suites for linguistic competence (BLiMP-style tasks), reading comprehension, knowledge probing, and others. For details on the exact tasks, refer to Table 8 in Appendix F. The results show that the observed loss reduction on the language modeling objective does not translate to better performance on linguistic benchmarks. Figure 5 shows the evaluation results for several zero-shot and fine-tuning tasks. The only stage-I condition that robustly improves the baseline results for English and Dutch is training on English texts. All other structural-transfer conditions do not significantly affect the average evaluation results positively nor negatively. Across languages, CA16 has a small positive effect. We note that some conditions improve or worsen the performance exclusively on fine-tuning or zero-shot. This inconsistency might indicate that the linguistic benchmarks cannot resolve the small differences induced by the structural data or that zero-shot and fine-tuning tasks are not entirely aligned, measuring different linguistic aspects. Even if some structural data tends to lead to an improvement, the magnitude of this effect appears small, on the order of one percentage point. We conclude that an improvement on the target objective (next-token prediction) does not generally entail an improvement on linguistic benchmarks. Detailed effects We fit a Bayesian hierarchical model to estimate the effect of each structural-transfer condition relative to the natural language baseline. Benchmark scores (averaged across seeds) are modeled as scorectl∼(μc+ut,σ)score_ctl ( _c+u_t,σ) for each language and as scorectl∼(μc+ut+ul,σ)score_ctl ( _c+u_t+u_l,σ) for the total, with a fixed effect μc _c per condition, a random effect utu_t per task, and a random effect per language ulu_l. We report the posterior mean difference Δc=μc−μbaseline _c= _c- _baseline relative to the effect of the primary baseline in percentage points with 95%95\,\% highest-density intervals (HDI) and posterior probability p(Δc>0)p( _c>0), estimated via NUTS sampling (Hoffman and Gelman, 2014; Abril-Pla et al., 2023) with four chains of 2,000 draws. Figure 6 shows posterior estimates for all conditions across languages combined. Training on Wikipedia in stage I has robust positive effects on both zero-shot (Δ=+1.2 =+1.2 p, HDI: [0.6,1.9][0.6,1.9], p=1.00p=1.00) and fine-tuning (Δ=+1.3 =+1.3 p, HDI: [−0.1,2.7][-0.1,2.7], p=0.97p=0.97) evaluations. The Random condition shows a small negative effect on zero-shot performance (Δ=−0.6 =-0.6 p, HDI: [−1.2,0.1][-1.2,0.1], p=0.04p=0.04). The CA16 has a small positive effect on both zero-shot and fine-tuning performance (Δ=0.4 =0.4 p, HDI: [−0.2,1.1][-0.2,1.1], p=0.88p=0.88 and Δ=0.7 =0.7 p, HDI: [−0.8,2.0][-0.8,2.0], p=0.83p=0.83, respectively). All structural data have effects close to zero with HDIs that include zero. Figure 6: Mean and 95%95\,\% highest-density intervals estimated by a Bayes model for linguistic benchmarks of zero-shot and fine-tuning across languages. Based on these results, we performed a hyperparameter search to train the submission model for the BabyLM challenge on a mixture of the most beneficial data types, see Appendix H for details. However, even the best model (in terms of validation loss) does not perform much better than the separately trained models. Syntactic Tasks We hypothesize that structural transfer is more beneficial for syntax-level tasks due to its hierarchical dependencies. To check this, we select tasks explicitly marked as “syntactic” from English BLiMP Warstadt et al. (2020), BLiMP-NL Suijkerbuijk et al. (2025) and ZhoBLiMP Liu et al. (2026). These tasks span syntactic features such as argument structure, ordering in wh-clauses, islands constraints, etc. However, both per-task and aggregated category results per data type do not improve over the baseline by more than two percentage points. 6 Discussion In summary, our results indicate that learning structural patterns enables faster performance gains on the next-token prediction task. However, these dynamics do not translate to better performance in linguistic benchmarks Chen et al. (2025). Rather, transfer depends on the task at hand: Both next-token prediction and linguistic benchmarks measure language competence yet via different metrics, so that optimizing for next-token prediction does not necessarily imply measurable transfer effects on linguistic tasks. Finding that none of the tested structural conditions match the efficiency of additional natural-language data, we conclude that the intricate correlational structures of language cannot easily be replaced by synthetic data. Given that the two PCFG variants perform equally, a Zipfian distribution alone does not appear to impact transfer significantly. This may be because a Zipfian distribution is a first-order approximation of the correlational structure, i.e., just unigrams, which are learned quickly when training on natural language. Although protein sequences encode highly complex biological information, this structure does not appear to be readily accessible to a transformer, at least not under our naive tokenization approach used for the next-token prediction task of one token per amino acid. Since even random numbers perform slightly better than a randomly initialized model without any structural training, this might indicate that even a very simple tweak to the weight initialization can improve training results. The structural data may initialize the weights to functionally useful representations leading to successful structural transfer. These representations might be interesting to explore in connection with the work on discovering capabilities emerging in attention heads during pretraining Olsson et al. (2022); Elhage et al. (2021); Crosbie and Shutova (2025); Aoyama et al. (2026), in which a learned matrix structure facilitates functions such as token copying or retrieval. In some respects, the well-performing data are language-like. For example, music has highly complex structure produced by humans, including recursions and a form of syntax. PCFGs explicitly model syntactic structure extracted from natural-language texts. In cellular automata, on the other hand, complexity emerges from simple rules. However, any structural data we train on lacks the semantic content of language. Thus, it would be interesting to further explore the degree to which semantics are helpful and necessary for transfer. 7 Conclusion We investigated whether structural transfer can improve sample-efficient multilingual language modeling. Our results show that training on structured symbolic data before natural language can improve next-token prediction learning efficiency. The weight-shift analysis supports this interpretation: Models that benefit most from structural transfer require smaller parameter changes during multilingual language training, indicating that prior structural exposure places models in a more favorable region of the parameter space. However, downstream gains are limited. Improvements on the language-modeling objective do not reliably translate into stronger performance on downstream linguistic benchmarks. Moreover, when additional natural language data is available, it remains more efficient than structural data, even when training on a mix of partially typologically distant languages such as English, Dutch, and Chinese. Although this may be useful when language data is limited, such as for low-resource languages or other data-constrained settings, structural transfer is not a direct substitute for relevant language input. More broadly, our findings suggest that structural data provides a direction for revisiting initialization strategies in language models. Carefully tailoring model weights before training may provide a useful head start, particularly in the context of limited compute and model capacity. Limitations Selected structures Although we aim for a wide range of structural data (synthetic grammars, music, proteins, CAs) that we expected a priori to possibly yield strong transfer to natural language, resembling language to varying degrees, we cannot account for all possible data types exhaustively. Model size We conduct our experiments on the GPT-2-small architecture. While this setup provides us with a reasonable overview of structural transfer, a variation of the model size could clarify the limits of the loss improvements on next-token prediction. Data-mixing effects We do not exhaustively explore the effects of data mixing during stage I. A wider variety of high-quality data in stage I would allow for a more detailed study of language-specific differences in structural transfer. Since the Wikipedia baseline is in English only, we cannot assess the benefits of structural transfer across different natural languages. Observing effects of structural transfer tailored to a specific language would be an interesting avenue for future research. Representations We evaluate transfer by analyzing the magnitude of the weight shift. A more detailed analysis through the lens of mechanistic interpretability could target specifically the representations that the model learns on the structural data and possibly reuses on natural language. Ethical considerations While we aim to advance the design of cognitively inspired models, we do not speculate here on how human language processing works. Explicit parallels between the processing capabilities of models and human cognition tend to be strained easily. Acknowledgments We thank the reviewers for their feedback. We thank Alexander Ecker and Edoardo Ponti for helpful discussions. This research is partially supported by the zukunft.niedersachsen program of the VolkswagenStiftung (L.B., Y.V.) and by a VENI grant (Vl.Veni.211C.039) from the Nederlandse Organisatie voor Wetenschappelijk Onderzoek (NWO) (L.B.). Author contributions Y.V.: Supervision, Conceptualization, Analysis, Software (training), Literature review, Writing—original draft (Abstract, Introduction, Related work, Conclusion, parts of Methods), Writing—review & editing. J.M.M.: Supervision, Conceptualization, Analysis, Software (data and analysis), Literature review, Visualization, Writing—original draft (Results and Discussion; parts of Introduction, Related work, and Methods; App. on CAs and music), Writing—review & editing. J.L..: Software (PCFG data), Writing—original draft (PCFGs). L.B.: Supervision, Conceptualization, Analysis, Writing—review & editing. References Abril-Pla et al. (2023) O. Abril-Pla, V. Andreani, C. Carroll, L. Dong, C. J. Fonnesbeck, M. Kochurov, R. Kumar, J. Lao, C. C. Luhmann, O. A. Martin, M. Osthege, R. Vieira, T. Wiecki, and R. Zinkov PyMC: a modern, and comprehensive probabilistic programming framework in python. PeerJ Computer Science 9, p. e1516. External Links: ISSN 2376-5992, Document, Link Cited by: §5.3. Adelani et al. (2024) D. I. Adelani, H. Liu, X. Shen, N. Vassilyev, J. O. Alabi, Y. Mao, H. Gao, and E. A. Lee SIB-200: a simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), St. Julian’s, Malta, p. 226–245. External Links: Document Cited by: Table 8. Aoyama et al. (2026) T. Aoyama, E. Wilcox, and N. Schneider Predicting the formation of induction heads. In First Workshop on CogInterp: Interpreting Cognition in Deep Learning Models, San Diego, CA, USA. External Links: Link Cited by: §6. Bandarkar et al. (2024) L. Bandarkar, D. Liang, B. Muller, M. Artetxe, S. N. Shukla, D. Husa, N. Goyal, A. Krishnan, L. Zettlemoyer, and M. Khabsa The belebele benchmark: A parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 749–775. External Links: Document, Link Cited by: Table 8. Berezovsky (2019) J. Berezovsky The structure of musical harmony as an ordered phase of sound: A statistical mechanics approach to music theory. Science Advances 5 (5), p. eaav8490. External Links: ISSN 2375-2548, Document, Link Cited by: §3.1. Bird and Loper (2004) S. Bird and E. Loper NLTK: The natural language toolkit. In Proceedings of the ACL 2004 on Interactive Poster and Demonstration Sessions, Barcelona, Spain, p. 31–es. External Links: Document, Link Cited by: Appendix A. Boutet et al. (2007) E. Boutet, D. Lieberherr, M. Tognolli, M. Schneider, and A. Bairoch UniProtKB/Swiss-Prot. In Plant Bioinformatics, D. Edwards (Ed.), Methods in Molecular Biology, Vol. 406, p. 89–112. External Links: Document, Link, ISBN 978-1-58829-653-5 978-1-59745-535-0 Cited by: Figure 1, §3.1. Bradshaw and Colton (2025) L. Bradshaw and S. Colton Aria-MIDI: A dataset of piano MIDI files for symbolic music modeling. In The Thirteenth International Conference on Learning Representations, Singapore. External Links: Link Cited by: Appendix C, §3.1. Chang et al. (2026) T. A. Chang et al. Global PIQA: Evaluating commonsense reasoning across 100+ languages and cultures. arXiv. External Links: 2510.24081, Document, Link Cited by: Table 8. Charpentier et al. (2025) L. Charpentier, L. Choshen, R. Cotterell, M. O. Gul, M. Hu, J. Jumelet, T. Linzen, J. Liu, A. Mueller, C. Ross, R. S. Shah, A. Warstadt, E. Wilcox, and A. Williams BabyLM turns 3: Call for papers for the 2025 BabyLM workshop. arXiv. External Links: 2502.10645, Document, Link Cited by: §2.1. Chen et al. (2025) Y. Chen, B. Huang, Y. Gao, Z. Wang, J. Yang, and H. Ji Scaling laws for predicting downstream performance in LLMs. Transactions on Machine Learning Research. External Links: ISSN 2835-8856, Link Cited by: §6. Choshen et al. (2026) L. Choshen, R. Cotterell, M. O. Gul, J. Jumelet, T. Linzen, A. Mueller, S. Salhan, R. S. Shah, A. Warstadt, and E. G. Wilcox BabyLM turns 4 and goes multilingual: Call for papers for the 2026 BabyLM workshop. arXiv. External Links: 2602.20092, Document, Link Cited by: Table 8, Appendix F, §1, §1, §2.1, §4, §5.3. Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv. External Links: 1803.05457, Document, Link Cited by: Table 8. Conneau et al. (2018) A. Conneau, R. Rinott, G. Lample, A. Williams, S. Bowman, H. Schwenk, and V. Stoyanov XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, p. 2475–2485. External Links: Document Cited by: Table 8. Crosbie and Shutova (2025) J. Crosbie and E. Shutova Induction heads as an essential mechanism for pattern matching in in-context learning. In Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, NM, USA, p. 5034–5096. External Links: Document, Link Cited by: §6. Diehl Martinez et al. (2023) R. Diehl Martinez, H. McGovern, Z. Goriely, C. Davis, A. Caines, P. Buttery, and L. Beinborn CLIMB – Curriculum learning for infant-inspired model building. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, Singapore, p. 84–99. External Links: Document, Link Cited by: §2.1. Elhage et al. (2021) N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, et al. A mathematical framework for transformer circuits. Technical report Technical Report 12, Anthropic. External Links: Link Cited by: §6. Gardner (1970) M. Gardner Mathematical games. Scientific American 223 (4), p. 120–123. External Links: ISSN 0036-8733, Document, Link Cited by: §3.1. Glorot and Bengio (2010) X. Glorot and Y. Bengio Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Vol. 9, Sardinia, Italy, p. 249–256. External Links: Link Cited by: §2.2. He et al. (2015) K. He, X. Zhang, S. Ren, and J. Sun Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, p. 1026–1034. External Links: Document, Link, ISBN 978-1-4673-8391-2 Cited by: §2.2. He et al. (2025) L. He, E. Nie, S. S. Dindar, A. Firoozi, V. Nguyen, C. Puffay, R. Shimizu, H. Ye, J. Brennan, H. Schmid, H. Schuetze, and N. Mesgarani XCOMPS: A multilingual benchmark of conceptual minimal pairs. In Proceedings of the 7th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, Vinenna. Austria, p. 75–81. External Links: Document, Link Cited by: Table 8. Hoffman and Gelman (2014) M. D. Hoffman and A. Gelman The No-U-Turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research 15 (1), p. 1593–1623. External Links: Link Cited by: §5.3. Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, New Orleans, LA, USA, p. 30016–30030. External Links: 2203.15556, Document Cited by: §1. Hu et al. (2026) H. Hu, Y. Zhang, H. Zhu, S. Song, R. Wang, R. Hu, L. He, L. Li, X. Ji, S. Wang, Z. Qian, and Y. Lin Evaluation Pipeline. Shared Task at NLPCC 2026. External Links: Link Cited by: Table 8. Hu et al. (2025) M. Y. Hu, J. Petty, C. Shi, W. Merrill, and T. Linzen Between circuits and Chomsky: Pre-pretraining on formal languages imparts linguistic biases. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 9691–9709. External Links: Document, Link Cited by: §1, §2.2, §2.3, §2.3, §2, §4. Jiang et al. (2026) L. Jiang, Z. Shinnick, A. van den Hengel, H. Saratchandran, and D. Teney Procedural pretraining: Warming up language models with abstract data. arXiv. External Links: 2601.21725, Document, Link Cited by: §2.3. Jumelet et al. (2026a) J. Jumelet, A. Fourtassi, A. Haga, B. Bunzeck, B. Shandilya, D. Galvan-Sosa, F. G. Haznitrama, F. Padovani, F. Meyer, H. Hu, J. Etxaniz, L. Prevot, L. He, M. Grandury, M. Marcheva, N. Foroutan, N. Theodoropoulos, P. Sadeghi, S. Song, S. Salhan, S. Zhou, Y. Paniv, Z. Zhang, A. Bisazza, A. Warstadt, and L. Choshen BabyBabelLM: A multilingual benchmark of developmentally plausible training data. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Rabat, Morocco, p. 3297–3329. External Links: Document, Link Cited by: Table 6, §1, §2.1, §3.2. Jumelet et al. (2026b) J. Jumelet, L. Weissweiler, J. Nivre, and A. Bisazza MultiBLiMP 1.0: A massively multilingual benchmark of linguistic minimal pairs. Transactions of the Association for Computational Linguistics 14, p. 193–216. External Links: ISSN 2307-387X, Document, Link Cited by: Table 8. Jumelet and Zuidema (2023) J. Jumelet and W. Zuidema Transparency at the source: Evaluating and interpreting language models with access to the true distribution. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, p. 4354–4369. External Links: Document, Link Cited by: §2.3. K et al. (2020) K. K, Z. Wang, S. Mayhew, and D. Roth Cross-lingual ability of multilingual BERT: An empirical study. In The Eighth International Conference on Learning Representations 2020, Online. External Links: Link Cited by: §2.1. Kuperman et al. (2025) V. Kuperman, S. Schroeder, C. Acartürk, N. Agrawal, D. M. Alexandre, L. S. Bolliger, J. Brasser, C. Campos-Rojas, D. Drieghe, D. F. Đurđević, L. V. Gadelha De Freitas, S. Goldina, et al. New data on text reading in English as a second language: The Wave 2 expansion of the Multilingual Eye-Movement Corpus (MECO). Studies in Second Language Acquisition 47 (2), p. 677–695. External Links: ISSN 0272-2631, 1470-1545, Document, Link Cited by: Table 8. Lee et al. (2026) D. Lee, S. Han, A. Kumar, and P. Agrawal Training language models via neural cellular automata. arXiv. External Links: 2603.10055, Document, Link Cited by: Appendix B, §1, §2.1, §2.2, §2.3, §2.3, §2. Lee et al. (2022) H. Lee, S. Li, and T. Vu Meta learning for natural language processing: A survey. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, WA, USA, p. 666–684. External Links: Document, Link Cited by: §2.3. Lin et al. (2022a) S. Lin, J. Hilton, and O. Evans TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, p. 3214–3252. External Links: Document Cited by: Table 8. Lin et al. (2022b) X. V. Lin, T. Mihaylov, M. Artetxe, T. Wang, S. Chen, D. Simig, M. Ott, N. Goyal, S. Bhosale, J. Du, R. Pasunuru, S. Shleifer, P. S. Koura, V. Chaudhary, B. O’Horo, J. Wang, L. Zettlemoyer, Z. Kozareva, M. Diab, V. Stoyanov, and X. Li Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, p. 9019–9052. External Links: Document, Link Cited by: Table 8. Liu et al. (2026) Y. Liu, Y. Shen, H. Zhu, L. Xu, Z. Qian, S. Song, K. Zhang, J. Tang, P. Zhang, B. Yang, R. Wang, and H. Hu A systematic assessment of language models with linguistic minimal pairs in Chinese. Transactions of the Association for Computational Linguistics 14, p. 755–771. External Links: ISSN 2307-387X, Document, Link Cited by: Table 8, §5.3. Manning and Schütze (2000) C. D. Manning and H. Schütze Probabilistic parsing. In Foundations of Statistical Natural Language Processing, p. 407–460. External Links: Link, LCCN 410.285 Cited by: §3.1. Marcus et al. (1993) M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz Building a large annotated corpus of English: The Penn Treebank:. Computational Linguistics 19 (2), p. 313–330. External Links: Link Cited by: Appendix A, §3.1. Mayer Martins et al. (2025) J. Mayer Martins, A. H. Bashir, M. R. Khalid, and L. Beinborn Once upon a time: Interactive learning for storytelling with small language models. In Proceedings of the First BabyLM Workshop, Suzhou, China, p. 454–468. External Links: Document, Link Cited by: §2.1. Mielke et al. (2019) S. J. Mielke, R. Cotterell, K. Gorman, B. Roark, and J. Eisner What kind of language is hard to language-model?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, p. 4975–4989. External Links: Document, Link Cited by: §2.1. Mordvintsev et al. (2020) A. Mordvintsev, E. Randazzo, E. Niklasson, and M. Levin Growing neural cellular automata. Distill 5 (2). External Links: ISSN 2476-0757, Document, Link Cited by: Appendix B. Nivre et al. (2020) J. Nivre, M. de Marneffe, F. Ginter, J. Hajič, C. D. Manning, S. Pyysalo, S. Schuster, F. Tyers, and D. Zeman Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis (Eds.), Marseille, France, p. 4034–4043. External Links: Link, ISBN 979-10-95546-34-4 Cited by: Table 8. Olsson et al. (2022) C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, S. Johnston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah In-context learning and induction heads. arXiv. External Links: 2209.11895, Document, Link Cited by: §6. Papadimitriou and Jurafsky (2020) I. Papadimitriou and D. Jurafsky Learning music helps you read: Using transfer to study linguistic structure in language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, p. 6829–6839. External Links: Document, Link Cited by: §1, §2.3, §2. Papadimitriou and Jurafsky (2023) I. Papadimitriou and D. Jurafsky Injecting structural hints: Using language models to study inductive biases in language learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, p. 8402–8413. External Links: Document, Link Cited by: §2.2, §2. Pires et al. (2019) T. Pires, E. Schlinger, and D. Garrette How multilingual is multilingual BERT?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, p. 4996–5001. External Links: Document, Link Cited by: §2.1. Poelman et al. (2025) W. Poelman, T. Bauwens, and M. de Lhoneux Confounding factors in relating model performance to morphology. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, p. 7273–7298. External Links: Document, Link Cited by: §2.1. Qi et al. (2023) J. Qi, R. Fernández, and A. Bisazza Cross-lingual consistency of factual knowledge in multilingual language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, p. 10650–10666. External Links: Document, Link Cited by: Table 8. Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever Language models are unsupervised multitask learners. OpenAI blog. External Links: Link Cited by: §4. Reger (1900) M. Reger Der Mond ist aufgegangen. In Sieben Geistliche Volkslieder, External Links: Link Cited by: Figure 10, Figure 1. Rohweder et al. (2026) J. Rohweder, S. Dutta, and I. Gurevych Hierarchical latent structures in data generation process unify mechanistic phenomena across scale. arXiv. External Links: 2603.06592, Document, Link Cited by: §2.3. Romanou et al. (2025) A. Romanou, N. Foroutan, A. Sotnikova, Z. Chen, S. H. Nelaturu, S. Singh, R. Maheshwary, M. Altomare, M. A. Haggag, I. Schlag, M. Fadaee, S. Hooker, and A. Bosselut INCLUDE: Evaluating multilingual language understanding with regional knowledge. In The Thirteenth International Conference on Learning Representations 2025, Singapore, p. 83291–83322. External Links: Link Cited by: Table 8. Rumelhart et al. (1986) D. E. Rumelhart, G. E. Hinton, and R. J. Williams Learning representations by back-propagating errors. Nature 323 (6088), p. 533–536. External Links: ISSN 0028-0836, 1476-4687, Document, Link Cited by: §2.2. Saffran and Kirkham (2018) J. R. Saffran and N. Z. Kirkham Infant statistical learning. Annual Review of Psychology 69 (1), p. 181–203. External Links: ISSN 0066-4308, 1545-2085, Document, Link Cited by: §1. Sakaguchi et al. (2021) K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi WinoGrande: An adversarial Winograd Schema Challenge at scale. Communications of the ACM 64 (9), p. 99–106. External Links: ISSN 0001-0782, 1557-7317, Document, Link Cited by: Table 8. Shani et al. (2026) C. Shani, Y. Reif, N. Roll, D. Jurafsky, and E. Shutova The roots of performance disparity in multilingual language models: Intrinsic modeling difficulty or design choices?. arXiv. External Links: 2601.07220, Document, Link Cited by: §2.1. Shinnick et al. (2025) Z. Shinnick, L. Jiang, and H. Saratchandran Transformers pretrained on procedural data contain modular structures for algorithmic reasoning. In Methods and Opportunities at Small Scale (MOSS), Vancouver, Canada. External Links: Link Cited by: §1. Siegelman et al. (2022) N. Siegelman, S. Schroeder, C. Acartürk, H. Ahn, S. Alexeeva, S. Amenta, R. Bertram, R. Bonandrini, M. Brysbaert, D. Chernova, S. M. Da Fonseca, N. Dirix, et al. Expanding horizons of cross-linguistic research on reading: The Multilingual Eye-movement Corpus (MECO). Behavior Research Methods 54 (6), p. 2843–2863. External Links: ISSN 1554-3528, Document, Link Cited by: Table 8. Siegelman et al. (2025) N. Siegelman, S. Schroeder, Y. B. Bao, C. Acartürk, N. Agrawal, L. S. Bolliger, J. Brasser, C. Campos-Rojas, D. Drieghe, D. Filipović Đurđević, S. Goldina, R. Ibáñez Orellana, et al. Wave 2 of the Multilingual Eye-Movement Corpus (MECO): New text reading data across languages. Scientific Data 12 (1), p. 1183. External Links: ISSN 2052-4463, Document, Link Cited by: Table 8. Snæbjarnarson et al. (2023) V. Snæbjarnarson, A. Simonsen, G. Glavaš, and I. Vulić Transfer to a low-resource language via close relatives: The case study on Faroese. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), Tórshavn, Faroe Islands. External Links: Link Cited by: §2.1. Soviany et al. (2022) P. Soviany, R. T. Ionescu, P. Rota, and N. Sebe Curriculum learning: A survey. International Journal of Computer Vision 130 (6), p. 1526–1565. External Links: ISSN 0920-5691, 1573-1405, Document, Link Cited by: §2.3. Spitznagel and Keuper (2026) M. Spitznagel and J. Keuper A new kind of network? Review and reference implementation of neural cellular automata. arXiv. External Links: 2604.24990, Document, Link Cited by: Appendix B. Suijkerbuijk et al. (2025) M. Suijkerbuijk, Z. Prins, M. D. H. Kloots, W. Zuidema, and S. L. Frank BLiMP-NL: A corpus of dutch minimal pairs and acceptability judgments for language model evaluation. Computational Linguistics 51 (4), p. 1267–1301. External Links: ISSN 0891-2017, 1530-9312, Document, Link Cited by: Table 8, §5.3. von Neumann and Burks (1966) J. von Neumann and A. W. Burks Theory of Self-Reproducing Automata. 1st edition, University of Illinois Press, Urbana, IL, USA. External Links: Link Cited by: §3.1. Wang et al. (2023) M. Wang, H. Adel, L. Lange, J. Strötgen, and H. Schuetze GradSim: Gradient-based language grouping for effective multilingual training. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, p. 4631–4646. External Links: Document, Link Cited by: §1, §2.1. Wang et al. (2021) X. Wang, Y. Chen, and W. Zhu A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (9), p. 4555–4576. External Links: ISSN 0162-8828, 2160-9292, 1939-3539, Document, Link Cited by: §2.3. Warstadt et al. (2023) A. Warstadt, A. Mueller, L. Choshen, E. Wilcox, C. Zhuang, J. Ciro, R. Mosquera, B. Paranjape, A. Williams, T. Linzen, and R. Cotterell Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, Singapore, p. 1–6. External Links: 2504.08165, Document, Link Cited by: §2.1. Warstadt et al. (2020) A. Warstadt, A. Parrish, H. Liu, A. Mohananey, W. Peng, S. Wang, and S. R. Bowman BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics 8, p. 377–392. External Links: Document, Link Cited by: Table 8, §5.3. Wikimedia Foundation (2023) Wikimedia Foundation English Wikipedia 2023-11-01. External Links: Link Cited by: §3.2. Williams et al. (2018) A. Williams, N. Nangia, and S. Bowman A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, LA, USA, p. 1112–1122. External Links: Document, Link Cited by: Table 8. Wolfram (1983) S. Wolfram Statistical mechanics of cellular automata. Reviews of Modern Physics 55 (3), p. 601–644. External Links: ISSN 0034-6861, Document, Link Cited by: §3.1. Ye et al. (2025) J. Ye, P. Liu, T. Sun, J. Zhan, Y. Zhou, and X. Qiu Data mixing laws: Optimizing data mixtures by predicting language modeling performance. In The Thirteenth International Conference on Learning Representations 2025, Singapore, p. 82263–82287. External Links: Link Cited by: §1, §2.1. Zellers et al. (2019) R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi HellaSwag: Can a machine really finish your sentence?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, p. 4791–4800. External Links: Document, Link Cited by: Table 8. Appendix A Synthetic grammars We induce a probabilistic context-free grammar from the UD-Treebank annotation. The grammar is delexicalized, i.e., the terminal symbols are word-class categories. We then create data by sampling parse trees from the grammar and instantiating word classes with vocabulary IDs, see Fig. 7. The two types PCFGuni and PCFGZipf differ only in the frequency with which the words are sampled in the last step. Table 1 summarizes shared settings; Table 2 details the vocabulary layout. Figure 7: From the treebank, which consists of sentences with part-of-speech tags, we derive the transition probabilities, e.g., a sentence S splits into a noun phrase NP and a verb phrase VP or into only a verb phrase. Similarly, a noun phrase can split into a determiner DT and a noun N or into a plural noun NNS. From the derived grammar, we can sample instances again, but the lexicalization is not semantic, represented by abstract integers instead. The model receives only the sequence integers, without part-of-speech tags. Parameter Value Corpus size 200 M tokens Grammar Penn Treebank PCFG Max. parse depth 16 Sampling (PCFGzipf) Zipf (α=1α=1) Sampling (PCFGuni) uniform Table 1: PCFG hyperparameters. English-based Grammar Induction and Normalization. We derive a PCFG from the Wall-Street-Journal portion of the English Penn Treebank that is distributed with NLTK Marcus et al. (1993); Bird and Loper (2004) by estimating relative frequencies. The grammar is entirely delexicalized during construction by removing word-level rules, resulting in 2,664 structural productions, 26 non-terminal symbols, and 22 terminal categories. Part-of-speech tags on the right-hand side are collapsed into coarse abstract categories, such as grouping various noun tags into a single noun category. Punctuation is mapped to dedicated terminal categories (e.g. period, comma), rather than being collapsed into a single punctuation class. Functional tags and trace indices are stripped from non-terminal names to merge similar Treebank labels, and productions containing empty nodes are excluded. After merging duplicate productions, the probabilities are normalized separately for each left-hand-side non-terminal. To guarantee that generation can terminate at a specified depth, some non-terminals require a fallback rule. For any non-terminal lacking a production that expands exclusively to terminal categories, we add a single fallback production with a raw probability of 10−610^-6. This fallback is included in the left-hand side group prior to normalization, allowing it to retain a small relative probability while slightly reducing the mass of the other rules. Tree generation and depth constraints. Parse trees are generated via stochastic top-down sampling from the induced PCFG, starting at the root symbol S. At each non-terminal node, a production is drawn with a probability proportional to its weight among the rules for that symbol, and expansion continues until every leaf is a terminal category. For example, S → NP VP NP → det noun VP → verb NP NP → noun ⇒ det noun verb noun To keep the trees finite, we enforce a maximum depth limit of 16, counting the root as depth 0. During expansion at depths 0 through 14, any available production for a given symbol may be used. When the sampler reaches depth 15, the active rule set is restricted strictly to terminal-only productions. This intervention truncates very deep structures and slightly shifts the generated distribution compared to the unbound grammar. Lexicon layout and token sampling The 512 integer IDs are shuffled using a fixed random seed and partitioned into disjoint slices for each syntactic category, see Table 2. Slot sizes are derived from Penn Treebank type counts (distinct word forms per category), not from token frequencies in running text. To account for the fact that only a small portion of the vocabulary serves as function words, we unevenly distribute the 512 vocabulary IDs into 74 slots (≈15%≈ 15\,\%) for closed-class terminals and 438 slots for open-class terminals. Once the tree is fully expanded, each terminal leaf is mapped to a concrete ID from the respective vocabulary slice. Under a Zipfian condition, tokens within a slice are sampled with weights proportional to their inverse rank using a global exponent of α=1α=1. Under the uniform condition, every token within the slice has an equal selection probability. Parameter Value Number of categories 22 (disjoint slices) Vocabulary size 512 opaque IDs Open / closed budget 438 / 74 (85 % / 15 %) Allocation within budgets PTB type counts Largest closed slice num (51) Within-slice sampling Zipf (α=1α=1) or uniform Table 2: Lexicon layout. Final Output and Corpus Statistics. Each generated tree yields one sequence of tokens by extracting the terminal IDs in left-to-right order. The final output consists of simple integer sequences ranging from 0 to 511. We stop the generation process at exactly 200,000,000200,000,000 tokens per corpus, possibly truncating the last sentence. This procedure results in 572,648572,648 sentences for each file and a mean sequence length of about 349. Appendix B Generating cellular automata To illustrate the mechanics of cellular automata, consider the transition rule in Fig. 8: The update pattern f(■■■)=□■□f( [rgb]0.9766,0.8516,0.3555 \, [rgb]0.6211,0.1172,0.3203 \, [rgb]0.6211,0.1172,0.3203 )= [rgb]0.9766,0.8516,0.3555 maps this three-cell neighborhood to the state ■ [rgb]0.9766,0.8516,0.3555 in the next time step. Figure 8: A cellular automaton consists of a row of cells, each in one of, e.g., K=2K=2 states ■,■\ [rgb]0.6211,0.1172,0.3203 , [rgb]0.9766,0.8516,0.3555 \. The next time step is determined by applying a set of update patterns that define the transition rule of the cellular automaton to that row. The rule space of classical cellular automata with K states consists of K3K^K^3 rules, which is astronomically large even for K=16K=16, and many of the rules produce either fixed points or uncorrelated noise. Rather than sampling this space uniformly, we parameterize transition rules through a small neural network fθf_θ. Each set of randomly sampled, frozen parameters θ defines a rule. This approach is known as neural cellular automata (NCA) Mordvintsev et al. (2020); Spitznagel and Keuper (2026). By defining a suitable probability distribution for sampling the network parameters θ, we avoid both entirely noisy and homogeneous dynamics, that are expected to be less beneficial for structure learning. Our setup of using NCA for structural transfer in language models is inspired by Lee et al. (2026) but uses a one-dimensional grid, a simplified architecture, and deterministic (argmax) updates for efficiency and interpretability. Figure 9: Schematic of the NCA architecture fθf_θ, which takes a three-cell neighborhood as input and outputs the next state for the middle cell. The middle cell in state ■ [rgb]0.3203,0.8438,0.3203 is updated based on itself and its neighbors ■ [rgb]0.9492,0.3828,0.3828 and ■ [rgb]0.9766,0.8516,0.3555 through the following steps: One-hot encoding of the state, projection E of this vector to a low-dimensional representation, then a convolution with a matrix W, applying a nonlinearity (GELU), unembedding E and taking the argmax over this vector to arrive at the next cell state ■→■ [rgb]0.3203,0.8438,0.3203 → [rgb]0.6914,0.5781,0.9492 . Every NCA still defines a classical CA rule. Given the fixed parameters θ, one can simply enumerate the output of the NCA fθf_θ for all the three-cell neighborhoods to generate a classical lookup table. The neural parameterization is thus purely used to steer the data generation, not changing the nature of the simulated CAs but only which rules are likely to be drawn. B.1 Technical implementation In our implementation, each rule is defined by a randomly sampled, fixed parameter set θ, see Table 3. These parameters define a function fθf_θ that is applied to every cell and its 33-cell neighborhood with periodic boundary conditions. This function is a neural network, which updates a cell i to the state ci(t) c_i^(t) at time t. Architecture The architecture for generating NCA is a simple sequence of three functional steps, which suffice to give rise to interesting dynamics by inducing a bottleneck, convolution, and nonlinearity on the input data. Figure 9 visualizes this architecture schematically. First, the cell states (a one-hot encoding of the K integers, i.e., the vocabulary) are projected into a low-dimensional representation with an embedding matrix E∈ℝd×KE ^d× K, such that j=Ecj(t)e_j=E1_c_j^(t) (5) for j∈i−1,i,i+1j∈\i-1,i,i+1\. Second, these three embedding vectors je_j are concatenated. A matrix W and bias b act as a convolution of the three cell states, followed by a GELU nonlinearity to arrive at a vector i=GELU(W[i−1‖i‖i+1]+)h_i=GELU(W[e_i-1\|e_i\|e_i+1]+b) (6) where ∥\| means concatenation of the vectors. Finally, we apply an unembedding matrix U to arrive at a distribution over K states and take the argmax to select an output state, cit+1=argmaxk∈K(Ui).c_i^t+1= _k∈ K(Uh_i)\,. (7) The rules are thus fully deterministic given the parameters and an initial condition. For an initial condition (0)c^(0), the update rule (t+1)=fθ((t))c^(t+1)=f_θ(c^(t)) defines a trajectory τ=((0),(1),…,(T−1))τ= (c^(0),c^(1),…c,c^(T-1) ) with T time steps. The data that the language model receives is a row-wise flattened version of the trajectories τ, separated by a token for the end of a sequence. Generation For each dataset, we sample 100,000100,000 simulated trajectories from about 1,8201,820 rules. Each rule instantiates a neural cellular automaton. The parameters θ are sampled from distributions scaled by a parameter s to elicit a variety of dynamical regimes, see Table 3. We generate two dataset variants, CA16 and CA256, differing in the number of states K, embedding dimension d, and hidden dimension h, see Table 4. We tuned the hyperparameters for generating CA16 manually to instill an initial bias favoring a medium range of entropy, which we hypothesize to be the most promising, cf. the spectrum of NCA visualized with four examples in Fig. 1. The variant CA256 contrasts this with a larger vocabulary and more noisy dynamics. We also randomly vary the grid size N, trajectory length T, and number of trajectories per rule for more variety in the dataset, to prevent the model from overfitting on a specific periodicity in the data, see Table 5. Each trajectory is initialized in a state (0)∈0,…,K−1Nc^(0)∈\0,…c,K-1\^N by drawing uniformly from four schemes: (i) uni, which draws each cell i.i.d. from the state set, 0,…,K−1U\0,…c,K-1\; (i) gradient, which interpolates linearly between two independently chosen states; (i) block, which creates regions of length ℓ∼1,⌊N/7⌋ \1, N/7 \, each with a single random state; and (iv) sparse, which picks a uniform background state with a random fraction ρ∼(0.01, 0.2)ρ (0.01,\,0.2) of cells replaced by random states. Parameter Shape Distribution s Weight scale scalar log(s) (s) ∼(log(0.1),log(3)) ( (0.1),\; (3)) E Embedding d×Kd× K (0,s2)N(0,\,s^2) W Conv. weights h×3dh× 3d (0,s2/(3d))N(0,\,s^2/(3d)) b Conv. bias h (0,(0.2s)2)N(0,\,(0.2s)^2) U Unembedding K×hK× h (0,s2/h)N(0,\,s^2/h) Table 3: Parameter initialization for each randomly sampled NCA rule θ. The weight scale s is shared across all parameters of a given rule and controls the dynamical regime of the resulting automaton. Hyperparameter CA16 CA256 K Number of states 16 256 d Embedding dimension 8 16 h Hidden dimension 8 6 Neighborhood size 3 3 Table 4: Architecture hyperparameters for the two variants of the NCA dataset. Statistic Value Number of rules (expected) ≈1,820≈1,820 N Grid size 10,100U\10,100\ T Trajectory length 10,100U\10,100\ Trajectories per rule 10,100U\10,100\ Total trajectories 100,000100,000 Table 5: Statistics for generating the NCA datasets. Grid size N, trajectory length T, and trajectories per rule are sampled independently per rule. Appendix C Music Figure 10: Piano pieces are encoded as MIDI format in the Aria-MIDI dataset. We extract the pitch, volume, pedal state, and shifts of the tape head and map each event to integers. This examples shows the beginning of Reger (1900). We use the deduplicated “Unique” subset of the Aria-MIDI dataset Bradshaw and Colton (2025), comprising 32,52232,522 piano pieces in MIDI format. We convert each piece into an integer sequence for the language model to train on. Event encoding From the MIDI data, we extract three event types: 1) on means a note begins (followed by two integers expressing pitch and velocity); 2) off means a note ends (followed by an integer for the pitch, respectively); (3) pedal_on and pedal_off mean the sustain pedal changes state. A shift token precedes events not simultaneous with the previous one, encoding elapsed time measured in bins of 10 milliseconds, effectively moving the tape head forward in time. Left and right hand are a single event stream. Integer mapping Each event type above is assigned a contiguous integer range for transcribing the events into a sequence of numbers. The four command tokens on, off, pedal_on, and pedal_off are encoded as 0 to 3, respectively. Pitch (MIDI range 0–127) maps to the range 4–131. Velocity (MIDI range 0–127) is quantized in 25 steps of size 5, mapping to 132–156. Time shifts are binned in steps of 10 milliseconds into 127 bins, mapping to 157–283. The full vocabulary thus contains 284284 tokens. Figure 10 illustrates the encoding on a short excerpt. Appendix D BabyLM natural-language data The multilingual training data from the BabyLM corpus is listed in Table 6. Language Domain Ratio (%) Chinese Conversational 7979 % Educational 99 % Written/Books 1212 % Dutch Conversational 33 % Educational 1717 % Written/Books 1616 % Subtitles 11 % Fallback/Other 6363 % English Conversational 3737 % Written/Books 4242 % Fallback/Other 2121 % Table 6: Within-language domain mix of the BabyBabelLM Chinese, Dutch, and English corpora Jumelet et al. (2026a). Total sizes: 138M138\,M, 110M110\,M, 99M99\,M tokens, respectively. Percentages sum to 100 % per language. Appendix E Training regime We keep hyperparameters fixed across different structural data types and training stages, listed in the Table 7. Our Python environment uses torch==2.12.0, transformers==4.47.0, datasets==3.0.0, flash_attn==2.83. All experiments were performed on a single NVIDIA A100 80GB GPU for a total of ≈250≈ 250 GPU hours. Hyperparameter Value Model Layers 12 Hidden size 768 Attention heads 12 Context length 512 Vocabulary size 16,897 Activation GELU Dropout 0.1 LayerNorm ϵε 1×10−51× 10^-5 Initializer range 0.02 Training Sequence length 512 Batch size 16 Optimizer AdamW Optimizer learning rate 1×10−41× 10^-4 Weight decay 0.05 Warmup ratio 0.01 Gradient clipping 1.0 Early stopping patience 3 Early stopping delta 0.01 Seed 11; 17; 42; 2,000; 3,407 Fine-tuning Learning rate 5×10−55× 10^-5 Weight decay 0.05 Batch size 16 Sequence length 128 Early stopping patience 3 Seed 12 Table 7: Training and optimization settings for the main experiments. Appendix F Evaluation Dataset Language Prediction Task Reference Zero-shot BLiMP EN Grammatical acceptability Warstadt et al. (2020) BLiMP-NL NL Grammatical acceptability Suijkerbuijk et al. (2025) ZhoBLiMP ZH Grammatical acceptability Liu et al. (2026) Hanzi ZH Grammatical acceptability Hu et al. (2026) MultiBLiMP EN, NL Grammatical acceptability Jumelet et al. (2026b) GlobalPIQA EN, NL, ZH Commonsense reasoning Chang and others (2026) MECO EN, NL, ZH Eye-tracking Siegelman et al. (2022); Siegelman et al. (2025); Kuperman et al. (2025) HellaSwag EN, NL, ZH Sentence completion Zellers et al. (2019) Winogrande EN, NL, ZH Coreference resolution Sakaguchi et al. (2021) XStoryCloze EN, NL, ZH Story completion Lin et al. (2022b) XCOMPS NL, ZH Property knowledge He et al. (2025) Fine-tuning ARC EN, NL, ZH Abstraction and reasoning Clark et al. (2018) Belebele EN, NL, ZH Reading comprehension Bandarkar et al. (2024) BMLama EN, NL, ZH Knowledge probing Qi et al. (2023) INCLUDE NL, ZH Knowledge probing Romanou et al. (2025) MNLI EN, NL, ZH Inference Williams et al. (2018) SIB-200 EN, NL, ZH Topic classification Adelani et al. (2024) TruthfulQA EN, NL, ZH Knowledge probing Lin et al. (2022a) POS EN, NL, ZH POS-tagging Nivre et al. (2020) XNLI EN, ZH Inference Conneau et al. (2018) Table 8: Overview of evaluation datasets used in the BabyLM 2026 multilingual track Choshen et al. (2026). EN = English, NL = Dutch, ZH = Chinese. All metrics, except MECO, accuracies. Note that MECO is therefore excluded from Fig. 5 and Fig. 6. We use the evaluation pipeline of the 2026 BabyLM Challenge Choshen et al. (2026). Table 8 lists the multilingual evaluation data for zero-shot and fine-tuning tasks. Appendix G Metric correlation The two evaluation metrics of loss ratio ρ and token efficiency τ are strongly related, Fig. 11, yet loss ratio is more robust to outliers due to early stopping. Token efficiency is highly sensitive to the chosen cutoff point in training steps. Integrating over the loss curve instead, allows for a more robust interpretation of the results. This robustness is due to the integration of the loss curve, which solves the problem of sensitivity to the cutoff point in training steps that token efficiency suffers from. Figure 11: Relation between the two evaluation metrics loss ratio ρ and token efficiency τ. Appendix H Challenge submission For the challenge submission, we train our model on an equal mix of the top three structural data types, namely, on Music, CA16, and PCFGZipf. We perform a hyperparameter search with early stopping for the learning rate, weight decay, and warm-up ratio. Next, we select the best configuration and continue parameter search during stage I. Each stage includes tuning for five seeds and ten epochs. The best model (in terms of validation loss), initialized with the seed 2000, is then evaluated on the zero-shot and fine-tuning tasks of the challenge, and results along with the checkpoints are available on HuggingFace. An overview of the searched hyperparameter space is given in the Table 9; all other parameters are kept constant, as listed in Table 7. Model tuning was performed on a single NVIDIA A100 80GB GPU for ≈2250≈ 2250 GPU hours, where each stage-I run was ≈4≈ 4 hours and each stage-I run was ≈15≈ 15 hours. The best model along with the results has been uploaded to HuggingFace and BabyLM leaderboard. Results are available in Table 10. Hyperparameter Value Optimizer learning rate 1×10−41× 10^-4, 3×10−43× 10^-4, 6×10−46× 10^-4, 1×10−21× 10^-2 Weight decay 0.01, 0.05, 0.1 Warmup ratio 0.01, 0.03 Seeds 11; 17; 42; 2,000; 3,407 Table 9: Hyperparameter search space for the challenge submission. Benchmark English Dutch Chinese MultiBLiMP 0.88 0.91 – BLiMP 0.72 0.78 0.76 WinoGrande 0.52 0.49 0.49 XStoryCloze 0.49 0.48 0.49 HellaSwag 0.27 0.26 0.27 XCOMPS – 0.53 0.53 Global PIQA (parallel) 0.27 0.23 0.18 Global PIQA (non-para.) 0.52 0.51 0.48 ARC 0.23 0.25 0.24 Belebele 0.30 0.23 0.27 BMLAMA 0.10 0.12 0.10 MNLI 0.58 0.59 0.57 SIB-200 0.33 0.26 0.31 TruthfulQA 0.25 0.26 0.29 XNLI 0.53 – 0.53 INCLUDE – 0.29 0.26 POS 0.89 0.91 0.81 Table 10: Evaluation results for the submission model trained on the mixed dataset in stage I and on a mix of English, Dutch, and Chinese in stage I.