Paper deep dive
Looped GPT-BERT: Trading Parameters for Computation in Small Language Modeling
Tingshuo Fan, Hongtao Mu, Tianyu Zhou, Hansen Liu, Tao Ji
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/10/2026, 3:11:50 AM
Summary
This paper introduces Looped GPT-BERT, a small language model architecture that combines GPT-BERT's hybrid masked next-token prediction (MNTP) and causal language modeling (CLM) objectives with depth-wise parameter sharing. By using only four physical Transformer layers and applying them recurrently (up to 12 times), the model achieves 12.18M parameters. Evaluated in the BabyLM 2026 Strict-small setting on a 7.48M-word corpus, the 4x12 looped model achieves an Overall Average of 35.42 and NLP Average of 48.48, demonstrating comparable performance to larger baselines like GPT-2 and GPT-BERT with significantly fewer parameters. The study highlights that while recurrent computation improves training stability and syntactic performance (BLiMP), it may limit representational space for tasks requiring entity tracking.
Entities (11)
Relation Signals (8)
Looped GPT-BERT → hasparametercount → 12.18M
confidence 98% · Our final 4x12 model... contains 12.18M parameters.
Looped GPT-BERT → achievesscoreon → BabyLM 2026
confidence 97% · The BabyLM 2026 leaderboard reports an Overall Average of 35.42 and an NLP Average of 48.48.
Looped GPT-BERT → uses → depth-wise parameter sharing
confidence 95% · combining GPT-BERT's masked next-token and causal language-modeling objectives with depth-wise parameter sharing.
Looped GPT-BERT → usesarchitecture → 4 physical layers
confidence 95% · Our final 4x12 model uses four physical layers for twelve recurrent traversals
Looped GPT-BERT → usesobjectiveratio → 1:3 (BERT:GPT)
confidence 93% · Our final setting uses 1:3, so one quarter of rows use MNTP and three quarters use CLM.
Looped GPT-BERT → trainson → new_data1
confidence 92% · We train on a preprocessed 7.48M-word English corpus... The resulting new_data1 corpus
Looped GPT-BERT → comparesfavorablyto → GPT-BERT
confidence 90% · Compared with public BabyLM 10M Strict-small... GPT-BERT baselines, it achieves comparable performance... with fewer parameters.
Looped GPT-BERT → comparesfavorablyto → GPT-2
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:When training data are limited, increasing parameter count is not the only way to improve language-model performance. A small parameter set, when repeatedly applied, can also deliver comparable performance. We study Looped GPT-BERT in the BabyLM 2026 Strict-small setting, combining GPT-BERT's masked next-token and causal language-modeling objectives with depth-wise parameter sharing. We train on a preprocessed 7.48M-word English corpus and compare objective ratios, non-looped and looped architectures, and loop counts. Our final $4\times12$ model uses four physical layers for twelve recurrent traversals and contains 12.18M parameters. The BabyLM 2026 leaderboard reports an Overall Average of 35.42 and an NLP Average of 48.48. Compared with public BabyLM 10M Strict-small GPT-2 and GPT-BERT baselines, it achieves comparable performance on selected linguistic and downstream metrics, including BLiMP and GLUE, with fewer parameters. The loop ablations show that additional recurrent computation can improve training and preserve strong performance on selected linguistic tasks, whereas poorer performance on other tasks may reveal an inherent limitation of the looped design: using only a few physical layers restricts the model's representational space.
Tags
Links
- Source: https://arxiv.org/abs/2609.09691v1
- Canonical: https://arxiv.org/abs/2609.09691v1
Trouble viewing inline? Open PDF directly →
Full Text
40,096 characters extracted from source content.
Expand or collapse full text
Looped GPT-BERT: Trading Parameters for Computation in Small Language Modeling Tingshuo Fan Hongtao Mu Tianyu Zhou Hansen Liu Tao Ji Affiliation: College of Foreign Languages and Literature, Fudan University Affiliation: tsfan24@m.fudan.edu.cn taoji@fudan.edu.cn Abstract When training data are limited, increasing parameter count is not the only way to improve language-model performance. A small parameter set, when repeatedly applied, can also deliver comparable performance. We study Looped GPT-BERT in the BabyLM 2026 Strict-small setting, combining GPT-BERT’s masked next-token and causal language-modeling objectives with depth-wise parameter sharing. We train on a preprocessed 7.48M-word English corpus and compare objective ratios, non-looped and looped architectures, and loop counts. Our final 4×124× 12 model uses four physical layers for twelve recurrent traversals and contains 12.18M parameters. The BabyLM 2026 leaderboard reports an Overall Average of 35.42 and an NLP Average of 48.48. Compared with public BabyLM 10M Strict-small GPT-2 and GPT-BERT baselines, it achieves comparable performance on selected linguistic and downstream metrics, including BLiMP and GLUE, with fewer parameters. The loop ablations show that additional recurrent computation can improve training and preserve strong performance on selected linguistic tasks, whereas poorer performance on other tasks may reveal an inherent limitation of the looped design: using only a few physical layers restricts the model’s representational space. envelope envelopefootnotetext: Corresponding author. 1 Introduction Language-model performance has commonly improved through jointly scaling model size, data, and training computation. Scaling-law studies describe predictable power-law relations between loss, parameter count, data, and compute (Kaplan et al., 2020), while compute-optimal analyses emphasize that parameter and token budgets should be allocated jointly (Hoffmann et al., 2022). These results largely concern much larger autoregressive models, where model capacity and per-example computation grow together. Under constrained data and parameter budgets, however, they need not: repeatedly applying the same parameters can create a deeper computational path without proportionally increasing the number of learned weights. We therefore ask whether recurrent computation induced by shared parameters can compensate for reduced physical depth and parameter capacity in low-resource language-model pretraining. The BabyLM Challenge provides a controlled setting for this question by restricting pretraining text to a developmentally plausible scale and evaluating linguistic, downstream, and human-like behavior (Warstadt et al., 2023). We work in the BabyLM 2026 Strict-small track on a cleaned 7.48M-word English corpus. Our starting point is GPT-BERT (Charpentier and Samuel, 2024), which trains masked next-token prediction (MNTP) and causal language modeling (CLM) in one Transformer parameter stack. It consequently supports both bidirectional masked prediction and autoregressive scoring or generation. This hybrid objective gives a further dimension to study: when the backbone is looped, does the relative amount of masked and causal supervision change the capabilities the architecture acquires? We introduce depth-wise recurrent parameter sharing on top of GPT-BERT. Universal Transformers repeatedly share transformations along depth (Dehghani et al., 2019), and ALBERT shows that cross-layer sharing can substantially reduce encoder parameters (Lan et al., 2020). Most directly, Looped Transformers inject a fixed input and the prior loop state into a shared backbone, increasing effective computational depth through iteration (Yang et al., 2024). Their strongest evidence, however, comes from in-context algorithm-learning tasks such as linear functions, decision trees, and neural-network regression. Natural-language pretraining must simultaneously acquire lexical knowledge, syntactic regularities, world knowledge, and discourse state; whether those computations can be supported by repeated shared layers remains an open empirical question. We address three questions: how the BERT:GPT training ratio changes the training objective and downstream abilities; whether effective depth formed by repeatedly applying a small number of physical layers can retain language ability with substantially fewer parameters; and whether training and downstream tasks continue to benefit as the number of loops increases. We summarize our contributions as follows: • We introduce Looped GPT-BERT, combining GPT-BERT’s hybrid MNTP/CLM objective with depth-wise recurrent parameter sharing. • We study how the BERT:GPT objective ratio affects training and downstream behavior in the looped architecture. • We compare a 12-layer non-looped model with four physical layers applied 1/3/6/12 times, showing how recurrent computation closes the gap from fewer learned layers and where additional loops yield diminishing returns. • We evaluate the final model in the BabyLM Challenge, where its 12.18M parameters achieve comparable performance to public Strict-small baselines on linguistic and downstream tasks. Cleaning operation Before After Style and grammar normalization Where are you at? Where are you? Disfluency removal a pound, or a hundred pounds today, is not the same A hundred pounds today is not worth the same amount later. Short-content filtering Right. Removed when it lacks independent semantic content. Case normalization LET’S GIVE IT UP FOR DANNY TANNER Let’s give it up for Danny Tanner. Structural-noise removal === Bullet the Blue Sky === Header removed; article text retained. Stage-direction removal I’l be right back. [leaves room.] I’l be right back. Speaker normalization *MOT: more what? *CHI: more tapioca. A: More what? B: More tapioca. Consecutive-turn merging B: Yeah. B: I’m in Texas. B: Yeah, I’m in Texas. Duplicate removal Repeated or near-identical sentences One normalized instance. Table 1: Representative corpus-cleaning operations. Examples are shortened for presentation. 2 Related Work Data-Efficient LM Pretraining BabyLM shifts attention from performance under web-scale training to learning efficiency under a controlled data budget. The first challenge showed that architecture, pretraining objective, preprocessing, and curriculum can all substantially affect small-model performance, with no single approach dominating every downstream task (Warstadt et al., 2023; Hu et al., 2024). Subsequent work further found that the effect of data composition depends on model scale: a genre mixture suitable for one model size need not be optimal for another (Yam and Paek, 2024). Other BabyLM studies explore variation sets, child-inspired data and vocabulary choices, explicit linguistic information, and self-distillation as complementary routes to sample-efficient pretraining (Haga et al., 2024; Ghanizadeh and Dousti, 2024; Edman et al., 2024; Nair et al., 2024). These approaches primarily improve sample efficiency through data selection, ordering, or objectives. Complementarily, we study parameter efficiency: keeping the training corpus and model width fixed, we replace part of the independently parameterized depth with repeated computation through shared physical layers. Hybrid Causal and Masked Modeling CLM predicts the next token from left context, whereas conventional MLM predicts selected tokens at their own positions from bidirectional context. The two objectives therefore assign different meanings to the same output position in a decoder-style model. GPT-BERT resolves this mismatch with MNTP (Charpentier and Samuel, 2024): when token xk+1x_k+1 is masked, supervision is shifted left so that the hidden state at position k predicts the original xk+1x_k+1. Consequently, causal and masked examples share a single next-token vocabulary projection and loss interface. GPT-mode rows retain their original tokens and use a lower-triangular attention mask; MNTP-mode rows corrupt selected input tokens, permit bidirectional attention, and compute loss only at the corresponding shifted next-token positions. Each training row is assigned one mode, rather than receiving both losses. AntLM similarly combines causal and masked language-modeling objectives in a BabyLM setting, alternating between them during training (Yu et al., 2024). Recurrent Depth and Parameter Sharing Universal Transformers update representations recurrently along depth while retaining parallel computation over sequence positions (Dehghani et al., 2019); ALBERT uses cross-layer parameter sharing primarily for parameter efficiency and shows that the sharing strategy affects downstream behavior (Lan et al., 2020). Looped Transformers use the recurrence zt+1=f(x+zt)z_t+1=f(x+z_t), where x is a fixed input embedding, ztz_t is the loop state, and f is a shared Transformer backbone (Yang et al., 2024). This makes effective computational depth grow with loop count while the parameter count is governed mainly by the number of physical layers. Kohli et al. (2026) provide a more direct natural-language reference: recurrent-depth Transformers can improve systematic generalization and depth extrapolation in implicit multi-hop reasoning, but excessive loops can also reduce prediction quality through overthinking. Neither line of work establishes that shared loops are uniformly suitable for lexical, syntactic, world-knowledge, and discourse-state learning. We therefore compare equal-application and equal-parameter configurations across linguistic and state-tracking tasks. 3 Method 3.1 Corpus Preparation and Tokenization We construct our training corpus by selecting and cleaning six source corpora: BNC Spoken, CHILDES, Project Gutenberg, OpenSubtitles, Simple English Wikipedia, and Switchboard. The resulting new_data1 corpus is produced by our rule-based normalization and filtering pipeline, which removes redundancy, short or low-quality content, case and structural noise, and normalizes dialogue formatting. The preprocessing code and source files are available in our preprocessing repository. Table 1 summarizes the operations and representative input–output examples. After cleaning, the six selected corpora yield 7,482,189 whitespace-delimited words. We use this complete post-cleaning corpus directly for pretraining rather than adding or padding data to reach the 10M-word upper bound. Their source-wise counts are reported in Table 2. We use two byte-pair encoding (BPE) tokenizers. The official 16k GPT-BERT tokenizer is used only for early non-looped ratio experiments, while the 8k tokenizer trained on the new_data1 corpus is used for the main and loop experiments. BPE provides a subword representation that can handle words beyond a fixed vocabulary (Sennrich et al., 2016). We choose the 8k vocabulary as a parameter-allocation trade-off for this small-data setting: with hidden size 384, its tied embedding table contains 8,192×384≈3.158,192× 384≈ 3.15M parameters. In our corpus, reducing the vocabulary from 16k to 8k increases the measured fertility from 1.4381 to 1.4787 tokens per whitespace-delimited word, a 2.82% increase. This indicates a modest change in word-level context coverage for the fixed 128-token input, while avoiding a larger embedding allocation. Source Words BNC Spoken 619,847 CHILDES 1,779,027 Gutenberg 2,502,709 OpenSubtitles 1,174,391 Simple Wikipedia 1,386,065 Switchboard 20,150 Total 7,482,189 Table 2: Word counts after preprocessing. 3.2 Hybrid GPT-BERT Objective Let x=(x1,…,xT)x=(x_1,…,x_T) be a token sequence and θ the shared model parameters. For GPT-mode rows, a causal attention mask is used and every non-padding next token is supervised: ℒGPT=−∑t=1T−1logpθ(xt+1∣x≤t).L_GPT=- _t=1^T-1 p_θ(x_t+1 x_≤ t). (1) For BERT-mode rows, predictable positions are sampled with a masking probability that decreases linearly from 0.3 to 0.15. Of the selected tokens, 80% are replaced by the mask token, 10% by a random token, and 10% are left unchanged. Bidirectional attention is permitted. The target is shifted to the preceding hidden-state position so that it remains aligned with the decoder-style next-token head. If x~ x denotes the corrupted input, the objective is ℒBERT=−∑t∈Mlogpθ(xt∣x~∖t).L_BERT=- _t∈ M p_θ(x_t x_ t). (2) Only shifted positions corresponding to selected masks contribute to this loss; all other labels are set to the ignore index. Thus, unmasked BERT positions never enter the loss. A batch-level ratio of rB:rGr_B:r_G assigns complete rows to the two modes, and the optimized objective averages valid supervised tokens with a z-loss regularizer weighted by 10−410^-4. Our final setting uses 1:31:3, so one quarter of rows use MNTP and three quarters use CLM. 3.3 Looped Transformer Backbone The non-looped baseline contains twelve independent Transformer layers. The looped backbone instead contains four independently parameterized physical layers, each with attention and feed-forward sublayers. Each loop adds the static token representation x to the preceding loop state ztz_t and applies the same four-layer stack FθF_θ: z0 z_0 =0, =0, (3) zt+1 z_t+1 =Fθ(x+zt),t=0,…,L−1. =F_θ(x+z_t), t=0,…,L-1. Thus, zt+1z_t+1 is the final representation produced by the current traversal and is simply carried into the next pass; it is not an additional network operation. Within physical layer i, attention and feed-forward transformations are applied sequentially: ut(i) u_t^(i) =DWAi,A(ht(i)+Attni(ht(i))), =DWA_i,A\! (h_t^(i)+Attn_i(h_t^(i)) ), (4) ht(i+1) h_t^(i+1) =DWAi,F(ut(i)+FFNi(ut(i))). =DWA_i,F\! (u_t^(i)+FFN_i(u_t^(i)) ). GPT-BERT’s dynamic weighted accumulation (DWA) is a learnable short-range residual mixer, not an additional recurrent state. We restrict DWA to representation fusion within each physical layer and reinitialize it on every loop; information across loops is carried exclusively by ztz_t. A 4×L4× L model therefore has four sets of Transformer parameters but performs 4L4L layer applications. The 4×34× 3, 4×64× 6, and 4×124× 12 variants perform 12, 24, and 48 layer applications, respectively, while keeping the same parameter count. This “effective depth” counts layer applications only: repeatedly applying shared parameters is not equivalent to a non-shared Transformer with the same number of layers. 3.4 Training Configuration Table 3 lists the main configuration. Beyond the structural hyperparameters shown there, optimization uses LAMB (You et al., 2020), a peak learning rate of 0.0141, a minimum learning rate of 0.00141, and cosine decay for 2,600 steps followed by a constant minimum rate. We select the peak learning rate and decay horizon through the controlled studies in Section 5. All pretraining and fine-tuning runs use random seed 42. Hyperparameter Value Physical layers 4 Loop iterations 12 Hidden size 384 FFN size 1,280 Attention heads 6 Vocabulary 8,192 Maximum sequence length 128 Batch size per GPU 32 Number of GPUs 8 Global batch (tokens) 32,768 Epochs 10 Optimizer LAMB Peak learning rate 0.0141 Decay steps 2,600 Minimum learning rate 0.00141 Gradient clipping 2.0 Parameters 12.18M Table 3: Main pretraining configuration. 4 Experiments 4.1 Evaluation Protocol We follow the BabyLM 2026 Strict-small evaluation pipeline. The final checkpoint receives the full evaluation: the zero-shot suite includes BLiMP and BLiMP Supplement minimal-pair syntax judgments (Warstadt et al., 2020), EWoK world knowledge (Ivanova et al., 2024), Entity Tracking, COMPS conceptual properties, GlobalPIQA commonsense reasoning, Reading, and Age of Acquisition (AoA). Fine-tuning covers BoolQ, MNLI, MRPC, MultiRC, QQP, RTE, and WSC, the GLUE/SuperGLUE-style natural-language-understanding subset used by the challenge (Wang et al., 2019). We report the leaderboard aggregates, where NLP Average pools NLP tasks and Human-like Average includes Reading, AoA, and related measures. We use the official causal backend for all reported GPT-BERT results. Beyond the final main revision, we retain intermediate checkpoints at one-million-word intervals from chck_1M through chck_10M, and at ten-million-word intervals from chck_20M through chck_70M. We run the official fast evaluation on each of these checkpoints. The ten-epoch training run contains approximately 74.8M word presentations, so chck_70M is the last complete ten-million-word milestone. The final model and all intermediate revisions are released in one public Hugging Face repository. 4.2 BabyLM 2026 Leaderboard Results Model Params Context Overall NLP BLiMP Sup. EWoK Entity COMPS GPIQA GLUE Official GPT-2 baseline 124M 512 37.38 48.99 65.23 57.25 50.63 19.10 51.81 35.09 63.80 Official GPT-BERT causal-focus 31M 128→ 512 28.46 35.64 71.66 63.21 49.49 – – – 65.13 Looped GPT-BERT (ours) 12.18M 128 35.42 48.48 71.19 55.06 49.10 15.78 51.01 34.67 62.55 Table 4: BabyLM 2026 Strict-small leaderboard comparison on aggregate and selected task-level metrics. “Context” denotes the maximum training sequence length, “Sup.” denotes BLiMP Supplement, “Entity” denotes Entity Tracking, “GPIQA” denotes GlobalPIQA, and “GLUE” denotes the official (Super)GLUE aggregate. Dashes indicate results not reported for the corresponding public entry. Aggregate scores for entries with unreported tasks are not directly comparable. Table 4 reports the final 4×124× 12 Looped GPT-BERT alongside two publicly released BabyLM Strict-small references: the official GPT-2 baseline and the GPT-BERT causal-focus baseline. The leaderboard snapshot was accessed on July 21, 2026.11 1 https://huggingface.co/spaces/BabyLM-community/BabyLM-Leaderboard-2026 With 12.18M parameters, the final model is substantially smaller than both public references. It reaches 71.19 on BLiMP, compared with 65.23 for GPT-2 and 71.66 for GPT-BERT causal-focus, while its GLUE aggregate is 62.55, compared with 63.80 and 65.13. This pattern illustrates the trade-off studied here: at the cost of additional computation, repeated application of shared layers can retain strong performance on selected tasks with a smaller independently parameterized model, with gains varying across tasks. 4.3 Effect of the BERT:GPT Ratio We examine the BERT:GPT ratio under two tokenizers. The official 16k tokenizer is used for early non-looped ratio exploration, whereas the 8k tokenizer trained on the new_data1 corpus is used for all main and loop experiments. The 15:115:1 configuration is BERT-heavy, 1:11:1 is balanced, and 1:31:3 is GPT-heavy. The two settings test whether the ratio trend is consistent; they are not used to compare tokenizer quality. Tables 5 and 6 report the principal 8k comparison, including training endpoints and available zero-shot results. Bert:Gpt Params Loss Acc. 1:1 29.9M 3.9925 30.03 1:3 29.9M 3.3625 36.47 Table 5: Non-looped training results across objective ratios with the custom 8k tokenizer. Bert:Gpt BLiMP Sup. COMPS Entity 1:1 63.77 55.06 50.87 38.66 1:3 70.54 57.19 51.41 22.87 Table 6: Non-looped zero-shot results across objective ratios with the custom 8k tokenizer. For context, Table 7 reports the corresponding early experiments with the official 16k tokenizer. Increasing the proportion of GPT rows lowers the final mixed training loss in both tokenizer settings, with 1:31:3 giving the strongest endpoint. Bert:Gpt Final loss Token accuracy 15:1 4.0039 33.42 1:1 3.9711 31.50 1:3 3.7925 32.37 Table 7: Non-looped training results across objective ratios with the official 16k tokenizer. The 1:31:3 ratio improves BLiMP, BLiMP Supplement, and COMPS under the 8k tokenizer, but reduces Entity Tracking from 38.66 to 22.87. This pattern suggests that objective mixing changes the distribution of learned capabilities rather than merely the aggregate loss. More CLM supervision repeatedly trains left-to-right next-token prediction and is therefore closely aligned with sequential generation, local dependencies, and causal syntactic scoring. More MNTP supervision requires reconstructing a target from both sides of its context and may better support integrating multiple positions and relations, which is useful for tracking entities and their states. We therefore use 1:31:3 as the main configuration for subsequent loop experiments targeting generative and syntactic ability. 4.4 Effect of Looped Depth We first examine training loss and token accuracy under the 1:31:3 objective. Figure 1 plots how these two measures change over training for the 12-layer non-looped model and for four-layer models with 11, 33, 66, or 1212 applications per forward pass. The 4×34× 3 and 12-layer non-looped models both execute twelve layer applications per forward pass, while the looped model uses only four independently parameterized layers. Figure 1: Training loss and token accuracy for the 8k-tokenizer, 1:31:3 objective across loop depths. The looped models use four physical layers; the non-looped model uses twelve independently parameterized layers. The 4×14× 1 control is visibly less stable at the beginning of training: it maintains a higher loss and lower token accuracy than the models with repeated layer applications. Increasing the loop count lowers the training loss and generally raises token accuracy, bringing the looped models closer to the 12-layer non-looped trajectory. The largest improvement occurs when moving from 4×14× 1 to 4×34× 3; the changes from 4×34× 3 to 4×64× 6 and from 4×64× 6 to 4×124× 12 are smaller. Thus, repeated computation recovers much of the training behavior of the non-looped model, while additional loops provide diminishing returns. Bert:Gpt = 1:3 Metric Non-loop 4×14× 1 4×34× 3 4×64× 6 4×124× 12 BLiMP 70.54 65.08 71.21 70.49 71.19 BLiMP Sup. 57.19 55.90 55.73 55.37 55.06 COMPS 51.41 51.21 51.40 51.69 51.01 Entity Tracking 22.87 17.38 16.47 28.66 15.78 Eye Tracking 8.61 8.62 7.85 8.62 8.25 Self-paced Reading 4.03 3.88 3.68 4.16 4.24 Bert:Gpt = 1:1 Metric Non-loop 4×14× 1 4×34× 3 4×64× 6 4×124× 12 BLiMP 63.77 62.28 64.45 69.38 64.21 BLiMP Sup. 55.06 57.03 54.96 56.11 56.07 COMPS 50.87 50.44 50.95 51.42 50.03 Entity Tracking 38.66 17.52 16.60 13.10 17.15 Eye Tracking 7.86 7.64 7.88 7.62 7.55 Self-paced Reading 3.80 3.84 3.57 3.56 3.63 Table 8: Zero-shot comparison across objective ratios and loop counts. We next turn to the zero-shot task results in Table 8. The table compares the 12-layer non-looped model with four physical layers applied 11, 33, 66, or 1212 times, using BERT:GPT ratios of 1:31:3 in the upper block and 1:11:1 in the lower block. Under 1:31:3, increasing the loop count from 4×14× 1 to 4×34× 3 substantially improves BLiMP, from 65.08 to 71.21, slightly exceeding the non-looped score of 70.54. Increasing the depth beyond 4×34× 3 does not produce a consistent additional gain: BLiMP is 70.49 for 4×64× 6 and 71.19 for 4×124× 12. Under 1:11:1, 4×64× 6 gives the highest BLiMP score (69.38), compared with 63.77 for the non-looped model and 62.28–64.45 for the other looped settings. Overall, the looped models preserve or improve several syntactic and reading-related scores despite using fewer independently parameterized layers, while the effect of recurrent depth varies across tasks. Unlike the syntactic measures above, Entity Tracking is more sensitive to parameter sharing. Overall, the looped variants are weaker than the corresponding non-looped model on this task, with one exception: under 1:31:3, 4×64× 6 reaches 28.66, compared with 22.87 for the non-looped model. Even in this setting, the score falls to 15.78 with 4×124× 12, so the additional computation from six to twelve applications does not preserve the 4×64× 6 improvement. Under 1:11:1, all looped variants remain below the non-looped score of 38.66, with scores between 13.10 and 17.52. One possible explanation is that entity tracking requires distinguishing and updating multiple entities, their states, and state changes (Kim and Schuster, 2023). Reusing four attention/FFN parameter sets rather than twelve independent sets may leave less representational space for the layer-specific transformations needed to maintain multiple entities and relations. This contrast suggests that recurrent depth can compensate for reduced parameterization on some linguistic tasks, but cannot uniformly replace the representational flexibility of independently parameterized layers. Considering both downstream performance and inference cost, 4×64× 6 is a practical intermediate configuration, but the table also shows that the preferred loop depth depends on the objective ratio and task. 4.5 Inference Cost We measure inference cost for the 12-layer non-looped model and the 4×34× 3, 4×64× 6, and 4×124× 12 looped models with the same batch size of 32, sequence length of 128, 20 warmup steps, and 100 timed forward passes on one RTX 3090. As shown in Table 9, 4×34× 3 is better than the 12-layer non-looped model on every measured systems metric: it has lower latency, higher token throughput, and lower peak memory use. Increasing the loop count from 4×34× 3 to 4×64× 6 and 4×124× 12 roughly doubles the latency at each step, while throughput decreases and memory use increases. Considering both downstream performance and inference speed, 4×64× 6 provides a practical compromise: it is slower than 4×34× 3 but performs better on some downstream tasks, while remaining substantially faster than 4×124× 12. Model Latency (ms) Tokens/s Memory (GiB) 12-layer non-loop 58 70,571 0.68 4×34× 3 55 74,128 0.51 4×64× 6 108 37,966 0.72 4×124× 12 212 19,295 1.15 Table 9: Inference speed measured with batch size 32, sequence length 128, 20 warmup steps, and 100 timed forward passes on one RTX 3090. 5 Optimization Ablations 5.1 Selecting the Peak Learning Rate We first select a peak learning rate that lowers the training objective quickly without producing clear instability. Before full training, we run four-epoch non-looped sweeps at 0.0075, 0.01, 0.0141, and 0.02 with all other settings fixed. As Table 10 shows, the first three settings decrease stably, whereas 0.02 degrades after approximately step 600 and produces gradient-norm spikes. Although 0.01 has the highest final token accuracy, 0.0141 achieves the lowest final loss (3.6847) and is selected as the peak learning rate for the main experiments. This sweep provides a preliminary learning-rate choice based on the non-looped model and does not fully optimize the rate for looped models. We examine the effect of decay speed on training stability separately below. Peak LR Final loss Final accuracy 0.0075 3.7442 32.66 0.0100 3.6931 33.07 0.0141 3.6847 32.95 0.0200 4.5864 22.84 Table 10: Four-epoch peak learning-rate sweep. 5.2 Learning-Rate Decay and Stability We select two candidate decay lengths from the training budget: 4,080 steps spans the full optimizer-step budget of the run, whereas 2,600 steps reaches the minimum earlier and remains there for the rest of training. With total training steps fixed, lr_schedule_steps determines how quickly cosine decay reaches its minimum: a smaller value decays earlier and remains at the minimum longer, while a larger value retains a higher learning rate later in training. We compare these decay rates in 4×64× 6 and 4×124× 12 models. The main 2,600-step schedule reaches 10% of the peak learning rate and then holds 0.00141; the slower 4,080-step schedule maintains a higher learning rate into the later stages. Because shared physical layers are repeatedly applied across loops, the effect of one parameter update can be propagated through multiple layer applications. A high early learning rate may therefore amplify update oscillations, while earlier decay may help shared parameters enter a useful region more smoothly. Figure 2: Training loss and token accuracy for the learning-rate decay ablation. Solid and dashed lines denote 2,600-step and 4,080-step decay, respectively; colors distinguish 4×64× 6 and 4×124× 12. Figure 2 shows that the 2,600-step schedule has its clearest advantage in early and middle training: loss declines faster and token accuracy rises sooner. Earlier decay, however, can reduce the opportunity to explore alternative parameter regions and leave the model near a suboptimal solution. The 4,080-step schedule gradually catches up later, consistent with its higher late-stage learning rate preserving additional exploration. Nevertheless, the 2,600-step curves are smoother overall and show no clear final-metric disadvantage. We therefore choose 2,600 steps as a practical compromise between stability and late-stage exploration under the current compute budget and single-seed setting, not as a schedule that is uniformly superior at every point in training. 5.3 Qualitative Generation Examples The causal interface also supports open-ended completion. Table 11 presents selected completions from the final 1:31:3, 4×124× 12 model using temperature 0.9, top-k 40, top-p 0.95, and seed 42. The examples illustrate locally coherent continuation and appropriate termination in short responses; they are qualitative examples rather than a controlled generation evaluation. Prompt Sampled completion I want to eat some dinner, all right? After the rain stopped, the children began to run for a walk. She put the toy in the drawer, and gave the doll a glass of milk. The bird flew over the mountains and in the morning the rain came in. Table 11: Selected sampled completions from the final 1:31:3, 4×124× 12 model. 6 Conclusion We combine GPT-BERT’s hybrid CLM/MNTP training with depth-wise recurrent parameter sharing to study a “fewer parameters, more computation” language-model design under BabyLM Strict-small constraints. Reusing four physical layers for twelve loops yields a 12.18M-parameter model whose training objective is close to that of a 29.9M non-looped model. The final model reaches 71.19 on BLiMP and 62.55 on GLUE, compared with 71.66 and 65.13 for the official GPT-BERT causal-focus reference, despite its smaller parameter budget. On the reported task-level metrics, looped models preserve or improve selected syntactic abilities, while Entity Tracking becomes weaker as a result of parameter sharing. The ablations further show that gains from additional loops saturate quickly and differ across tasks. Looped GPT-BERT is therefore an architecture with clear advantages and limitations: under constrained data and parameter budgets, it can support syntactic understanding and generation, but its objective ratio and recurrent depth require further adjustment. Limitations • Data and tokenizer. The main experiments use one cleaned corpus and one 8k tokenizer. We do not independently ablate raw data, rule-based cleaning, alternative cleaning procedures, and tokenization. • Training variance. Each model is trained with a single random seed, so sub-percentage-point differences may fall within training variance. • Compute matching. The 4×124× 12 model performs substantially more layer applications and computation than the 12-layer non-looped baseline. This is a parameter-efficiency study, not a FLOP-matched comparison. • Objective mixing. Changing the Bert:Gpt ratio also changes supervision density and attention visibility; mixed training losses are therefore not perfectly homogeneous across ratios. • Capability boundary. The final model is weak on AoA and Entity Tracking, indicating limited cognitive-similarity and state-tracking ability. • Scope. Our conclusions are limited to approximately 12M parameters, a 7.48M-word corpus, ten epochs, and a maximum sequence length of 128; they should not be directly extrapolated to larger models or longer contexts. Ethics Statement The model uses English text supplied by, or derived from, the BabyLM Challenge and introduces no newly collected personal data. The sources include books, subtitles, encyclopedic text, and dialogue, so they may retain social biases or inappropriate content from the original material; automated cleaning cannot guarantee their removal. Cleaning may also remove dialectal, conversational, or minority-language patterns and introduce stylistic preferences through normalization. This small model is intended for research and should not be deployed directly in high-stakes applications. Reproducibility Statement The Hugging Face repository contains the final model and sixteen intermediate checkpoints. The main revision stores the final model; chck_1M–chck_10M are one-million-word checkpoints, and chck_20M–chck_70M are ten-million-word checkpoints. It also includes configuration_gpt_bert.py, modeling_gpt_bert.py, and tokenizer files, allowing the model to be loaded with trust_remote_code=True.22 2 https://huggingface.co/Survivor613/BabyLM2026-Strict-Small-looped_GPT-BERT Training configurations and scripts are available in the accompanying code repository,33 3 https://github.com/Survivor613/BabyLM-2026 and the data-cleaning workflow is documented separately.44 4 https://github.com/JT-Ushio/babylm26-nlp-spring Acknowledgments The authors thank the reviewers for their helpful comments and suggestions, as well as the BabyLM organizers for maintaining the datasets, evaluation pipeline, and leaderboard. This work was partially funded by the National Natural Science Foundation of China (No. 62506079). References Charpentier and Samuel (2024) L. G. G. Charpentier and D. Samuel GPT or BERT: why not both?. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, Miami, FL, USA, p. 262–283. External Links: Link Cited by: §1, §2. Dehghani et al. (2019) M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser Universal transformers. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Edman et al. (2024) L. Edman, L. Bylinina, F. Ghorbanpour, and A. Fraser Are BabyLMs second language learners?. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, Miami, FL, USA, p. 166–173. External Links: Link Cited by: §2. Ghanizadeh and Dousti (2024) M. A. Ghanizadeh and M. J. Dousti Towards data-efficient language models: a child-inspired approach to language learning. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, Miami, FL, USA, p. 22–27. External Links: Link Cited by: §2. Haga et al. (2024) A. Haga, A. Fukatsu, M. Oba, A. Bisazza, and Y. Oseki BabyLM challenge: exploring the effect of variation sets on language model training efficiency. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, Miami, FL, USA, p. 252–261. External Links: Link Cited by: §2. Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, Vol. 35, p. 30016–30030. Cited by: §1. Hu et al. (2024) M. Y. Hu, A. Mueller, C. Ross, A. Williams, T. Linzen, C. Zhuang, L. Choshen, R. Cotterell, A. Warstadt, and E. G. Wilcox Findings of the second BabyLM challenge: sample-efficient pretraining on developmentally plausible corpora. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, Miami, FL, USA, p. 1–21. External Links: Link Cited by: §2. Ivanova et al. (2024) A. A. Ivanova, A. Sathe, B. Lipkin, U. Kumar, S. Radkani, T. H. Clark, C. Kauf, J. Hu, R. T. Pramod, G. Grand, et al. Elements of world knowledge (EWoK): a cognition-inspired framework for evaluating basic world knowledge in language models. arXiv preprint arXiv:2405.09605. External Links: Link Cited by: §4.1. Kaplan et al. (2020) J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. External Links: Link Cited by: §1. Kim and Schuster (2023) N. Kim and S. Schuster Entity tracking in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, p. 3835–3855. External Links: Document, Link Cited by: §4.4. Kohli et al. (2026) H. Kohli, S. Parthasarathy, H. Sun, and Y. Yao Loop, think, & generalize: implicit reasoning in recurrent-depth transformers. arXiv preprint arXiv:2604.07822. External Links: Document, Link Cited by: §2. Lan et al. (2020) Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut ALBERT: a lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2. Nair et al. (2024) A. Nair, A. Hancharova, M. Kumar, and A. Gharaee BabyLM challenge: experimenting with self-distillation and reverse-distillation for language model pre-training on constrained datasets. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, Miami, FL, USA, p. 28–36. External Links: Link Cited by: §2. Sennrich et al. (2016) R. Sennrich, B. Haddow, and A. Birch Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1715–1725. External Links: Document, Link Cited by: §3.1. Wang et al. (2019) A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman GLUE: a multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, External Links: Link Cited by: §4.1. Warstadt et al. (2023) A. Warstadt, A. Mueller, L. Choshen, E. Wilcox, C. Zhuang, J. Ciro, R. Mosquera, B. Paranjape, A. Williams, T. Linzen, and R. Cotterell Findings of the BabyLM challenge: sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, Singapore, p. 1–34. External Links: Link Cited by: §1, §2. Warstadt et al. (2020) A. Warstadt, A. Parrish, H. Liu, A. Mohananey, W. Peng, S. Wang, and S. R. Bowman BLiMP: the benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics 8, p. 377–392. External Links: Document, Link Cited by: §4.1. Yam and Paek (2024) H. M. Yam and N. Paek What should baby models read? exploring sample-efficient data composition on model performance. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, Miami, FL, USA, p. 284–291. External Links: Link Cited by: §2. Yang et al. (2024) L. Yang, K. Lee, R. Nowak, and D. Papailiopoulos Looped transformers are better at learning learning algorithms. In International Conference on Learning Representations, External Links: Link Cited by: §1, §2. You et al. (2020) Y. You, J. Li, J. Hseu, X. Song, J. Demmel, and C. Hsieh Large batch optimization for deep learning: training BERT in 76 minutes. In International Conference on Learning Representations, External Links: Link Cited by: §3.4. Yu et al. (2024) X. Yu, B. Guo, S. Luo, J. Wang, T. Ji, and Y. Wu AntLM: bridging causal and masked language models. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, Miami, FL, USA, p. 324–331. External Links: Link Cited by: §2.