Paper deep dive
Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers
Sourabh Kasliwal
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/8/2026, 6:40:54 AM
Summary
This paper investigates the mechanics of multi-step arithmetic reasoning in compact 'Tiny' Transformers (~10.6M non-embedding parameters). It identifies that sequence padding causes gradient starvation, collapsing accuracy, which is remediated by sequence packing. Linguistic pretraining is shown to be a critical prerequisite for learning the syntactic structure of scratchpads. Modern architectural primitives (RoPE, RMSNorm, SwiGLU) and Sparse Mixture of Experts (MoE) significantly improve additive reasoning over baseline GPT-2. The study demonstrates that algorithmic scratchpad formulation is crucial: a deterministic Digit-by-Digit Long Division scratchpad within a 4-stage Hierarchical Developmental Curriculum elevates single-digit division accuracy from 4.0% to 86.7%. However, multi-digit multiplication remains challenging due to the FOIL scratchpad's inability to handle simultaneous summation, and performance collapses on unseen 4-digit operands or with unbuffered training.
Entities (12)
Relation Signals (11)
Digit-by-Digit Long Division → increasesaccuracyof → Single-Digit Division
confidence 96% · dramatically elevates single-digit division from 4.0% to 86.7% accuracy
Sequence Packing → remediates → Gradient Starvation
confidence 95% · dataloader sequence padding creates an 83% gradient starvation artifact that collapses accuracy from 40% to 1%, remediated via continuous sequence packing
Digit-by-Digit Long Division → usedin → Hierarchical Developmental Curriculum
confidence 95% · Introducing a deterministic Digit-by-Digit Long Division scratchpad within a 4-stage Hierarchical Developmental Curriculum dramatically elevates single-digit division
Unbuffered Training → induces → Catastrophic forgetting
confidence 94% · unbuffered training induces catastrophic forgetting, collapsing division accuracy from 86.7% down to 0.00%.
FOIL → failson → Multi-Digit Multiplication
confidence 93% · our FOIL scratchpad failed because it forced a simultaneous summation of up to nine multi-digit terms in a single step
Linguistic Pretraining → isprerequisitefor → Arithmetic Reasoning
confidence 92% · linguistic pretraining is an essential prerequisite (<= 2.0% without it)
Sparse Mixture-of-Experts → improves → Additive Reasoning
confidence 90% · modern architectural primitives (RoPE, RMSNorm, SwiGLU) and Sparse Mixture of Experts (MoE) substantially improve additive reasoning over baseline GPT-2
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Autoregressive Large Language Models (LLMs) frequently struggle with deterministic multi-step algorithmic tasks such as multi-digit multiplication and long division. In this paper, we investigate the mechanics of multi-step arithmetic in compact "Tiny" Transformers (~10.6M non-embedding parameters, 49.3M total) trained on synthetic data across four basic operations (+, -, *, /) unrolled as step-by-step scratchpads. First, we establish the necessary training foundations: (1) dataloader sequence padding creates an 83% gradient starvation artifact that collapses accuracy from 40% to 1%, remediated via continuous sequence packing; (2) linguistic pretraining is an essential prerequisite (<= 2.0% without it); and (3) modern architectural primitives (RoPE, RMSNorm, SwiGLU) and Sparse Mixture of Experts (MoE) substantially improve additive reasoning over baseline GPT-2. Second, we demonstrate that algorithmic scratchpad formulation directly dictates success. Introducing a deterministic Digit-by-Digit Long Division scratchpad within a 4-stage Hierarchical Developmental Curriculum dramatically elevates single-digit division from 4.0% to 86.7% accuracy on a 4,000-problem held-out benchmark. In contrast, multi-digit multiplication remained challenging: detailed error analysis revealed that while the model correctly computed single-digit sub-products and place-value zeros, our FOIL scratchpad failed because it forced a simultaneous summation of up to nine multi-digit terms in a single step without pairwise intermediate accumulation. Finally, we identify two key boundaries: performance collapses to 0.00% on unseen 4-digit operands, and unbuffered training induces catastrophic forgetting, collapsing division accuracy from 86.7% down to 0.00%.
Tags
Links
- Source: https://arxiv.org/abs/2610.09003v1
- Canonical: https://arxiv.org/abs/2610.09003v1
Trouble viewing inline? Open PDF directly →
Full Text
44,105 characters extracted from source content.
Expand or collapse full text
Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers ∗ Sourabh Kasliwal sourabh.kasliwal@gmail.com Abstract Autoregressive Large Language Models (LLMs) frequently struggle with deterministic multi-step algorithmic tasks such as multi-digit multiplication and long division. In this paper, we investigate the mechanics of multi-step arithmetic in compact “Tiny” Transformers (∼10.6M non-embedding parameters, 49.3M total) trained on synthetic data across four basic operations (+, −, ×, /) unrolled as step-by-step scratchpads. First, we establish the necessary training foundations: (1) dataloader sequence padding creates an 83% gradient starvation artifact that collapses accuracy from 40% to 1%, remedi- ated via continuous sequence packing; (2) linguistic pretraining is an essential prerequisite (≤2.0% without it); and (3) modern architectural primitives (RoPE, RMSNorm, SwiGLU) and Sparse Mixture of Experts (MoE) substantially improve additive reasoning over baseline GPT-2. Second, we demonstrate that algorithmic scratchpad formulation directly dictates suc- cess. Introducing a deterministic Digit-by-Digit Long Division scratchpad within a 4-stage Hierarchical Developmental Curriculum dramatically elevates single-digit division from 4.0% to 86.7% accuracy on a 4,000-problem held-out benchmark. In contrast, multi-digit multiplication remained challenging: detailed error analysis revealed that while the model correctly computed single-digit sub-products and place-value zeros, our FOIL scratchpad failed because it forced a simultaneous summation of up to nine multi-digit terms in a single step without pairwise intermediate accumulation. Finally, we identify two key boundaries: performance collapses to 0.00% on unseen 4-digit operands, and unbuffered training induces catastrophic forgetting, collapsing division accuracy from 86.7% down to 0.00%. 1 Introduction Autoregressive Transformers have achieved widespread empirical success across diverse natural language processing domains [3, 15]. However, their ability to perform exact mathematical reasoning, particularly multi-digit multiplication and division, remains fragile [4, 6, 20]. Scaling laws suggest that increasing model capacity to hundreds of billions of parameters mitigates these reasoning deficits. Nevertheless, recent investigations into small language models (SLMs), such as TinyStories [7], indicate that specialized inductive biases, targeted data engineering, and explicit step-by-step unrolling can induce complex structured behaviors at miniature scales. In standard autoregressive language modeling, token generation is governed by causal next- token prediction: P (x 1 ,x 2 ,...,x T ) = T Y t=1 P (x t | x <t ;θ)(1) When applied directly to multi-digit arithmetic (e.g., predicting 983×43 = 42269), next-token prediction requires the network to evaluate a complex composite function within the fixed forward-pass depth of its attention and feed-forward layers. To evaluate whether autoregressive ∗ Code, model weights, and benchmark datasets are available at https://github.com/SoKasl/AIMathLab. 1 arXiv:2610.09003v1 [cs.LG] 6 Oct 2026 transformers can execute deterministic algorithms without being confounded by linguistic sentence comprehension (such as story word problems in GSM8K [4]), we formulate a controlled experimental laboratory focused purely on canonical arithmetic operations (+,−,×,/) unrolled as sequential state transitions. In this work, we investigate whether compact Transformer models (∼10.6M non-embedding parameters) can reliably execute multi-step arithmetic algorithms. We construct a controlled experimental testbed to address four interconnected questions: 1.Linguistic Pretraining as an Inductive Foundation: Does natural language pretrain- ing act as a necessary syntactic scaffold for acquiring formal arithmetic reasoning syntax, and how does pretraining duration scale downstream convergence? 2. Architectural Inductive Biases: Do modern architectural upgrades—Rotary Position Embeddings (RoPE) [19], Root Mean Square Layer Normalization (RMSNorm) [21], SwiGLU activations [17], and Sparse Mixture of Experts (MoE) [18]—expand arithmetic capacity at comparable active compute? 3. Data Engineering and Gradient Density: How do sequence batching mechanics (padding vs. packing) affect gradient flow and algorithmic convergence? 4. Algorithmic Scratchpad Decomposition & Curriculum Staging: Why do direct FOIL and partial-quotient scratchpads [14,20] fail on multi-digit multiplication and division, and can algorithmic reformulation paired with hierarchical curriculum staging overcome these limitations? 2 Related Work Arithmetic Reasoning and Scratchpads. Early investigations demonstrated that standard transformers struggle with direct arithmetic computation [13]. Nye et al. [14] introduced scratchpads—intermediate execution steps that allow language models to emit scratch memory before producing final outputs. Chain-of-Thought (CoT) prompting [20] popularized this technique for few-shot prompting in frontier models. However, Dziri et al. [6] showed that transformers perform computation via subgraph matching rather than true algorithmic execution, exhibiting severe failure modes when composite problem graphs exceed the network’s effective reasoning horizon. Arithmetic in Small Transformers. Most relevant to our investigation is the work of Lee et al. [10], who demonstrated that compact (∼10M parameter) NanoGPT models can master addition and subtraction when provided with reversed digit formatting and linguistic pretraining. Our work builds upon and substantially extends their findings in several key dimensions: 1.Algorithmic Reform of Long Division: Lee et al. focused on addition, subtraction, and multi-digit multiplication. We investigate multi-digit division and demonstrate that traditional partial-quotient guessing fails in tiny models, whereas refactoring division into a deterministic digit-by-digit quotient descent algorithm achieves high accuracy. 2. Hierarchical Curriculum Staging: Rather than training on mixed datasets simultaneously, we formulate an explicit 4-stage developmental progression based on algorithmic sub-task dependencies. 3. Mixture of Experts Routing: We examine sparse conditional routing (MoE) in compact models to study whether parameter modularity benefits multi-task arithmetic. 2 4.Large-Scale Generalization & Drift Evaluation: We evaluate 4,000 held-out samples to establish exact length generalization boundaries and analyze catastrophic developmental drift. Length Generalization and Positional Inductive Biases. Extrapolating learned arith- metic procedures to operands longer than those seen during training remains an open challenge [1]. Kazemnejad et al. [8] showed that positional encodings fundamentally govern length extrapolation in transformers, finding that NoPE (no positional embeddings) or relative representations often outperform standard learned embeddings. We systematically evaluate RoPE [19] on multi-digit operands up to 4 digits. Curriculum Learning and Catastrophic Forgetting. Curriculum learning, introduced by Bengio et al. [2], posits that presenting training samples in order of increasing difficulty accelerates optimization. In sequential multi-task settings, however, neural networks suffer from catastrophic interference [9, 11], where learning new tasks overwrites previously mastered representations unless protected by rehearsal buffers [16]. 3 Experimental Setup and Architectures All models in our ablation suite are constrained to a compact budget ofL= 6 transformer layers, hidden dimension d model = 384, n heads = 6, and context block size T = 256. Parameter Accounting. The 6-layer transformer backbone comprises∼10.6M non-embedding parameters. When coupled with the standard OpenAI GPT-2 vocabulary (V= 50,257) embed- ding and unembedding matrices, the total parameter count is 49.3M for dense configurations. For the Sparse MoE configuration (with 4 SwiGLU experts per layer), the total parameter count is 70.5M, though only 2 experts are dynamically activated per token (maintaining active compute at approximately 1.5× dense FLOPs). 3.1 Architectural Configurations We examine four architectural configurations: 1.Baseline GPT-2: Standard scaled dot-product attention, learned absolute positional embeddingsW pos ∈R T×d , Layer Normalization [15], and standard GELU feed-forward networks with expansion ratio 4. 2. FlashAttention Optimization: Incorporates IO-aware exact FlashAttention [5]. As an exact mathematical reformulation of scaled dot-product attention, FlashAttention serves as a computational sanity check—yielding identical mathematical outputs while accelerating GPU/SRAM memory transfers from O(T 2 ) to O(T ). 3.Modern Dense Transformer: Replaces legacy GPT-2 components with modern state- of-the-art primitives: •Rotary Position Embeddings (RoPE): Replaces absolute positional embeddings with relative rotational transformations [19] applied directly to query and key projections: R d Θ,m = diag R θ 1 ,m ,R θ 2 ,m ,...,R θ d/2 ,m (2) 3 •RMSNorm: Eliminates mean-centering operations, stabilizing activations via root- mean-square scaling [21]: RMSNorm(x) = x q 1 d P d i=1 x 2 i + ε ⊙ γ(3) • SwiGLU Activations: Replaces standard two-layer MLPs with gated linear units utilizing the Swish activation function [17]: SwiGLU(x) = (Swish(xW gate )⊙ (xW up ))W down (4) 4.Sparse Mixture of Experts (MoE): Substitutes dense MLP blocks with a Sparse MoE layer [18] containing E = 4 distinct SwiGLU experts with Top-2 parameterized routing: H(x) = X i∈Top-2(G(x)) G(x) i · Expert i (x), G(x) = Softmax(Top-2(xW g ))(5) KV-Cached Inference Engine. For accelerated evaluation on large benchmarks, we imple- mented an autoregressive Key-Value (KV) cache engine with position offset tracking. During decoding, key and value states are cached across layers, reducing generation complexity per token fromO(N) toO(1) and achieving an inference throughput of∼8.8 problems/sec on Apple Silicon. Training Hyperparameters. To ensure an exact, unconfounded comparison, all models were trained using identical, untuned hyperparameters: AdamW optimizer (β 1 = 0.9,β 2 = 0.999, weight decay 0.01), learning rateη= 3×10 −4 without decay, batch size 4, and context length 256. 4 Task Formulation and Synthetic Data Generation To evaluate algorithmic reasoning strictly independent of linguistic comprehension artifacts, we formulate our mathematical benchmark entirely around pure arithmetic equations rather than natural language story problems. Each training instance follows a standardized prompt- scratchpad format: Q: What is A [op] B? : <thought> [Trace] </thought> [Final Answer] By enforcing a structured thought delimiter (<thought> ... </thought>), the autoregressive objective transforms standard next-token prediction into an unrolled simulation of algorithmic execution. We generate synthetic instances covering four core operations: 4.1 Place-Value Addition Operand pairs (a,b∈[10,999]) are decomposed by decimal place-value (hundreds, tens, ones). The scratchpad unrolls intermediate grouping and summation: Q: What is 482 + 357? A: <thought > 482 + 357 = 400 + 300 + 80 + 50 + 2 + 7 = 700 + 130 + 9 = 839 </thought > 839 4 4.2 Borrow-State Subtraction For subtractions (a≥ b), the trace explicitly computes column-wise differences from right to left, tracking borrow state propagation (10s and 100s): Q: What is 824 - 457? A: <thought > 4 ones - 7 ones (borrow 10) -> 14 - 7 = 7. 1 tens - 5 tens (borrow 100) -> 110 - 50 = 60. 7 hundreds - 4 hundreds -> 700 - 400 = 300. Answer = 300 + 60 + 7 = 367 </thought > 367 4.3 Multi-Digit Multiplication (FOIL Scratchpad) Multiplication is decomposed into polynomial partial products with base-10 zero accounting (e.g., 900× 40→ 9× 4 = 36 with 3 zeros → 36000), followed by an unrolled summation: Q: What is 84 * 32? A: <thought > 84 * 32 = (80 + 4) * (30 + 2). 80*30: 8*3=24 with 2 zeros -> 2400. 80*2: 8*2=16 with 1 zeros -> 160. 4*30: 4*3=12 with 1 zeros -> 120. 4*2: 8. Sum: 2400 + 160 + 120 + 8 = 2688. </thought > 2688 4.4 Deterministic Digit-by-Digit Long Division Unlike prior recursive partial-quotient estimation which guesses large multiples, our digit-by-digit algorithm explicitly models elementary school long division: Q: What is 3024 / 6? A: <thought > Divide 3024 by 6: Take 30: 30 / 6 = 5 (6 * 5 = 30), rem = 30 - 30 = 0. Bring down 2 -> 2: 2 / 6 = 0 (6 * 0 = 0), rem = 2 - 0 = 2. Bring down 4 -> 24: 24 / 6 = 4 (6 * 4 = 24), rem = 24 - 24 = 0. Final quotient = 504. </thought > 504 5 Data Engineering: The Token Density Collapse During fine-tuning, the mechanics of sequence batching substantially influence optimization dynamics. We evaluate three distinct dataloader paradigms (Figure 1): Streaming (Windowing) ...400+110=510</thought> Q: What is 82-14? A: <thought> 2 ones - 4... Density: 100% Flaw: Truncated prompts. (a) Naive Streaming Single-Problem Padded Q: What is 23+45? A: <thought>...68 [EOS] [PAD] [PAD] [PAD]... Density:∼17% Flaw: 83% gradient starvation. (b) Padded Instruction Sequence Packed Q1:..A1:..[EOS] Q2:..A2:..[EOS] Q3:..A3:..[EOS][PAD] Density:∼97% Benefit: High density + intact. (c) Sequence Packing Figure 1: Comparison of data loading strategies for 256-token context blocks. 5.1 The 40%→ 1% Performance Collapse In our initial experiments, switching from continuous streaming to an isolated single-problem padded dataloader resulted in a sharp drop in arithmetic performance: overall accuracy dropped from 40.0% to 1.0% across all model architectures (Table 1). Root Cause Analysis: Because a standard two-digit addition problem comprises only∼45 tokens, padding it into a 256-token block with masked cross-entropy loss (ignoreindex=−100) 5 MetricContinuous Streaming Single-Problem Padded Overall Accuracy40.0%1.0% Addition Accuracy84.0%4.0% Subtraction Accuracy76.0%0.0% Active Math Tokens / Step1,024∼180 Total Active Tokens (5k steps)5,120,000∼900,000 (5.7× drop) Gradient Sparsity0%82.8% Table 1: Empirical collapse under single-instruction padded training. Continuous Streaming Single-Problem Padded Sequence Packing 0 1 2 3 4 5 6 Active Tokens in 5k Steps (Millions) 5.12M 0.90M 4.98M Dataloader Packaging: Token Starvation vs Convergence 0 10 20 30 40 50 Math Accuracy (%) 40.0% 1.0% 45.0% Figure 2: Active token throughput in millions (blue, left axis) versus downstream arithmetic accuracy (red, right axis) across dataloader packaging strategies over 5,000 steps. Padded training induces severe gradient starvation, collapsing performance. 6 Run ArchitectureReasoning DatasetWiki Steps Math Steps Add (%) Sub (%) Mul (%) Div (%) Overall (%) 1Baseline GPT-2Standard FOIL20,0005,0008.012.00.00.05.0 2FlashAttentionStandard FOIL20,0005,0004.020.00.00.06.0 3Modern DenseStandard FOIL20,0005,00056.076.00.00.033.0 4MoE (Top-2)Standard FOIL20,0005,00060.084.00.00.036.0 5Modern DenseExtended FOIL20,000100,000100.0100.08.08.054.0 6MoE (Top-2)Extended FOIL20,000100,000100.0100.08.08.054.0 7Modern DenseZero-Padding Curricu- lum 20,000100,000100.0100.00.04.051.0 8MoE (Top-2)Zero-Padding Curricu- lum 20,000100,000100.0100.00.020.055.0 9Modern DenseHierarchical Devel- opmental 20,00030,00096.0100.00.056.063.0 10MoE (Top-2)Hierarchical Devel- opmental 20,00030,00088.096.00.060.061.0 Table 2: Master experimental ablation results across all 10 evaluation runs on the 100-problem evaluation set. Runs 1–4 evaluate baseline scratchpad configurations; Runs 9 and 10 introduce our deterministic digit-by-digit long division algorithm, elevating division from 4.0% to 60.0%. leaves 83% of the sequence empty. Over 5,000 gradient steps, the model receives updates from only 900,000 active tokens—equivalent to fewer than 900 steps of dense training. The model converges on syntactic formatting (emitting valid thought tags and halting tokens), but its internal arithmetic weights remain uncalibrated. 5.2 Sequence Packing with Prompt Masking We resolve this issue by implementing Sequence Packing with Instruction Masking. Multiple complete, intact equation-scratchpad-answer triples are concatenated sequentially within each 256-token block until full. Prompt tokens (Q) for each problem are masked with target label−100, while loss is computed exclusively on thought traces, final numerical answers, and ⟨|endoftext|⟩ stopping tokens: L =− T X t=1 1 y t ̸=−100 logP (x t | x <t ;θ)(6) This delivers∼97% token density, eliminates truncated sequences, and restores full convergence. 6 Empirical Ablations and Optimization Dynamics Table 2 summarizes the complete ablation matrix across all ten experimental configurations evaluated on our standardized 100-problem benchmark (25 problems per operation: Addition, Subtraction, Multiplication, Division). 6.1 Ablation 1: Validating the Necessity of Linguistic Pretraining To evaluate whether mathematical alignment can emerge independently of linguistic scaffolding, we varied WikiText-103 [12] pretraining duration from 10 to 50,000 steps prior to arithmetic fine-tuning. Across all architectures, the model was subsequently fine-tuned for 5,000 math steps using identical hyper-parameters (Table 3). As shown in Table 3, models initialized with only 10 steps of language exposure consistently fail to acquire arithmetic reasoning (scoring≤2.0%). Inspection of generated traces indicates that without language pretraining, models cannot learn the hierarchical delimiter syntax required to generate closed thought traces (<thought> ... </thought>). Conversely, 20,000 steps establishes robust syntactic comprehension, yielding up to 36.0% multi-task accuracy. Extending language training to 50,000 steps degrades performance (dropping to 19.0% in Modern Dense), 7 WikiText Pretraining StepsBaseline GPT-2 FlashAttention Modern Dense Sparse MoE (Top-2) 10 steps (Cold Start)0.0%1.0%2.0%2.0% 5,000 steps2.0%2.0%32.0%— 10,000 steps2.0%1.0%31.0%37.0% 20,000 steps (Optimal Scaffold)5.0%6.0%33.0%36.0% 50,000 steps (Linguistic Saturation)—2.0%19.0%33.0% Table 3: Impact of natural language pretraining duration on downstream arithmetic accuracy after 5,000 mathematical fine-tuning steps. Insufficient pretraining (≤10 steps) results in near-total failure across all architectures (≤2.0%), while 20,000 steps provides the optimal syntactic foundation. consistent with observations by Lee et al. [10] that prolonged linguistic pretraining can constrain parameter plasticity for downstream arithmetic specialization in compact models. 6.2 Ablation 2: Architectural Inductive Biases at Constant Scale Comparing Runs 1–4 isolates the impact of modern transformer primitives at an identical non-embedding parameter budget (∼10.6M) and fixed training budget (20,000 Wiki steps, 5,000 Math steps), presented in Table 4: Architecture ConfigurationAddition (%) Subtraction (%) Multiplication (%) Division (%) Overall (%) Run 1: Baseline GPT-28.012.00.00.05.0 Run 2: + FlashAttention4.020.00.00.06.0 Run 3: + RoPE + RMSNorm + SwiGLU56.076.00.00.033.0 Run 4: + Sparse MoE (Top-2 Routing)60.084.00.00.036.0 Table 4: Architectural ablation at constant∼10.6M non-embedding parameter scale. Modern inductive biases (RoPE, RMSNorm, SwiGLU) and Sparse MoE provide a substantial boost over standard GPT-2. • Exact FlashAttention Sanity Check (Run 1 vs. Run 2): Incorporating FlashAt- tention yields statistically indistinguishable performance (5.0% vs. 6.0%), confirming that exact tiling maintains mathematical parity with scaled dot-product attention while speeding up execution. • Legacy GPT-2 to Modern Dense (Run 1 vs. Run 3): Replacing absolute positional embeddings with RoPE, LayerNorm with RMSNorm, and standard GELU MLPs with SwiGLU raises Addition from 8.0% to 56.0% and Subtraction from 12.0% to 76.0%. RoPE’s relative rotary encoding allows attention heads to track relative place-value distance across multiple unrolled columns. • Dense vs. Sparse MoE (Run 3 vs. Run 4): Introducing Top-2 gating across 4 SwiGLU experts further increases Addition to 60.0% and Subtraction to 84.0%. While MoE increases total parameter count (70.5M vs. 49.3M), inference FLOPs remain comparable due to sparse Top-2 routing. 6.3 Ablation 3: Extended Training Plateau (5,000 to 100,000 Steps) In Runs 5 and 6, we extended mathematical training from 5,000 to 100,000 steps (a 20×compute increase). While Addition and Subtraction achieved 100.0% saturation, Multiplication and Division plateaued at 8.0%. Between step 50,000 and 100,000, overall performance increased by only 2% (from 52% to 54%, Figure 3). This demonstrates that raw training duration cannot overcome the inductive bottlenecks of ill-conditioned scratchpads. 8 020406080100 Math Training Steps (Thousands) 0 20 40 60 80 100 Accuracy (%) Overall Accuracy Plateau (10k to 100k Steps) Dense (FOIL) MoE (FOIL) Dense (Curriculum) MoE (Curriculum) 55% Ceiling 020406080100 Math Training Steps (Thousands) Run 8 Per-Operation Saturation (MoE 100k Steps) Addition (+): 100% Subtraction (-): 100% Division (/): 20% Multiplication (*): 0% Figure 3: Left: Overall multi-task accuracy plateau from 10,000 to 100,000 steps across Dense and MoE architectures, illustrating the sharp 55% ceiling. Right: Per-operation accuracy trajectory in Run 8 (MoE), demonstrating that Addition and Subtraction saturate at 100% while Multiplication remains at 0% and Division caps at 20%. 6.4 Ablation 4: Hierarchical Curriculum and Long Division Advance To resolve the plateau on division and multiplication, we designed a 4-Stage Hierarchical Developmental Curriculum structured around algorithmic sub-task dependency: 1. Stage 1: Additive Foundations (30k samples): 50% Addition, 50% Subtraction with explicit place-value carry and borrow tracking. 2.Stage 2: Single-Digit Multiplication Table (30k samples): 100% exhaustive coverage of the 10×10 multiplication table (0×0 to 9×9, 40%), combined with 60% Add/Sub rehearsal. 3.Stage 3: Multi-Digit FOIL with Zero-Accounting (40k samples): 60% Multi-digit multiplication, coupled with 40% additive rehearsal. 4.Stage 4: Digit-by-Digit Long Division (50k samples): 45% Long Division, 35% Multi-digit Multiplication, 10% Addition, 10% Subtraction. Replacing partial-quotient estimation with deterministic digit-by-digit descent produced a substantial performance gain on development evaluations (Runs 9 and 10 in Table 2): •Modern Dense (Run 9): Division reached 56.0% (14/25) on the development test set, lifting overall accuracy to 63.0%. • Sparse MoE (Run 10): Division reached 60.0% (15/25) at step 30,000, lifting overall accuracy to 61.0%. Interaction of Scratchpads and Curriculum Staging. Comparing Run 7 and Run 9 illustrates the necessary synergy between scratchpad formulation and curriculum staging. In Run 7, applying a curriculum while retaining the ill-conditioned partial-quotient scratchpad yielded only 4.0% division accuracy after 100,000 steps. This demonstrates that curriculum staging cannot compensate for scratchpads whose intermediate steps require implicit multi- digit estimation. When paired with deterministic digit-by-digit quotient descent, however, the hierarchical curriculum enabled rapid convergence, reaching 56%–60% accuracy within 30,000 steps. 9 Checkpoint Selection Disclosure: Checkpoints designated as “best” were saved based on peak overall accuracy on this 100-problem developmental set. Because a 25-problem operational slice carries a wide 95% confidence interval (≈±19%), this developmental metric serves primarily as a checkpoint selection signal; Section 7 provides the definitive, high-powered evaluation across 4,000 independent samples. 01020304050 Math Training Steps (Thousands) 0 20 40 60 80 100 Accuracy (%) Run 9: Modern Dense Architecture Addition (+) Subtraction (-) Division (/) Multiplication (*) Overall Accuracy 01020304050 Math Training Steps (Thousands) Run 10: Sparse Mixture of Experts (MoE) Addition (+) Subtraction (-) Division (/) Multiplication (*) Overall Accuracy Figure 4: Learning trajectories across mathematical operations throughout the 4-stage Hierar- chical Developmental Curriculum for Modern Dense (left, Run 9) and Sparse MoE (right, Run 10). Notice the steep rise of Long Division (red triangles) to 56–60% as the training corpus transitions into Stage 4. 7Large-Scale 4,000-Problem Benchmark and Developmental Dynamics While the 100-problem evaluation set (Table 2) provides an initial benchmark for architectural screening, its sample size (N= 25 per operation) carries a 95% Wilson score confidence interval of≈±19%, introducing potential variance in fine-grained comparisons. To establish rigorous statistical certainty, isolate length generalization limits, and analyze late-stage training dynamics, we generated a comprehensive 4,000-problem balanced evaluation benchmark (1,000 problems per arithmetic operation). 7.1 Benchmark Design and Evaluation Protocol The benchmark was synthesized deterministically using a fixed pseudo-random seed (seed=42) with the following operational partitions: •Addition (1,000 problems): 514 in-distribution problems (a,b∈[10,999], matching the training range) and 486 out-of-distribution problems (a,b∈[1000,4999], strictly unseen 4-digit operands). • Subtraction (1,000 problems): 517 in-distribution problems (a,b ∈[10,999],a ≥ b) and 483 out-of-distribution problems (a,b∈ [1000, 4999]). • Multiplication (1,000 problems): 250 1× 2-digit problems, 500 2× 2-digit problems, and 250 3× 2-digit problems. • Division (1,000 problems): 300 single-digit divisor problems (b∈[2,9], dividends up to 5 digits) and 700 multi-digit divisor problems (b∈[10,99]), constructed with exact integer quotients (a = b× c, remainder zero). 10 Standard FOIL (Runs 1-4) Extended FOIL 100k (Runs 5-6) Zero-Padding CoT Dense (Run 7) Zero-Padding CoT MoE (Run 8) Hierarchical Long Div Dense (Run 9) Hierarchical Long Div MoE (Run 10) 0 10 20 30 40 50 60 70 Division Accuracy (%) 0.0% 8.0% 4.0% 20.0% 56.0% 60.0% Deterministic Digit-by-Digit Long Division (15x Gain!) The Long Division Breakthrough: Paradigm Comparison in ~10M Transformers Figure 5: Long Division Performance: Comparison of Division accuracy across algorithmic paradigms in∼10M parameter networks. Transforming division from partial quotient guessing to deterministic digit-by-digit descent produces a substantial improvement (0–4%→60.0% on development tests). In-distribution instances were sampled independently from the same integer domain as the train- ing distribution, whereas 4-digit addition and subtraction instances tested unseen length regimes. All evaluations were executed via autoregressive categorical sampling (torch.multinomial at temperatureT= 1.0) up to a maximum context length ofT= 256 tokens. To extract the predicted numerical answer, the evaluator parses the integer immediately succeeding the </thought> delimiter, falling back to the terminal numerical string if the tag is unclosed. 7.2 Inference Acceleration via KV Caching Generating multi-step unrolled scratchpad reasoning traces across 16,000 total instances (4 model checkpoints×4,000 problems) imposes significant computational overhead. Standard autoregressive generation without key-value caching recomputes past keys and values across all prior sequence positions at each generated token step, requiring repeated full-context forward projections. By integrating a persistent key-value cache (K,V ∈R B×n heads ×T×d head ), linear projection layers are evaluated exclusively for the newly emitted token. While attending over past keys still requiresO(T) attention operations per step, eliminating redundant query-key-value trans- formations across preceding tokens significantly reduced generation latency across all benchmark sweeps. 7.3 Benchmark Results and Statistical Confidence We evaluated two developmental snapshots for both Modern Dense (Run 9) and Sparse MoE (Run 10): (1) the Step 30,000 checkpoint (peak multi-task performance), and (2) the Step 50,000 checkpoint (final saturated training). Results are detailed in Table 5, and key developmental phenomena are illustrated in Figure 6. Across the 4,000-sample benchmark, the 95% Wilson score margin of error is bounded by±3.1% for individual 1,000-problem operations and±1.5% overall. The difference between Step 30,000 and Step 50,000 on single-digit long division (86.67% 11 vs. 0.00% in Modern Dense; 84.33% vs. 1.67% in MoE) is statistically significant (Fisher’s exact test, p < 10 −15 ). Task / Operational SliceEvaluated SamplesModern Dense (Run 9)Sparse MoE (Run 10) Step 30,000 (Peak)Step 50,000 (Final)Step 30,000 (Peak)Step 50,000 (Final) Addition (≤ 3 digits, In-Dist)51493.39% (480/514)99.22% (510/514)91.44% (470/514)98.83% (508/514) Addition (4 digits, Out-of-Dist)4860.00% (0/486)0.00% (0/486)0.00% (0/486)0.00% (0/486) Subtraction (≤ 3 digits, In-Dist)51798.07% (507/517)99.42% (514/517)96.52% (499/517)99.42% (514/517) Subtraction (4 digits, Out-of-Dist)4831.86% (9/483)1.66% (8/483)0.62% (3/483)1.66% (8/483) Multiplication (1–3 digits, All)1,00011.20% (112/1,000)2.50% (25/1,000)5.10% (51/1,000)2.50% (25/1,000) Division (Single-digit divisor)30086.67% (260/300)0.00% (0/300)84.33% (253/300)1.67% (5/300) Division (Multi-digit divisor)7009.43% (66/700)0.14% (1/700)11.57% (81/700)7.71% (54/700) Division (All 1,000 problems)1,00032.60% (326/1,000)0.10% (1/1,000)33.40% (334/1,000)5.90% (59/1,000) Overall 4,000-Problem Benchmark4,00035.85% (1,434/4,000)26.45% (1,058/4,000)33.92% (1,357/4,000)27.85% (1,114/4,000) Table 5: Large-scale 4,000-problem benchmark comparing Step 30,000 (peak multi-task) and Step 50,000 (final) checkpoints across Modern Dense and Sparse MoE architectures. At Step 30,000, single-digit long division achieves up to 86.67% accuracy. Prolonged training to Step 50,000 drives addition and subtraction toward 99.4% saturation but induces catastrophic developmental drift on division. Unseen 4-digit addition completely collapses (0.00%), defining a strict length generalization boundary. Addition (In-Dist, <=3d) Addition (OOD, 4-digit) Subtraction (In-Dist, <=3d) Subtraction (OOD, 4-digit) 0 20 40 60 80 100 Accuracy (%) 99.2% 0.0% 99.4% 1.7% 98.8% 0.0% 99.4% 1.7% Length Boundary (Unseen 4 Digits) (a) Length Generalization Collapse (4-Digit OOD) Modern Dense (Step 50k) Sparse MoE (Step 50k) Modern Dense (Run 9)Sparse MoE (Run 10) 0 20 40 60 80 100 Single-Digit Division Accuracy (%) 86.7% 84.3% 0.0% 1.7% Catastrophic Drift: -86.7% Collapse (b) Catastrophic Drift on Single-Digit Division Step 30,000 (Peak Multi-Task) Step 50,000 (Overwritten / Drift) Figure 6: Large-Scale Benchmark Dynamics across 4,000 held-out problems. (a) Length Generalization Collapse: While models achieve near-deterministic accuracy (>99.2%) on in-distribution operands (≤3 digits), accuracy abruptly collapses to 0.00% on unseen 4-digit operands due to fixed positional and causal depth horizons. (b) Catastrophic Developmental Drift: Single-digit division performance peaks at Step 30,000 (86.67% Dense, 84.33% MoE), but prolonged un-buffered training to Step 50,000 reduces division accuracy down to 0.00% as late-stage addition gradients overwrite parameters. 7.4 Key Empirical Findings The large-scale evaluation exposes four critical empirical principles: 1. Single-Digit Long Division Execution: At Step 30,000, both Modern Dense and Sparse MoE models reliably execute deterministic digit descent, scoring 86.67% and 84.33% respectively on 300 unseen single-digit division problems (b∈[2,9], dividends up to 5 digits). This confirms that tiny models can reliably execute multi-step long division when the quotient search space is constrained to single-digit trial descent. 12 2.In-Distribution Additive Saturation: By Step 50,000, in-distribution addition and subtraction reach near-perfect saturation, scoring 99.22% and 99.42% across over 1,000 combined test samples. 3.The Length Generalization Boundary: Across both checkpoints and both architectures, 4-digit addition collapses entirely to 0.00% (0/486). Because rotary positional embeddings and causal attention layers were parameterized over sequences bounded by 3-digit operands during training, the network is unable to recursively propagate carry bits across longer unrolled sequence lengths without recurrence or explicit length-curriculum data. 4.Catastrophic Developmental Drift and Wrap-Around: Continuing training from Step 30,000 to Step 50,000 without balanced experience replay erodes algorithmic division capabilities: single-digit division plummets from 86.67% to 0.00% in Dense and 84.33% to 1.67% in MoE. Analysis of the dataloader reveals that the sequential curriculum file wrapped around back to Stage 1 (pure addition/subtraction), overwriting network weights previously tuned for division. 8 Qualitative Error Analysis and Failure Modes 8.1 Multiplication Summation Bottleneck In Runs 7 and 8, we trained models on a specialized curriculum dataset incorporating explicit base-10 zero accounting (e.g., 80×50→8×5 = 40, append two zeros→4000). Despite this innovation, multi-digit multiplication scored 0.0% on strict end-to-end evaluation. To uncover why multiplication failed, we conducted a token-by-token inspection of the generated evaluation traces: Listing 1: Actual generation trace from Run 9 (Math Step 30,000) Q: What is 983 * 43? A: <thought > 983 * 43 = (900 + 80 + 3) * (40 + 3). 900*40: 9*4=36 with 3 zeros -> 36000. [CORRECT] 900*3: 9*3=27 with 2 zeros -> 2700. [CORRECT] 80*40: 8*4=32 with 2 zeros -> 3200. [CORRECT] 80*3: 8*3=24 with 1 zeros -> 240. [CORRECT] 3*40: 3*4=12 with 1 zeros -> 120. [CORRECT] 3*3: 9. [CORRECT] Sum: 36000 + 2700 + 3200 + 240 + 120 + 9 = 43992. </thought > 43992 Expected: 42269 | Extracted: 43992 [Evaluated: FALSE] Trace Analysis: In our qualitative examination of generated multiplication traces, the model does not fail at single-digit multiplication facts (9×4 = 36, 8×4 = 32, 3×3 = 9), nor at base-10 zero accounting. Intermediate sub-products and zero-padding are correctly formed. The dominant point of failure occurs at the final multi-term summation line: Sum = 36000 + 2700 + 3200 + 240 + 120 + 9 = 43992(7) The prompt format required the model to aggregate up to nine multi-digit terms in a single autoregressive step without intermediate scratchpad accumulation. While a∼10.6M parameter network reliably computes two-operand addition with near-perfect accuracy, computing a nine-term summation in a single forward pass exceeds the representational depth of a 6-layer transformer. Because of an arithmetic summation error on this single step, the entire problem was evaluated as incorrect. 13 9 Discussion and Limitations 9.1 Pairwise Column-Summation Scratchpads Our analysis suggests an architectural direction for multi-digit multiplication in tiny models. Since qualitative traces indicate that models reliably generate individual sub-products (900× 40 = 36000), the primary bottleneck is the unrolled multi-term summation. Refactoring the final stage into an explicit pairwise accumulation scratchpad (e.g., 36000 + 2700 = 38700; 38700 + 3200 = 41900, etc.) would align with the model’s demonstrated 2-operand addition capability, potentially closing the multiplication gap. 9.2 Catastrophic Drift and the Necessity of Rehearsal Buffers A central finding from our large-scale 4,000-problem evaluation (Section 7) is the vulnerability of compact networks to catastrophic representational drift. Between Step 30,000 and Step 50,000, addition and subtraction saturated from∼93% to over 99.2% in-distribution, yet single-digit division execution collapsed from 86.67% to 0.00% in Modern Dense and 84.33% to 1.67% in Sparse MoE. Because compact∼10.6M parameter networks operate with minimal parameter redundancy, late-stage gradient updates overwrite previously learned algorithmic representations when training batches lack balanced task diversity. In continual learning and sequential task training, this phenomenon is conventionally mitigated through rehearsal buffers (or experience replay) [16], where a fixed proportion of historical task distributions (e.g., 20% division, 20% multiplication, 30% addition, 30% subtraction) is persistently interleaved into every active training batch. Our empirical results demonstrate that while hierarchical staging is essential for initial skill acquisition, maintaining composite algorithmic reasoning across developmental stages strictly requires permanent replay buffers. 9.3 Tokenization Confound A notable consideration in our experiments is the use of the standard OpenAI GPT-2 Byte Pair Encoding (BPE) tokenizer (V= 50,257). Unlike character-level or digit-level tokenizers, standard BPE tokenizes numbers non-uniformly: many 2-digit and 3-digit numbers are represented as single discrete tokens, whereas 4-digit numbers are split irregularly into subword tokens. This token boundary shift represents a plausible confounding factor in our out-of-distribution length generalization collapse at 4 digits, suggesting that future investigations into small arithmetic models should evaluate digit-level tokenizers. 9.4 Limitations We explicitly acknowledge several limitations of our empirical study: 1.Cold-Start Algorithmic Staging: Runs 9 and 10 demonstrate the efficacy of pairing digit-by-digit scratchpads with staged developmental curricula. While Run 7 establishes that curriculum staging alone cannot overcome ill-conditioned scratchpads (plateauing at 4.0% division), future work should evaluate whether digit-by-digit quotient descent can be acquired effectively from a uniform cold-start distribution, or whether hierarchical sub-skill mastery is strictly prerequisite. 2. Single-Seed Evaluations: Due to computational constraints across ten extended runs, the experimental trajectories reflect a single random seed. While the large-scale 4,000- problem benchmark provides strong statistical power on evaluation, multi-seed training runs will be valuable to characterize seed-level variance. 14 3.Decoding Stochasticity: Evaluation traces were generated via categorical sampling (T= 1.0), introducing stochastic variation across individual problem evaluations compared to deterministic greedy decoding. 4.Fixed Hyperparameters: All configurations used identical, untuned hyperparameters (η= 3×10 −4 ). Performance across individual architectures may further improve with targeted learning rate schedules and batch size tuning. 5.Untested Rehearsal Buffers: While the drift mechanism was isolated to curriculum wrap-around, implementing dynamic replay buffers during training remains future work. 10 Conclusion In this paper, we systematically investigated the arithmetic reasoning limits of compact (∼10.6M non-embedding parameter) Transformers. We validated that linguistic pretraining and sequence- packed token density are essential prerequisites for arithmetic convergence. We isolated the specific failure mode in multi-digit multiplication, showing that models reliably learn sub- multiplications and zero-padding but falter when required to aggregate up to nine terms simultaneously without pairwise intermediate accumulation. Finally, we demonstrated that restructuring long division into a deterministic digit-by-digit developmental curriculum elevates division accuracy from 4% to 60% on development tests and 86.7% on single-digit divisors on a 4,000-problem held-out benchmark. Our findings demonstrate that algorithmic unrolling aligned with transformer forward-pass capacity enables multi-step arithmetic execution in parameter-constrained regimes. References [1]Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ra- masesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. Advances in Neural Information Processing Systems, 35:38546–38556, 2022. [2] Yoshua Bengio, J ́erˆome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 41–48, 2009. [3]Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020. [4] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [5]Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher R ́e. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022. [6]Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena Hwang, Soumya Sanyal, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36:70244–70275, 2023. 15 [7]Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759, 2023. [8]Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36:36093–36129, 2023. [9] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017. [10]Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. arXiv preprint arXiv:2307.03381, 2023. [11]Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109–165, 1989. [12]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. [13]Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Investigating the limitations of trans- formers with simple arithmetic tasks. arXiv preprint arXiv:2102.13019, 2021. [14] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. [15] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019. [16]Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal for neural networks. Connection Science, 7(2):123–146, 1995. [17] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. [18]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. [19]Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. [20]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022. [21] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32:12360–12371, 2019. 16