Paper deep dive

Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models

Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz, Margaret Li, Mike Lewis, Luke Zettlemoyer, Srinivasan Iyer

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 122

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 9/16/2026, 8:28:04 AM

Summary

This paper investigates the scaling trends of distilled byte models versus token models. It introduces two methods for converting token logits to byte logits: Marginalize-It (approximate) and End-Of-Token (exact). The study finds that while token models perform better at low compute, byte models (specifically End-Of-Token) surpass them asymptotically in performance and data efficiency. The distilled End-Of-Token-1B model is predicted to outperform open-weight models like Llama 3.2-1B and Gemma-3-1B-pt.

Entities (11)

End-Of-Token · method · 98%Marginalize-It · method · 98%End-Of-Token-1B · model · 95%Token-1B · model · 95%Bytes · tokenizationscheme · 92%Cross-Entropy · trainingobjective · 92%Distillation · trainingobjective · 92%Tokens · tokenizationscheme · 92%Gemma-2B · model · 90%Gemma-3-1B-pt · model · 90%Llama 3.2-1B · model · 90%

Relation Signals (9)

Marginalize-It → isvariantof → LogitConversion

confidence 95% · introduce two variants to efficiently convert token logits to Byte Logits: 1) approximate: Marginalize-It

End-Of-Token → isvariantof → LogitConversion

confidence 95% · introduce two variants to efficiently convert token logits to Byte Logits: ... 2) exact: End-Of-Token

End-Of-Token-1B → usestokenization → Bytes

confidence 95% · byte models (End-Of-Token-1B and Bytes-1B)

Token-1B → usestokenization → Tokens

confidence 95% · Token-1B models outperform byte models

End-Of-Token → isexact → Marginalize-It

confidence 90% · approximate: Marginalize-It, and 2) exact: End-Of-Token

End-Of-Token-1B → outperforms → Token-1B

confidence 90% · byte models start worse yet surpass Token-1B models with more compute

End-Of-Token-1B → outperforms → Llama 3.2-1B

confidence 85% · distilled End-Of-Token-1B models asymptotically surpass the Llama 3.2-1B... by up to 6.5%

End-Of-Token-1B → outperforms → Gemma-3-1B-pt

confidence 85% · surpass the... Gemma-3-1B-pt... by up to 8.1%

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to efficiently convert token logits to Byte Logits: 1) approximate: Marginalize-It, and 2) exact: End-Of-Token. We then present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously: the tokenization scheme (Tokens, Bytes, Bytes w/ eot) and the training objective (Distillation vs. Cross-Entropy), sweeping layer-parameter-matched models with roughly 1 billion parameters up to 1 trillion bytes of data. Across eight benchmarks spanning three categories: Multiple Choice QA, Language Generation, and Machine Translation, we find that Token-1B models outperform byte models (End-Of-Token-1B and Bytes-1B) in the low-FLOP regime but eventually plateau; byte models start worse yet surpass Token-1B models with more compute, reaching a higher downstream task performance ceiling. Extrapolating the average top-1 error vs. validation BPB scaling laws predicts that, asymptotically, distilled End-Of-Token-1B outperforms distilled Token-1B by up to 4%. They are also far more data efficient, matching the performance of distilled Token-1B using only one-sixth of the training data. Moreover, by operating over a small vocabulary of 256 bytes instead of on the order of 100K tokens, they circumvent the need for top-k truncation during logit dumping, while also reducing logit storage costs to roughly one-fifth. Finally, our downstream performance scaling laws predict that our distilled End-Of-Token-1B models asymptotically surpass the Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B models on averaged downstream tasks by up to 6.5%, 8.1%, and 2.1%, respectively.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

119,003 characters extracted from source content.

Expand or collapse full text

Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models Kalyani Marathe 1‡†∗ , Artidoro Pagnoni 2†∗ , Tomasz Limisiewicz 12 , Margaret Li 1 , Mike Lewis 2# , Luke Zettlemoyer 12# , Srinivasan Iyer 2†#∗ 1 University of Washington, Seattle ‡ Work done at Meta FAIR 2 Meta FAIR † Joint first author. # Joint last author. Abstract Small models are made more capable through distillation from a larger one that shares their tokeniza- tion scheme. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to eﬀiciently convert token logits toByte Logits: 1) approximate:Marginalize-It, and 2) exact:End-Of-Token .We then present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously: thetokenization scheme(Tokens,Bytes,Bytes w/ <eot>) and thetraining objective(Distillation vs. Cross-Entropy), sweeping layer-parameter- matched models with≈1 billion parameters up to 1 trillion bytes of data. Across eight benchmarks spanning three categories: Multiple Choice QA, Language Generation, and Machine Translation, we find that Token-1Bmodels outperform byte models (End-Of-Token-1BandBytes-1B) in the low-FLOP regime but eventually plateau; byte models start worse yet surpassToken-1Bmodels with more compute, reaching a higher downstream task performance ceiling. Extrapolating the average top-1 error vs. validation BPB scaling laws predicts that, asymptotically, distilledEnd-Of-Token-1B outperforms distilledToken-1Bby up to4%on averaged downstream task performance. They are also far more data eﬀicient, matching the performance of distilledToken-1Busing onlyone-sixthof the training data. Moreover, by operating over a small vocabulary of≈256bytes instead of≈100K tokens, they circumvent the need for top-ktruncation during logit dumping, while also reducing logit storage costs to roughlyone-fifth. Finally, our downstream performance scaling laws predict that our distilledEnd-Of-Token-1Bmodels asymptotically surpass theLlama 3.2-1Bmodel (Meta AI, 2024),Gemma-3-1B-pt(Kamath et al.,2025), andGemma 2B(Team et al.,2024) models on averaged downstream tasks by up to6.5%, 8.1%, and2.1%, respectively. 1 Introduction Scaling has become the prevailing approach for improving language model performance (Henighan et al.,2020;Hoffmann et al.,2022;Kaplan et al.,2020). However, deploying models in real-world settings demands more than scale alone: the cost of serving can outweigh the benefits of scale (Dettmers & Zettlemoyer ,2023;Pope et al.,2023). Models are often 1) distilled into smaller yet capable models that, by training against the teacher’s next-token distribution rather than ground-truth tokens, outperform on downstream tasks (Beyer et al.,2022;Hinton et al.,2015), and 2) overtrained to amortize inference costs over a model’s lifetime (Gadre et al.,2024;Sardana et al.,2023). Yet it remains unclear how the choice of student tokenization interacts with these techniques as compute scales. In this work we compare how bytes and tokens affect distillation scaling trends as the compute budget grows. Bytes are a compelling alternative for two reasons: First, by compressing the vocabulary to just ≈256 values, transformers operating on bytes can preserve the distribution of a large token language model exactly ( Hayase et al.,2025;Phan et al.,2024) and potentially can bypass the need for top-k logit truncation for offline distillation. Second, they also possess interesting properties: they learn relation- ships between bytes from fewer samples than token models require for tokens ( Hestness et al.,2017), ∗ Correspondence:kmarathe@uw.edu,artidoro, sviyer@meta.com 1 arXiv:2609.12303v1 [cs.CL] 11 Sep 2026 which makes them data eﬀicient learners (Xue et al.,2022;Zheng et al.,2025). Distillation into such models, though, hinges on open questions: 1) how to eﬀiciently convert token logits to byte logits, 2) how does tokenization and training objectives affect the BPB vs FLOPs power laws 3) whether distillation outperforms supervised byte (and token) level training on downstream tasks, 4) at what compute budget byte-level surpasses token-level distillation, 5) how the trained models and the asymptotic predictions compare to the open-weights model of the comparable model sizes and what are the data and storage eﬀiciency implications. This work addresses all five. We summarize our key contributions in section1.1 below: 0.880.961.041.12 Validation BPB (descending -->) 25 30 35 40 45 50 Average top-1 Accuracy (%) Llama-3.2-1B Gemma-3-1B-pt Gemma 2B Marginalize-It Distilled End-Of-Token Distilled Token Distilled +4.0% +6.5%+8.1%+2.1%+1.9% Token Supervised Token Distilled Interpolation Extrapolation Asymptotic point (BPB, Acc) Bytes Supervised Marginalize-It Distilled Token Distilled Marginalize-It Distilled Llama-3.2-1B Bytes w/ <eot> Supervised End-Of-Token Distilled Gemma-3-1B-pt Gemma 2B Figure 1: Downstream task performance scaling laws for lr=4e-3 averaged across six benchmarks: Multi- ple Choice QA(ARC-Easy, ARC-Challenge, HellaSwag, PIQA) and Language Generation (MBPP, Natu- ral Questions) tasks. End-Of-Token Distilledmodel, by preserving the teacher distribution exactly as well as with the highest amount of compute spent on fixed amount of data raises the asymptotic perfor- mance ceiling.End-Of-Token Distilledasymptotically outperforms theToken Distilledmodel by 4%, theMarginalize-It Distilledmodel by1.9%, and the three open weights models: Llama 3.2-1B model (Meta AI,2024), Gemma-3-1B-pt (Kamath et al.,2025), and Gemma 2B (Team et al.,2024) models on averaged downstream tasks by upto6.5%, 8.1%, 2.1%respectively. 2 1.1 Summary of Contributions Token logits to Byte logits conversion in a single forward pass.Distilling a smaller byte model from a large token model first requires converting the teacher’s token logits onto byte logits. A token comprises several bytes, so the probability distribution must be split across byte positions. Prior meth- ods (Hayase et al.,2025;Phan et al.,2024) to convert token distribution to byte distribution require multiple forward passes which is prohibitive when teacher inference is computationally expensive. We introduce two single-pass variants for converting token logits toByte Logitsby marginalizing over byte positions: 1) approximate:Marginalize-Itre-normalizes over the prefix-matching tokens, while 2) exact:End-Of-Tokenadds an<eot>token to the vocabulary so no probability is lost. Bits-Per-Byte vs Training FLOPs Power Laws Across Tokenization and Training Objec- tives.With a logit conversion in hand, we ask how the resulting byte models scale compared to to- ken models and their supervised counterparts. We present a large-scale study of overtraining layer- parameter matched decoder-only dense transformers along two axes: the training objective (Distilla- tion loss vs Supervised/Cross-Entropy loss) and the tokenization scheme (Tokens,Bytes,Bytes w/ <eot>): sweeping decoder-only transformer models (Vaswani et al.,2017) with fixed 1.28 billion layer parameters up to 1 trillion bytes data. We train multipleToken-1B(1.81B),Bytes-1B(1.28B), and End-Of-Token-1B(1.28B) models to fit Validation BPB vs Training FLOPs power laws to show that distilledEnd-Of-Token-1Bmodels show better asymptotic behavior compared to distilledToken-1Band Bytes-1B. Average Downstream Task Accuracy vs Training FLOPs Scaling Trends.To understand how the scaling trends in validation loss translate into downstream task performance for Marginalize-It(ap- proximate) andEnd-Of-Token(exact) we now turn to benchmark evaluations. Across three categories and eight benchmarks: Multiple Choice QA (ARC-Easy, ARC-Challenge, HellaSwag, PIQA), Language Generation (MBPP, Natural Questions), and Machine Translation (Flores) we find thatToken-1Btrans- formers (supervised and distilled) with show strong performance at low compute regimes but plateau quickly.Bytes-1B(1.28B) andEnd-Of-Token-1B(1.28B), lag initially behind theToken-1B(1.81B) models, continue to improve at a steeper rate with additional compute despite fewer total parameters due to the small vocabulary size of≈256 bytes. Comparing Distillation Scaling Trajectories Across Compute Budgets.Downstream task per- formance vs training FLOPs scaling trend raises the practical question of which method to choose at a given compute budget. We fit Downstream error vs. Validation BPB scaling laws (Figure 1) and introduce Feather Plots(Figure11), which connect IsoFLOP points on the downstream-error vs. Validation- BPB curves. These plots let us compare downstream performance trajectories across compute budgets for each of the tokenization scheme. By extrapolating the downstream error vs validation BPB scaling laws, our experiments predict that distilledEnd-Of-Token-1Btransformers raise the asymptotic ceiling of the averaged downstream task performance. We then predict that by both preserving the teacher dis- tribution exactly as well as by spending ≈30.94%more compute on a fixed amount of data compared toMarginalize-It(approximate) method, distilledEnd-Of-Token-1B(exact) improves over distilled Token-1Bby up to4%and distilledBytes-1Bby up to1.9%asymptotically. End-Of-Token-1B: Data and Storage Eﬀiciency, Asymptotic win over Open-Weight models. The asymptotic accuracy gains fromEnd-Of-Tokenmethod also come with eﬀiciency gains. Distilled End-Of-Token-1Bmodel also surpass distilledToken-1Bmodels with up toone-sixthof the training data with suﬀicient training despite having fewer total parameters. By operating over a small vocabulary of≈256bytes instead of the≈100K tokens they circumvent the need for top-k truncation during logit dumping, reducing logit storage costs to roughly one-fifth. Finally, comparisons with the open-weight models with the asymptotic predictions reveal our distilledEnd-Of-Token-1Bmodel surpass theMeta AI(2024),Kamath et al.(2025), andTeam et al.(2024) models on averaged downstream tasks by upto 6.5% and 8.1%, 2.1%respectively. Paper Outline : We introduce two logit conversion methods:Marginalize-ItandEnd-Of-Tokento convert the token logits to byte logits (Section2). We then describe the experimental setup, the six scenarios used for our scaling study, and the benchmarks used for evaluations (Section 3). We then study the validation BPB vs FLOPs scaling laws (Section4), the downstream task performance vs training FLOP scaling trends (Section 5) and the downstream performance vs validation BPB scaling laws (Section 3 6). We then use these scaling laws to find out when byte distillation will be advantageous over token distillation baseline (Section7). Through the lens of downstream error scaling laws, with the help of Feather Plotswe demonstrate how the relative performance ordering between theToken Distilled, Marginalize-It Distilled, andEnd-Of-Token Distilledmodels evolve with the training compute budget. We then benchmark our trained checkpoints and our asymptotic predictions against the open weight models (Section8). Next, we contextualize our work within the existing literature (Section9). Lastly we reflect on the findings, limitations and opportunities for future work (Section10). 2 Logit Conversion Methods In this section we describe two methods:Marginalize-ItandEnd-Of-Token, we use in our scaling study to eﬀiciently convert token logits toByte Logits. Figure2demonstrate the byte logits formation process with the help of an example. We provide the pseudocode for the two methods in AppendixF.1 and discuss special cases in AppendixF.2. 2.1 Marginalize-It Logits For each token used in teacher inference, we have a|V|-sized logit tensor denoting the next-token dis- tribution, whereVis the vocabulary of the BPE tokenizer. We also have the decoded byte sequence of every BPE token inV. To compute thefirstbyte distribution of each BPE token, we marginalize over the full vocabularyVbased on the first byte. For eachsubsequentbyte, we restrict the vocabulary to the tokens whose prefixes match the ground-truth bytes and compute the byte distribution conditioned on this subset. We refer to this as the Marginalize-Itmethod. While this approach always yields an exact distribution for the first byte, the distributions over subsequent bytes are onlyapproximate. Figure2 (column 1) shows theMarginalize-Itlogit conversion process step by step. To predict the distribution B3, the continuations of tokenis(whose probability is 0.125) are silently dropped and only the tokens isuandiskare carried forward. While obtaining the next token distribution of tokeniscan fix this problem, it requires running additional teacher inference over the sequence:[<bos>, T, iram, is]. The number of additional inferences required to obtain exact conversions can get intractable for long sequences. To avoid that problem, in Marginalize-Itmethod remaining probability is redistributed among tokensisuandisk: P(isu) = 0.5 0.5 + 0.125 = 0.8,P(isk) = 0.125 0.5 + 0.125 = 0.2, During training, the byte sequence used is:[<bos>,T,i,r,a,m,i,s,u,<eos>]. We provide additional details in Appendix F.2.Marginalize-Itserves as the main baseline throughout the study. 2.2 End-Of-Token Logits TheEnd-Of-Tokenmethod is essentially theMarginalize-Itmethod with two key changes. First, while forming the byte logits after teacher inference, we append an<eot>token to mark the end of each BPE token as shown in Figure2(column 2). Second, we expand the byte vocabulary by one to store the<eot>probabilities and then apply marginalization over byte positions to form the byte logits. During training, we append<eot>after every BPE token in the pretraining dataset:[<bos>, <eot>,T,<eot>,i,r,a,m,<eot>,i,s,u,<eot>,<eos>,<eot>]. This resolves the missing-probability prob- lem of Marginalize-It. To computeB3distribution, unlikeMarginalize-It,End-Of-Tokenconsid- ersalltokens with prefixis. Since the total probability of tokens starting with prefixisis 0.5 + 0.125 + 0.125 = 0.75: P(isu<eot>) = 0.5 0.75 = 0.667,P(isk<eot>) = 0.125 0.75 = 0.167,P(is<eot>) = 0.125 0.75 = 0.167. In this way, the probability mass of the possible continuations ofisis absorbed into the<eot>token. This approach is exact, does not require additional inferences over teacher. Inserting<eot>tokens also has an unintended but remarkable consequence: the ≈30.94%additional compute it spends per unit of data (one<eot>every approx 4.5 bytes) turns out to further improve asymptotic performance relative to Marginalize-Itmethod. 4 Marginalize-It method End-of-Token method Tokens Bytes TokensBytes Prefix: “Tiram” B1 Prefix: “Tiram” B1 0.40.10.10.050.050.10.050.050.10.80.10.10.40.10.10.050.050.10.050.050.10.80.10.1 iiiiiimmnimniiiiiimmnimn sssmmniaesssmmniae ukpaotxwuk <eot> paotxw lp <eot><eot> lp <eot><eot><eot><eot> <eot><eot> Prefix: “Tirami” B2 Prefix: “Tirami” B2 0.50.1250.1250.06250.0625 0.125 0.50.1250.125 0.06250.06250.125 i i mmn 0.750.1250.125 i i mmn 0.75 0.1250.125 sssmmniae smn sssmmniaes mn ukpaotxwuk <eot> paotxw lp <eot><eot> lp <eot> <eot><eot><eot> <eot><eot> Prefix: “Tiramis” B3 Prefix: “Tiramis” B3 0.80.20.660.167 0.167 iiiiiimmnii i iiimmn sssmmniae 0.80.2 sssmmniae 0.66 0.167 0.167 ukpaotxwukuk <eot> paotxwu k <eot> lp <eot><eot> lp <eot><eot><eot><eot> <eot><eot> Prefix: “Tiramisu” B4 BPE BPE 1.0 iiiiiimmniiiiiimmn sssmmniaesssmmniae ukpaotxwuk <eot> paotxw 1.0 lp <eot><eot> lp <eot><eot><eot><eot><eot> <eot><eot> Figure 2: A step-by-step illustration of the two logit conversion methods. The first column demon- strates theMarginalize-ItByte Logitsformation process. Consider the tokenized promptTiramisu = [<bos>, T, iram, isu, <eos>], and in particular the next-token distribution following the token iram, for which the ground truth isisu. Each column represents the decoded byte sequence of a BPE token from the full vocabulary (pruned here for clarity) of the Llama 3-8B model, with the token’s probability shown at the top of the column; columns sharing a color share a byte prefix. To compute the first-byte distribution, the columns sharing the byte prefixB1are aggregated. Next, we subset the vocabulary by the ground-truth byteiand aggregate the probabilities overB2. Finally, we repeat this process forB3. A limitation of the Marginalize-Itmethod is that it does not account for all possible continuations after the BPE tokeniswhile predictingB3: here, for example, only the tokensisuand iskare considered. The End-Of-Tokenmethod (second column) addresses this problem by preserving the language modeling distribution exactly, assigning the residual probability mass to the<eot>token, as illustrated for the prediction of byteB3. 5 3 Scaling Study Design 3.1 Experimental Setup In this section we describe the Model Architecture, Pretraining Dataset and Training Details. We provide specifics of these settings in SectionA. Model Architecture.We use the transformer architecture (Vaswani et al.,2017) to train all of the layer- parameter-matched models with≈1 billion parameters. We use three variants:Token-1B,Bytes-1B , andEnd-Of-Token-1Blisted in Appendix (Table9). While the three model architectures have equal total layer parameters (1.28B), theToken-1B(1.81B) is in fact larger due to the larger vocabulary size. We stick to the Llama-3 (Grattafiori et al.,2024) implementation for the model design choices. Training Dataset.We use the Llama-2 (Meta AI,2024) training mixture for our supervised and dis- tillation experiments. Our study is designed such that both the token logits and byte logits occupy the same amount of memory for the same amount of data. See sectionA.6for the full storage cost breakdown. Training Details.We train our models with the AdamW (Loshchilov & Hutter,2017) optimizer and sweep three learning rates∈1e-3, 4e-3, 8e-3. We use 10% warmup steps for each data scale. We provide additional details on the loss function in Appendix (A.3) 3.2 Scaling Along Two Axes: Tokenization and Training Objectives Our study seeks to explore scaling trends of layer-parameter-matched models with overtraining for six experiments described below. This enables us to understand the complex relationships between the Supervised (Cross-Entropy Loss) and Distillation pretraining as well as the effects of the tokenization schemes: Tokens,Bytes,Bytes w/ <eot>on the training dynamics. Table1details the six experi- mental settings in our overtraining study. Model size and FLOPs/unit values are obtained from Table 9. ForMarginalize-ItandEnd-Of-Tokendistillation we use the logit conversion process described in Section2. ExperimentTokenizationVocab. size ObjectiveArchitectureModel size FLOPs/unit Token SupervisedTokens(Llama 3-8B)128,256 Cross-EntropyToken-1B1.81B8.18×10 9 Token DistilledTokens(Llama 3-8B)128,256 Distillation (top-k)Token-1B1.81B8.18×10 9 Bytes SupervisedBytes(256 + 4 special)260Cross-EntropyBytes-1B1.28B8.81×10 9 Marginalize-It DistilledBytes(256 + 4 special)260Distillation (approximate)Bytes-1B1.28B8.81×10 9 Bytes w/ <eot> SupervisedBytes w/ <eot>(256 + 5 special)261Cross-EntropyEnd-Of-Token-1B1.28B9.44×10 9 End-Of-Token DistilledBytes w/ <eot>(256 + 5 special)261Distillation (Exact)End-Of-Token-1B1.28B9.44×10 9 Table 1: Six experiments in our overtraining study across the training-objective and tokenization axes. 3.3 Benchmark Evaluations Multiple Choice Question Answering. We evaluate performance on four benchmarks which evaluate the ability of a model to solve multiple choice questions. Specifically we use ARC-Easy (P. Clark et al., 2018), ARC-Challenge (P. Clark et al.,2018), HellaSwag (Zellers et al.,2019), and PIQA (Bisk et al., 2020) to benchmark the token and byte checkpoints with and without distillation. Language Generation Tasks.We further evaluate the language modeling capabilities of the byte and token models on MBPP (Austin et al.,2021) and Natural Questions (Kwiatkowski et al.,2019). Machine Translation.Lastly we also evaluate our trained checkpoints on English to German and German to English translation using the Flores (Goyal et al.,2022) benchmark. 6 4 Scaling Trends I: Validation BPB vs Training FLOPS In this experiment we analyze the overtraining behavior of the six experiments described in3.2. We plot the BPB of the model on a held out validation dataset. 4.1 Validation BPB vs Training FLOP Plots We fit the power law for validation BPB with respect to FLOPs using equation (1) wherexis the number of FLOPs and y is the BPB at a given compute budget (Table2). They, asymptotically approaches c.: y=b·x a +c(1) 10 20 10 21 10 22 10 23 Training FLOPs 0.925 0.950 0.975 1.000 1.025 1.050 1.075 Bits per Byte (BPB) LR = 4e-3 Token Supervised (c=0.9568) Token Distilled (c=0.9407) Byte Supervised (c=0.8967) Marginalize-It Distilled (c=0.9016) Bytes w/ <eot> Supervised (c=0.8891) End-Of-Token Distilled (c=0.8983) Data Scales 20B40B68B102B149B203B298B678B1.2T 16B33B55B82B121B165B242B550B968B 15B30B50B110B220B500B Bytes w/ <eot> Bytes Tokens 220B Token ScaleInterpolationExtrapolation Figure 3: The figure shows our scaling curves for different data scales, for a learning rate of 4e-3 and a fixed parameter count of approximately 1 billion parameter transformer models(see Table 9for details). Note that data scales plot serves as an alternative x axis to training flops. Since the FLOPs per unit changes for the three models, we specify the number of units used for training. The distilled models are trained using Llama 3-8B as a teacher model. The validation BPB of the teacher model is 0.85511. 7 Table 2: Power-law scaling equationsy=b·x a +cfitted to BPB vs. training FLOPs for LR=4e-3. ScenarioEquationR 2 Token Supervisedy= 1.075×10 9 ·x −0.500369 + 0.95680.9999 Token Distilledy= 2.803×10 8 ·x −0.475375 + 0.94071.0000 Bytes Supervisedy= 4.467×10 6 ·x −0.365844 + 0.89670.9980 Marginalize-It Distilledy= 2.706×10 7 ·x −0.407427 + 0.90160.9974 Bytes w/ <eot> Supervisedy= 3.080×10 6 ·x −0.354594 + 0.88910.9982 End-Of-Token Distilledy= 2.734×10 7 ·x −0.404740 + 0.89830.9990 Table 3: We observe that the amount of FLOPs spent on a fixed amount of data (one BPE token in this case) affects the asymptotic BPB. Both supervised and distilled models asymptotically follow descending BPB trend as the Total FLOPs Per Token increase. The FLOPs/Unit values are obtained from Table9. Depending on the model architecture, a unit can be a token or a byte. We note thatEnd-Of-Token-1B uses≈30.94%extra FLOPs per token compared toBytes-1B. MethodArchitectureFLOPs / Unit Units Total FLOPs Asymptotic BPB Per Token Per Token4e-3 Token SupervisedToken-1B8.18×10 9 1.08.18×10 9 0.9568 Bytes SupervisedBytes-1B8.81×10 9 4.539.65×10 9 0.8967 Bytes w/ <eot> SupervisedEnd-Of-Token-1B9.44×10 9 5.551.92×10 9 0.8891 Token DistilledToken-1B8.18×10 9 1.08.18×10 9 0.9407 Marginalize-It DistilledBytes-1B8.81×10 9 4.539.65×10 9 0.9016 End-Of-Token DistilledEnd-Of-Token-1B9.44×10 9 5.551.92×10 9 0.8983 We notice several trends from these plots: 1.First, bothMarginalize-It Distilled(approximate) andEnd-Of-Token Distilled(exact) meth- ods demonstrate improvements over their supervised counterparts. 2.Marginalize-It Distilledtraining, despite being an approximation works very well in practice. End-Of-Token Distilledmethod is exact and yields consistent improvements over the supervised training. 3.Second, we also observe thatToken-1Bmodels have the lowest BPB at lower compute budgets, and appear to saturate faster. The End-Of-Token-1Bhave the worst BPB to begin with yet yield the best asymptote in the infinite compute regime. Bytes-1Btraining runs, appear to be sandwiched between the other two scenarios at the beginning and the end of the training runs. 4.From Table3, we observe that asymptotically for both supervised and distilled models: 1)End-Of-Token-1B BPB<Bytes-1BBPB<Token-1BBPB. We also observe that the supervised and distilled models with the same tokenization scheme exhibits small differences in the asymptotic value. 5.We also observe interesting trends across all training runs. At around1.8×10 21 FLOPs, the Token Supervisedtraining run appears to have the highest BPB of all, with all other five curves demonstrating lower BPB values. Does this mean that all other five models at that compute budget are better than Token Supervised? To find out, we turn to benchmark evaluations in Section5. 5 Scaling Trends I: Task Performance vs Training FLOPs In this experiment we seek to understand if the observations from the validation BPB curves from section 4are consistent on a variety of benchmarks. We evaluate the capabilities of our trained models on a total of eight benchmarks spanning three categories: Multiple choice question answering, Generative, and Machine Translation tasks (Refer to section 3.3for more details). 8 5.1 Results on Multiple Choice Question Answering Benchmarks To evaluate the capabilities of trained models on four multiple choice question answering tasks we plot the % Accuracy vs Training FLOP trends for our six experiments in Figure4. In addition we also report the averaged accuracy across four benchmarks. We observe that: 1.The averaged choice accuracy improves for all of the 6 experiments, and the distilled models con- sistently show better performance compared to the supervised models. This trend also holds for the benchmarks individually. 2.Token-1Bmodels start best, outperforming all of the byte curves at lower compute budgets. All of the byte curves start worse but appear to be improving faster than the token models, eventually catching up with them. 3.If we compare within the four byte experiments,both supervised and distilledEnd-Of-Token-1B start worse, and eventually surpass theBytes-1Bmodels methods respectively. 4.Token-1Bmodels, both supervised and distilled have the best performance at the beginning and appear to be eventually saturating, a pattern also observed in Validation BPB vs Training FLOPs curves. 5.Lastly we observe that at 220B token supervised budget only the distilledToken-1Boutperforms its supervised counterpart curve but the byte curves, both supervised and distilledBytes-1Band End-Of-Token-1Bmodels are much worse, which is counterintuitive to the key observation in5. 10 20 10 21 10 22 Training FLOPs (log scale) 42.5 45.0 47.5 50.0 52.5 55.0 57.5 60.0 Choice Average Accuracy (%) LR=4e-3 Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled 220B Token Supervised Figure 4: We plot Accuracy vs training FLOP curves for all the six scenarios described in section3.2. For all of the Multiple Choice Question Answering Benchmarks, we observe that the token models show better performance at low compute budgets and eventually plateau. Bytes-1BandEnd-Of-Token-1B models: both supervised and distilled, start worse yet improve at a much faster rate. 5.2 Results on Language Generation Benchmarks In this experiment we study the language generation capabilities of the trained models on two benchmarks ( Figure5) (more details in section3.3). We observe that: 9 1.The trends are similar to Multiple Choice benchmarks. However the saturation effect ofToken-1B models seems to be less pronounced, potentially because the tasks are much harder. 2.In all cases, distilled models appear to be better for the same compute budget and the result about observation5at1.8×10 21 FLOPs budget is observed in this case as well. Only theToken Distilledmodel has better performance while all other four byte curves are much worse in most cases. 10 20 10 21 10 22 Training FLOPs (log scale) 0 2 4 6 8 10 12 Generation Average Accuracy (%) LR=4e-3 Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled 220B Token Supervised Figure 5: We visualize % Accuracy vs training FLOPs curves for language generation tasks. On generative tasks, token models achieve better performance at lower compute budgets and appear to plateau with the training FLOPs. Bytemodels andBytes w/ <eot>models show a different scaling behavior, with much worse performance at a lower compute budget which continues to improve with FLOPs with no sign of saturation. 5.3 Results on Machine Translation Benchmarks We also evaluate models on machine translation benchmarks and plot BLEU score (Papineni et al.,2002) vs Training FLOP curves. The plots (Figure6) show that: 1.Byte model BLEU scores appear to be following an S-shaped trend with training FLOPs, improv- ing slowly at low compute regimes, followed by a steeper improvement regions and then finally saturating at high compute budgets. 2.Next, the results on these benchmarks contradict the observation5here as well. 3.End-Of-Token Distilledmodels by preserving the teacher distribution exactly show significant improvements overBytes w/ <eot> Supervisedmodels. 10 10 20 10 21 10 22 Training FLOPs (log scale) 0 5 10 15 20 25 30 MT Average BLEU LR=4e-3 Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled 220B Token Supervised Figure 6: We plot BLEU score for the translation tasks for different training budgets. We observe clearly different scaling behaviors for token and byte models. Token models can achieve significantly better performance at smaller compute budgets. The BLEU score appears to saturate with overtraining. Byte models both with and without the<eot>demonstrate extremely low BLEU scores at lower compute budgets yet the rate of improvement with overtraining is better compared to token models. 5.4 Concluding Remarks In section5.1,5.2, and5.3, we consistently saw that despite achieving better validation BPB at≈1.8×10 21 FLOPs, all four byte models, both distilled and supervised continue to be much worse compared to Token Supervisedmodel. What could explain this behavior? In section6we set out to explain this phenomenon. For the data points we observe, the distilled models achieve lower BPB and better overall down- stream task performance than their supervised counterparts. However, this does not always hold when comparing the Token-1Bagainst theBytes-1BandEnd-Of-Token-1Bmodels: aToken-1B can have worse BPB yet better downstream performance, as observed at the 220B token data scale. 6 Scaling Trends I: Task Performance vs Validation BPB We now turn to explaining how the observations in sections4and5fit together. For all of the three downstream task categories we study the relationship between the downstream task performance and the validation BPB. To do so, we use the formulation of scaling laws for downstream tasks in ( Gadre et al., 2024). We plot the Average top-1 error, essentially,(1−% Accuracy)against the validation BPB, for Multiple Choice Question Answering (section 6.1) and Language Generation (section6.2). Lastly, since Machine Translation (section6.3) uses BLEU as an evaluation metric, we plot the BLEU score vs the validation BPB. 11 6.1 Results on Multiple Choice Question Answering Benchmarks 1.Across all benchmarks as well the averaged performance (Figure7), all of the six scaling trends demonstrate significantly different behavior. First, if we compare the performances for constant BPB, they imply wildly different error rates. Second, the iso-error lines intersect with theEnd-Of-Token andTokencurves at very different compute budgets. 2.Second: Changes in the tokenization scheme i.e. the amount of FLOPs spent to model a fixed amount of data can dramatically change the scaling behavior. For instance,Bytes-1BandEnd-Of-Token-1B models do not show significantly different scaling behaviors compared toToken-1Bmodels. 3.Lastly, the training objective (supervised as well as distillation) also appear to alter the scaling behavior. 0.951.001.051.101.15 Validation BPB (descending -->) 0.40 0.50 0.60 Choice Average Error (1 - % Accuracy) LR=4e-3 Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled Figure 7: We visualize Average-top-1 error vs the validation BPB curves for the six scenarios for LR=4e-3. All of these curves including the averaged metric appear to follow an exponential decay. Token-1Bshows significantly different trend compared toBytes-1BandEnd-Of-Token-1B. Both the training objective (Cross-Entropy vs Distillation) as well as the tokenization scheme:Tokens,Bytes,Bytes w/ <eot> appear to influence the Average Error vs Validation BPB trends. 6.2 Results on Language Generation Benchmarks We now visualize the Average top-1 error vs Validation BPB curves for the language generation tasks described in section3.3. In addition, we also plot the averaged generation performance (in row 1). 12 1.We observe that for language generation as well, the same BPB can mean different averaged down- stream performance for the three experiment categories - (Token,Bytes,Bytes w/ <eot>) as well as training objectives - supervised and distillation. 0.951.001.051.101.15 Validation BPB (descending -->) 0.88 0.90 0.92 0.94 0.96 0.98 1.00 Generation Average Error (1 - % Accuracy) LR=4e-3 Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled Figure 8: We plot the Average-top-1 error vs BPB curves for the MBPP and the NQ benchmarks. We notice an exponential decay for all three scenarios, with significantly different trends for the Token-1Bvs Bytes-1BandEnd-Of-Token-1Bmodels. Distilled models as well demonstrate slightly different scaling behavior compared to the supervised counterparts. 6.3 Results on Machine Translation Benchmarks Lastly we plot BLEU score vs Validation BPB curves for Machine Translation tasks (Figure 9). We observe that: 1.TheToken-1Bmodels,Bytes-1Bmodels andEnd-Of-Token-1Bmodels show different scaling be- haviors. 2.On average,End-Of-Tokendistillation appears to exhibit monotonic trend compared to the noisy Bytes w/ <eot> Supervisedcurves. 13 0.951.001.051.101.15 Validation BPB (descending -->) 0 6 12 18 24 30 MT Average BLEU LR=4e-3 Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled Figure 9: We plot averaged BLEU score vs Validation BPB performance, and on the averaged BLEU performance for the six scenarios for LR=4e-3. We notice the differences in the scaling behavior across tokenization schemes as well as the training objectives. End-Of-Tokendistillation appears to exhibit monotonic trend compared to the noisy supervised curves. 6.4 Fitting the Downstream Task Error vs Validation BPB Scaling Law We fit the equation2for our six experiments on averaged top-1 error calculated over total 4 Multiple Choice and 2 language generation benchmarks (from3.3) whereydenotes the BPB value,z(y)is the Averaged Top-1 Downstream Task Error, andε, k andγare fit using data. The scaling law equations are detailed in Table 4. The interpolations and extrapolations are visualized in Figure10. The curves are bounded by the asymptotic BPB c, obtained from the power law equations in Table2. z(y) =ε−k·e −γy (2) 14 Table 4: Fitted downstream error scaling lawsz(y) =ε−k·e −γy (equation2) for each model variant and learning rate of 4e-3, whereyis the validation BPB. MethodModel Architecture Scaling law equationz(y) Token SupervisedToken-1B0.7501−934.3212e −8.7782y Token DistilledToken-1B0.7383−1292.0124e −9.2132y Bytes SupervisedBytes-1B0.7686−1231.1241e −9.3532y Marginalize-It DistilledBytes-1B0.7721−1137.3004e −9.2265y Bytes w/ <eot> SupervisedEnd-Of-Token-1B0.7972−185.6303e −7.1822y End-Of-Token DistilledEnd-Of-Token-1B0.7776−827.6076e −8.8125y 0.880.961.041.12 Validation BPB (descending -->) 0.46 0.50 0.60 0.70 Average Error (1 - % Accuracy) Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled InterpolationExtrapolationAsymptotic point (BPB, Err) Figure 10: We fit the downstream error scaling laws for six benchmarks with respect to Validation BPB. All six experiments follow a smooth trend.Token-1Bcurves show significantly different behavior com- pared to the Bytes-1BandEnd-Of-Token-1Bcurves. The training objective (distillation vs supervised (Cross Entropy) also affects the scaling laws.End-Of-Token Distilled, by preserving the teacher distribution exactly as well as with the highest amount of compute spent on fixed amount of data outper- forms all of the other models asymptotically. ByteandBytes w/ <eot>models, despite having fewer total parameters asymptotically surpass theTokenmodels. 6.5 Downstream Task Performance Upper Bound To calculate the asymptotic downstream task performance upper bound, we use the Average top-1 error at asymptotic validation BPB c, for each curve obtained from equation2. We then compute the asymptotic 15 performance at that BPB as shown in Figure10. Table4shows the asymptotic BPB as well as the task performance of the six experiments. We observe that: 1.We notice that distilledToken-1Basymptotically outperforms over its supervised counterpart. 2.End-Of-Token Distilledmethod outperforms bothToken DistilledandBytes w/ <eot> Supervised asymptotically. 3.DistilledBytes-1Bperforms on par or suboptimally compared to its supervised model potentially because it uses an approximation of the teacher distribution. It however still outperforms asymptotic distilledToken-1Baccuracy. 4.Lastly, we note that by spending around≈30.94%additional compute on a fixed amount of data,Bytes w/ <eot> Supervisedshows better asymptotic behavior compared to theBytes Supervisedmodel. 5.End-Of-Token Distilledshows better asymptotic downstream performance (lowest Average top- 1 error) compared to theBytes w/ <eot> Supervisedmodel. Table 5: Asymptotic validation BPB (c) and the corresponding downstream average accuracya ⋆ = 1−%z ⋆ wherez ⋆ is the Asymptotic Average top-1 error predicted by the scaling law equation:2, z ⋆ =ε−k·e −γc , wherecis obtained from the power law equations in Table4for each model variant and learning rate. Methodca ⋆ (%) Token Supervised0.9568 46.0 Token Distilled0.9407 48.4 Bytes Supervised0.8967 51.2 Marginalize-It Distilled0.9016 50.5 Bytes w/ <eot> Supervised0.8891 51.6 End-Of-Token Distilled0.8983 52.4 6.6 When Lower BPB Does Not (Always) Imply Better Models Our observations show that BPB (Bits-Per-Byte) does not allow fair comparison across models with different training objectives (Distillation and Cross-Entropy) and tokenization schemes:Tokens,Bytes ,Bytes w/ <eot>. We show that the relationship between the BPB acrossTokens,Bytes,Bytes w/ <eot> is nuanced in the context of benchmark evaluations. These disparities potentially arise from the differences in probability distributions learned by spending more compute per unit data on a small amount of data vs small amount of compute spent per unit data on larger amount of data under a fixed compute budget. Although all of the Bytes-1BandEnd-Of-Token-1Bmodels surpass theToken SupervisedBPB training curve, they exhibit worse downstream task performance across benchmarks, falsifying observation5when the models being compared have different training objectives and tokeniza- tion schemes, emphasizing the importance of this final calibration step while determining the quality of a language model before real-world deployment. See appendix G.1and section10for additional discussion. 7 The Case forEnd-Of-TokenLogits To understand ifEnd-Of-Tokenlogits are advantageous, it is essential to find out when they will if at all surpass the distilled token models. To achieve that we use the downstream error prediction scaling laws from 6.4as shown below: 16 7.1Feather Plot: Visualizing Iso-FLOP Point Connections 0.880.961.041.12 Validation BPB (descending -->) 0.50 0.60 0.70 Average Error (1 - % Accuracy) LR=4e-3 10 20 10 22 10 24 10 26 10 28 10 30 Training FLOPs (isoFLOP lines) Token Distilled Token Distilled asymptotic error End-Of-Token Distilled End-Of-Token Distilled asymptotic error Marginalize-It Distilled Marginalize-It Distilled asymptotic error Asymptotic point (BPB, Err) Crossover point: reaches Token Distilled asymptotic error IsoFLOP budget projection (10^20.6 FLOPs) Interpolation (within data range) Extrapolation (beyond data, to asymptote) Figure 11: The figure shows aFeather Plotfor a learning rate of 4e-3. We show thatEnd-Of-Token Distilled transformer model has the lowest Average top-1 downstream task performance error asymp- totically compared to theMarginalize-ItandTokendistillation. We also visualize the Validation BPB values and the Average top-1 downstream task error at the compute budget of10 20.63 FLOPs (a randomly sampled value). We note the differences in observed BPB value and the task performance by projecting iso-FLOP points on the X and Y axes for the three curves. At that compute budget,Tokendistillation leads significantly, while bothEnd-Of-Tokendistillation andMarginalize-Itdistillation lag behind. We visualize lines connecting Iso-FLOP points which we callFeather Plotsto understand how the BPB(Bits-Per-Byte) value as well as the downstream task performance evolves with compute budget. To plot them we sample points from FLOP rangef1tof2. For each sampled pointxwe computeyusing equation 1. Next we use the obtained validation loss in terms of BPByto compute the downstream error z(y)using equation2. We connect the points obtained for three methods:Token,Marginalize-It, End-Of-Tokendistillation to analyze the trend. We observe for example the learnig rate of 4e-3: 1.At low compute budgets, distilledToken-1Bperformance leads significantly compared toBytes-1B andEnd-Of-Token-1Bmodels. 2.As the compute budget progresses, distilledEnd-Of-Token-1Bovertakes theBytes-1Bmodels. 17 3.Eventually, distilledEnd-Of-Token-1Bcatches up with theBytes-1Btoo and continues to improve until the asymptotic value of the performance. 7.2 Calculating Downstream Error Cross-Over Point with Token Distillation Given the validation BPB prediction and downstream error prediction scaling laws for distilledToken-1B ,Bytes-1B,End-Of-Token-1Bmodels we want to find out at what compute budgetMarginalize-It DistilledandEnd-Of-Token Distilledmethods will give the same asymptotic downstream error prediction as that of aToken Distilledmodel. Table 6: Training compute at which distilledBytes-1BandEnd-Of-Token-1Bmatch the asymptotic downstream performance ofToken-1B. We takeToken Distilledasymptotic BPB (thecterm of its fitted BPB–FLOPs power law; column Token Dist. BPB) and map it throughToken Distilled benchmark scaling law to obtain its asymptotic downstream accuracy (column Accuracy). For each byte method we then invert its own benchmark scaling law at that accuracy to get the BPB it must attain (column Method BPB), and invert its BPB–FLOPs power law at that BPB to get the required training compute (column Training FLOPs). MethodToken Dist. BPB Accuracy (%) Method BPB Method Training FLOPs Marginalize-It Distilled0.940748.4%0.91022.04×10 23 End-Of-Token Distilled0.940748.4%0.91456.33×10 22 Table 7: Effective training-data volume at the compute budget required to achieve theTokendistilla- tion asymptotic performance from Table 5forMarginalize-ItandEnd-Of-Tokenmethods.Bytes =Byte Units×Multiplier converts byte units into actual UTF-8 bytes of unique text, accounting for the fraction of each unit that is genuine content (Multiplier= 1forMarginalize-It Distilled;= 4.5/5.5 for End-Of-Token Distilled, whose sequences carry an extra<eot>per4.5content bytes).Data Savings=Token to Byte/Bytes is how much less unique text the byte method needs to reach the same performance. To calculate storage savings we use our storage cost calculations from Table10where we equate the storage cost of oneTokenwith 4.5Bytesto obtain k (for top-k).Storage Savings = (Tokens×4.5)/Byte Units compares the byte footprint ofToken Distilledapproach against the raw byte-unit count. MetricFormulaMarginalize-It DistilledEnd-Of-Token Distilled Training FLOPs Compute to achieveTokendistillation perf. (Table5)2.04×10 23 6.33×10 22 FPU bytes FLOPs per byte unit (Table9)8.81×10 9 9.44×10 9 FPU tokens FLOPs per token (Table9)8.18×10 9 8.18×10 9 Byte UnitsTraining FLOPs/FPU bytes 2.32×10 13 6.70×10 12 Multiplier(for actual data)14.5/5.5 BytesByte Units×Multiplier2.32×10 13 5.48×10 12 TokensTraining FLOPs/FPU tokens 2.50×10 13 7.73×10 12 Token to Byte Tokens×4.51.12×10 14 3.48×10 13 Data Savings Token to Byte/Bytes4.85×6.35× Storage Savings Tokens/(Byte Units/4.5)4.85×5.19× 7.3 Results 1.We summarize our results in Table7. DistilledEnd-Of-Token-1Btraining runs asymptotically out- performToken-1Bmodel, hence we can compute the crossover points ofBytes-1BandEnd-Of-Token-1B models withToken-1Bmodel. 2.SinceEnd-Of-Tokenuses the<eot>tokens every 4.5 bytes, the actual text data is less than the byte units used for training. Taking this into account,End-Of-TokenByte Logitscan reduce the storage cost to approximatelyone-fifthcompared to token distillation when trained for upto 6.7 Trillion byte units or data equivalent to 1.2 Trillion Llama 3-8B tokens. 18 End-Of-Token-1Bdemonstrates better asymptotic downstream performance compared to distilled Token-1B. It requiresone-sixthtext data and it can save storage costs uptoone-fifth. 8 Comparisons with Open-Weight models We now compare our asymptotic predictions for each of the six experiments described in section3.2with the open weight models:Llama 3.2-1Bmodel (Meta AI,2024),Gemma-3-1B-pt(Kamath et al.,2025), andGemma 2B(Team et al.,2024) in Table8. We evaluate these models using (Gao et al.,2024). Table 8: Predicted asymptotic downstream accuracy (%) in the for our LR=4e-3 obtained from table 15, obtained by extrapolating the scaling trend. Comparisons with the open-weights model with our asymptotic predictions reveal our distilledEnd-Of-Token-1Bmodel surpass theMeta AI(2024),Kamath et al.(2025), andTeam et al.(2024) models on averaged downstream tasks by upto6.5% and8.1%, 2.1% respectively. Additional details for llama and gemma models are provided in table 18 ModelMethodAsymptotic Avg. Accuracy(%) Token-1BToken Supervised46.0 Token-1BToken Distilled48.4 Bytes-1BBytes Supervised51.2 Bytes-1BMarginalize-It Distilled50.5 End-Of-Token-1BBytes w/ <eot> Supervised51.6 End-Of-Token-1BEnd-Of-Token Distilled52.4 Llama-3.2-1BMeta AI(2024)45.9 (≈9T BPE tokens) Gemma-3-1B-ptKamath et al.(2025)44.3 (≈2T BPE tokens) Gemma 2bTeam et al.(2024)50.3 (≈3T BPE tokens) 9 Related Work Scaling Laws with Overtraining.Which factors affect the power law exponent with overtraining has been a topic of great interest in deep learning.Kaplan et al.(2020) show that changing the architecture from transformers to LSTMs does not change the power law exponent for shorter contexts. (Bahri et al., 2024) show that the power law exponent changes with data distribution. (Henighan et al.,2020) show that the power law exponents change across modalities and also across image resolutions. ( Bahri et al., 2024;Bansal et al.,2022) show that the noise level in data can also significantly affect the exponents. In the context of language modeling (Hestness et al.,2017) show that character models learn relationships between characters with fewer samples compared to word level language models learn from words. Sev- eral recent works focus on the overtraining of language models. (Sardana et al.,2023) demonstrate that when the inference budget is large enough, it is better to train a small model for longer.Gadre et al. (2024) derive scaling laws with overtraining for different model sizes and training datasets.Limisiewicz et al.(2026) analyze the impact of data compression on scaling laws.Lee et al.(2026) show that the autoregressive byte models approach performance parity with their BPE counterparts as compute scales. Distinct from prior work, we study power laws for≈fixed model size with overtraining along two axes: training objective and the tokenization scheme. We observe that both axes affect the exponent as seen from table ( 11) and table (15). We show that a simple addition of an<eot>token every≈4.5 bytes can improve a model’s asymptotic downstream performance: an effect explained by the additional FLOPs spent for a fixed amount of data from the increased sequence length at fixed context. Scaling Laws for Downstream Tasks.Schaeffer et al.(2023) argue that the capabilities that ap- pear to be emergent in language models are due to the researcher’s choice of metric. In the context of transfer learning Isik et al.(2025) study how the choice of the pretraining data affects downstream cross entropy and Machine Translation performance (BLEU) score. Gadre et al.(2024) establish the relationship between the Average top-1 downstream error across a number of benchmarks and the vali- dation Cross-Entropy Loss. They demonstrate that using validation loss alone while comparing models 19 trained on different data distributions can be misleading.Owen(2024) show that the aggregate bench- mark performance is decently predictable, though predicting individual performance is harder.Bhagia et al.(2024) establish a two stage approach where they use N (# of parameters) and D (# of tokens) to predict the intermediate task loss which is then mapped to the task performance.Lourie et al.(2025) conduct a meta-analysis of the downstream tasks considered inGadre et al.(2024) to demonstrate that the task-specific scaling trends may vary. They further compare them with the findings fromMagnusson et al.(2025) to show that the scaling behavior depends on the experimental setup and that the choice pretraining corpus, validation corpus, or downstream task can affect the scaling laws. Our work studies how the training objective and the tokenization scheme can affect the downstream performance with Training FLOPs. Pretraining Distillation.Hinton et al.(2015) introduced distillation and showed that a small model can be improved by using the output probabilities of a large model.Busbridge et al.(2025) build scaling laws to determine the optimal compute allocation between student and the teacher.Peng et al.(2025) in- vestigate the effect of different design decisions in token to token distillation. Beyond language modeling Beyer et al.(2022) proposed the “patient and consistent teacher” hypothesis for ResNet-50 models (He et al.,2016) trained on Imagenet (Deng et al.,2009). In this work we study pretraining distillation for both token and byte models. Token to Bytes transfer.Minixhofer et al.(2025) present an approach for adapting pretrained token- level models to operate on raw bytes.Minixhofer et al.(2026) andBao et al.(2026) propose methods for distilling token models into byte models in the fine-tuning stage.Hayase et al.(2025) andPhan et al. (2024) study the exact BPE to bytes transfer to primarily tackle theprompt boundary problemwhere the prompt, when ends in the middle of a BPE token, results into distorted next token distributions at inference time. These methods can be used to convert token logits to byte logits. However they remain computationally expensive and require multiple inference passes over the teacher to obtain exact con- versions. We circumvent this problem by simply marginalizing the prefix-matched token distributions in Marginalize-Itmethod and introducing an additional<eot>token in the vocabulary and the sequences during training to preserve the distribution in theEnd-Of-Tokenmethod. Byte Level Models.Al-Rfou et al.(2018),Choe et al.(2019),El Boukkouri et al.(2020) laid the groundwork for byte level language modeling.J. H. Clark et al.(2022) introduce CANINE, an ar- chitecture with downsampling, followed by a deep transformer stack and upsampling layers for eﬀicient character level modeling. Xue et al.(2022) train an encoder-decoder model operating on UTF-8 bytes and show that byte level transformers are data eﬀicient learners, though take longer to train and are much ex- pensive during inference time.Yu et al.(2023) introduce the concept of patching to enable sub-quadratic self attention.Nawrot et al.(2023) propose a hierarchical architecture for byte level modeling jointly performs language modeling and token segmentation. Pagnoni et al.(2025) propose a new architecture with a local encoder and decoder that operates on patches of raw bytes.Hwang et al.(2026) propose dynamic chunking for end to end byte level sequence modeling. Very recently EvaByteZheng et al.(2025) trained byte level transformers with various tricks like multi-byte prediction and linearized attention. We study three decoder-only transformer variants (Vaswani et al.,2017) with vocabulary sizes of 128256 ( Token-1B), 260 (Bytes-1B), and 261 (End-Of-Token-1B). They differ in both parameter count (from the embedding layers) and FLOPs (by varying sequence lengths). By preserving the teacher distribution exactly while spending≈30.94%additional compute on a fixed amount of data, theEnd-Of-Token-1B model asymptotically outperforms the distilledToken-1B. We also note thatEnd-Of-Token-1Bmodel despite having fewer parameters can asymptotically surpass Token-1Bon downstream tasks. 10 Discussion Marginalize-It vs End Of Token.Our findings show thatMarginalize-It, despite being an ap- proximation asymptotically outperforms the Tokendistillation baseline.End-Of-Tokenoffers two ad- vantages: 1) it preserves the teacher distribution exactly, and 2) it improves asymptotic performance of the supervisedEnd-Of-Token-1Bbaseline itself at the cost of an extra≈30.94%compute per unit of data. Due to these two properties,End-Of-Tokendistillation asymptotically outperforms all of the other five models on downstream tasks. 20 The role of<eot>tokens on power laws.Our work shows that simply adding<eot>token every≈4.5 bytes on average affects the power law coeﬀicients3. We also observe that supervisedEnd-Of-Token-1B yields better asymptotic downstream performance compared to theBytes-1Bbaseline. This finding opens many exciting avenues for future work. Inference Costs.While our scaling laws show that, distilling Llama 3-8B (Grattafiori et al.,2024) model intoEnd-Of-Token-1Bmodels with 1.28B total parameters is asymptotically better than distill- ing intoToken-1Bmodels with 1.81B total parameters,End-Of-Token-1Bmodels remain significantly expensive at inference time. Comparing the asymptotic performance of a largerToken-1Bmodel against that of a smallerEnd-Of-Token-1Btransformers with matched inference cost could reveal whether dis- tilledEnd-Of-Token-1Btransformers are also eﬀicient at inference time; this is beyond the scope of this study. BPB vs. Benchmark Performance Discrepancy.Our observations reveal that BPB is not a metric that enables fair comparison across models with different training objectives and tokenization schemes. Many other works observe the same phenomenon for different optimizers (Zhang et al.,2026), data dis- tributions (Gadre et al.,2024), long-context setting, (Fang et al.,2025), byte modeling (Lee et al.,2026), and in theoretical analyses (Liu et al.,2023;Veličković et al.,2026). We identify additional settings in which both the training objective (Cross Entropy Supervised vs Distillation) and the tokenization scheme (Token,Bytes,Bytes w/ <eot>) can alter the downstream-task-performance vs. validation- BPB scaling trajectories, implying that a model with a lower BPB may not be necessarily a better model. 10.1 Future work Our scaling study focuses on dense transformer models. It would be interesting to explore how introducing a third axis of sparsity, by training Mixture-of-Experts models affects these power laws. Next, our study examines scaling trends across varying data budgets with approximately fixed model size of 1.28 billion layer parameters, ranging up to 1 trillion bytes. We also note that some of our larger training runs epoch. Future work should explore training with larger model sizes and more data. Next we report our findings based on eight benchmarks spanning three categories: Multiple Choice Question Answering, Language Generation, and Machine Translation. Future work should explore evaluating models on more benchmarks. Lastly, understanding the training dynamics of distilling large byte transformers into smaller ones remains an open question. Acknowledgments We thank Jonathan Hayase, Alisa Liu, and Benjamin Minixhofer for thoughtful discussions and feedback on this work. 21 Contents 1 Introduction1 1.1 Summary of Contributions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 2 Logit Conversion Methods4 2.1 Marginalize-It Logits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 2.2 End-Of-Token Logits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 3 Scaling Study Design6 3.1 Experimental Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 3.2 Scaling Along Two Axes: Tokenization and Training Objectives. . . . . . . . . . . . . . .6 3.3 Benchmark Evaluations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 4 Scaling Trends I: Validation BPB vs Training FLOPS7 4.1 Validation BPB vs Training FLOP Plots. . . . . . . . . . . . . . . . . . . . . . . . . . .7 5 Scaling Trends I: Task Performance vs Training FLOPs8 5.1 Results on Multiple Choice Question Answering Benchmarks. . . . . . . . . . . . . . . .9 5.2 Results on Language Generation Benchmarks. . . . . . . . . . . . . . . . . . . . . . . . .9 5.3 Results on Machine Translation Benchmarks. . . . . . . . . . . . . . . . . . . . . . . . .10 5.4 Concluding Remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 6 Scaling Trends I: Task Performance vs Validation BPB11 6.1 Results on Multiple Choice Question Answering Benchmarks. . . . . . . . . . . . . . . .12 6.2 Results on Language Generation Benchmarks. . . . . . . . . . . . . . . . . . . . . . . . .12 6.3 Results on Machine Translation Benchmarks. . . . . . . . . . . . . . . . . . . . . . . . .13 6.4 Fitting the Downstream Task Error vs Validation BPB Scaling Law. . . . . . . . . . . .14 6.5 Downstream Task Performance Upper Bound. . . . . . . . . . . . . . . . . . . . . . . . .15 6.6 When Lower BPB Does Not (Always) Imply Better Models. . . . . . . . . . . . . . . . .16 7 The Case forEnd-Of-TokenLogits16 7.1Feather Plot: Visualizing Iso-FLOP Point Connections. . . . . . . . . . . . . . . . . .17 7.2 Calculating Downstream Error Cross-Over Point with Token Distillation. . . . . . . . . .18 7.3 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 8 Comparisons with Open-Weight models19 9 Related Work19 10 Discussion20 10.1 Future work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 A Experimental Setup Details24 A.1 Model Architecture and FLOP Calculations.. . . . . . . . . . . . . . . . . . . . . . . . . .24 A.2 Pretraining Dataset, Tokenization, and Data Scales. . . . . . . . . . . . . . . . . . . . .24 A.3 Loss Function and Optimization Settings. . . . . . . . . . . . . . . . . . . . . . . . . . .24 A.4 Sequence Length and Global Batch Size.. . . . . . . . . . . . . . . . . . . . . . . . . . . .25 A.5 Bits Per Byte Calculation.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25 A.6 Storage Cost Calculations of Token and Byte Logits. . . . . . . . . . . . . . . . . . . . .25 B Additional Results using Validation Dataset 127 C Comparisons with open-weight models: Additional details37 D On Scaling Laws Sensitivity38 D.1 Leave One Out Error Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38 D.2 Residual Weighting Scheme Sensitivity Analysis. . . . . . . . . . . . . . . . . . . . . . .39 D.3 Choice of the validation data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40 22 E Additional Results using Validation Dataset 242 E.1 BPB vs Training FLOPs power law equations. . . . . . . . . . . . . . . . . . . . . . . . .42 E.2 Average top-1 Error Scaling Laws. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42 E.3Feather Plots: Comparisons using validation dataset 2. . . . . . . . . . . . . . . . . .44 E.4 Predicting the asymptotic Average Downstream task performance. . . . . . . . . . . . . .44 F Distillation Methods: Additional Details46 F.1Marginalize-ItandEnd-Of-TokenPseudocode. . . . . . . . . . . . . . . . . . . . . . .46 F.2 ComparingMarginalize-Itmethod withEnd-Of-Token. . . . . . . . . . . . . . . . .46 G On BPB vs Benchmark Performance Discrepancy49 G.1 Why the same BPB may imply different downstream performance?. . . . . . . . . . . . .49 23 A Experimental Setup Details In this section we provide additional details on the design decisions for our experiments. A.1 Model Architecture and FLOP Calculations. To train both the token and byte student models, we use the transformer (Vaswani et al.,2017) archi- tecture with 1 billion parameters. We use SwiGLU (Shazeer,2020) activation function.The differences in the number of parameters and training FLOPs arise from the differences in the vocabulary size – byte models use fewer parameters and FLOPs per byte. We use Llama3-8B (Grattafiori et al.,2024) as our teacher model. PropertyFormulaToken-1BBytes-1BEnd-Of-Token-1B Model Dimensiond204820482048 FFN dimensiond ffn =⌈4· 2 3 · d 256 ⌉·256563256325632 Number of Layersn252525 Number of Headsn heads 161616 Vocabulary sizeV128256260261 Attention Parameters per LayerN attn = 4·d 2 16777216 1677721616777216 FFN Parameters per LayerN ffn = 3·d·d ffn 34603008 3460300834603008 RMSNorm Parameters per LayerN norm = 2·d409640964096 Total Layer ParametersN layers =n·(N attn +N ffn +N norm )1284608000 12846080001284608000 Pre Output Norm ParametersN preoutputnorm =d204820482048 Total Non-Embedding ParametersN nonembed =N layers +N preoutputnorm +d·V1547278336 12851425281285144576 Total ParametersN total =N nonembed +d·V1809946624 12856750081285679104 Sequence Lengthl2048921611264 Self Attention FLOPsF attn = 4·n·(d//n heads )·n heads · l+1 2 209817600 9438208001153536000 Self Attention QKVO FLOPsF qkvo = (2 + 2)·2·n·d 2 838860800 838860800838860800 Feed Forward FLOPsF ffn = 2·n·2·d·d ffn 1153433600 11534336001153433600 De Embedding FLOPsF deemb = 2·d·|V|525336576 10649601069056 Total FLOPs per unitF total = 3·(F attn +F qkvo +F ffn +F deemb )8182345728 88115404809440698368 Table 9: Model Architecture Comparison A.2 Pretraining Dataset, Tokenization, and Data Scales We use the Llama-2 ( Touvron et al.,2023) pretraining mixture for all of our experiments. We tokenize the data using two tokenizers: BPE-Tiktoken (Llama3-8B) tokenizer and the byte tokenizer. Our byte tokenizer uses a vocabulary size of 260: 256 UTF-8 bytes and 4 special tokens. We train multiple 1B models with varying D where D is the number of tokens/byte units. A.3 Loss Function and Optimization Settings A typical knowledge distillation setup involves two components:αL KL , a loss term computed using the teacher’s logits and(1−α)L CE , the next token prediction loss computed against the ground truth data. Whenα= 0, the objective reduces to supervised training (Cross-Entropy loss with respect to data) and whenα= 1, the model is trained purely via teacher distillation. We use equation 3as our loss function and useα, the mixing coeﬀicient to control our supervised and distillation runs. L KD =α·L KL + (1−α)·L CE (3) L KL = 1 n n ∑ t=1 ∑ c p T (c|x <t )log p T (c|x <t ) p S (c|x <t ) (4) L CE =− 1 n n ∑ t=1 ∑ c y t,c logp S (c|x <t )(5) where: •tis the position index in the sequence 24 •nis the total number of tokens in the sequence •cis the token index over the vocabularyV •y t,c ∈0,1is a one-hot indicator that the ground-truth token at positiontisc •p T (c|x <t )is the teacher model’s predicted probability distribution over the next token given all preceding tokensx <t •p S (c|x <t )is the student model’s predicted probability distribution over the next token given all preceding tokensx <t •α∈[0,1]is a weighting factor balancing the KL divergence and cross-entropy losses We train our models using the AdamW(Loshchilov & Hutter,2017) optimizer withβ 1 = 0.9andβ 2 = 0.95 and a cosine learning rate schedule. We set warmup to 10% of the maximum number of steps for each data scale use and three peak learning rates∈1e-3, 4e-3, 8e-3. We use independent weight decay (Kosson et al.,2025) of 1e-4 and a grad clip norm of 1.0. A.4 Sequence Length and Global Batch Size. We set sequence length to 2048 for ourToken-1Bmodels. We train models using 64 H200s per training run and use a global batch size is 0.5M tokens. For training byte transformers, our initial explorations with the exact same context and data (8K×4×64 = 2M bytes) per optimization step resulted in loss spikes. To stabilize the runs we use qk-norm and set the sequence length to 9216 forBytes-1Band 11264 forEnd-Of-Token-1B, and maintain the global batch size as that of the token models. If the byte sequence formed by the 2048 tokens exceeds the byte sequence length, we truncate it; if the sequence is shorter, we pad it with padding tokens to preserve the context length. We use 64 H200s for each Bytes-1BandEnd-Of-Token-1Bas well, maintaining the global batch size of≈0.5M byte units as that of the token models. Hence, overall our byte models use much less data compared to token based models for the same compute budget. A.5 Bits Per Byte Calculation. We use BPB (Bits per Byte) (equation 6), a smooth tokenizer independent metric on a held out validation set to plot the scaling curves. The BPB is defined as the total cross-entropy lossL CE summed over the validation dataxdivided by the total number of bytesBinxand scaled by a constant. Intuitively, it captures the average number of bits a model uses to represent the data. BPB= L CE (x)·log 2 e B (6) Finally, We use Llama 3-8B ( Grattafiori et al.,2024) and Llama3.2-1B (Meta AI,2024) models as ref- erence points to compare with our token and byte asymptotes. Since these models are overtrained on several trillion tokens, they simulate the infinite data regimes. A.6 Storage Cost Calculations of Token and Byte Logits The biggest pain point of using offline distillation to train small language models is the storage cost of the teacher logits. Each BPE token of a Llama3-8B model, for instance, produces 128256 logit values. A billion parameter model overtrained to 2T tokens, for instance, would require and2T×4×128256bytes = 1.026048 Exabytes assumingfloat32datatype. To circumvent this storage problem, researchers store only thetop-klogits per token, wherektypically is of the order of several hundreds. On the contrary, byte transformers use a fixed vocabulary of only ≈ 260 bytes, trading vocabulary size for increased sequence length. Our controlled study is designed to characterize this tradeoff. In table 10we detail the storage cost of storing one BPE token and theirtop-klogits. Note that for BPE tokens, we use the sparse representation of logits while for bytes logits we store the full distribution. By equating the total cost of BPE-tiktoken with bytes, we solve fork= 4685/8, which turns out to be 25 Table 10: Storage cost analysis for top-k logits per token. We solve for k and found it to be≈600 Token type Data AttributeDatatype Bytes per value # Values Total Cost BPE-tiktoken Token Logit valuesfloat324k 4k+ 4k+ 4Token Logit indicesint324k Tokensint3241 Bytes Byte Logitsfloat324260×4.5 4680 + 9 Bytesint1624.5 585.625≈600. Therefore for our controlled comparison we store 600 logits per BPE-tiktoken token and all of the byte logits i.e. 260 values for distillation. 26 B Additional Results using Validation Dataset 1 In this section we provide additional results for the learning rates of 1e-3 an 8e-3 for validation dataset 1 (which is used for analysis in the paper). 10 20 10 21 10 22 10 23 0.900 0.925 0.950 0.975 1.000 1.025 1.050 1.075 Bits per Byte (BPB) LR = 1e-3 Token Supervised (c=0.9559) Token Distilled (c=0.9452) Byte Supervised (c=0.9124) Marginalize-It Distilled (c=0.9092) Bytes w/ <eot> Supervised (c=0.9074) End-Of-Token Distilled (c=0.9074) 10 20 10 21 10 22 10 23 0.900 0.925 0.950 0.975 1.000 1.025 1.050 1.075 Bits per Byte (BPB) LR = 4e-3 Token Supervised (c=0.9568) Token Distilled (c=0.9407) Byte Supervised (c=0.8967) Marginalize-It Distilled (c=0.9016) Bytes w/ <eot> Supervised (c=0.8891) End-Of-Token Distilled (c=0.8983) 10 20 10 21 10 22 10 23 Training FLOPs 0.900 0.925 0.950 0.975 1.000 1.025 1.050 1.075 1.100 Bits per Byte (BPB) LR = 8e-3 Token Supervised (c=0.9575) Token Distilled (c=0.9441) Byte Supervised (c=0.9080) Marginalize-It Distilled (c=0.9133) Bytes w/ <eot> Supervised (c=0.9165) End-Of-Token Distilled (c=0.9073) Data Scales 20B40B68B102B149B203B298B678B1.2T 16B33B55B82B121B165B242B550B968B 15B30B50B110B220B500B Bytes w/ <eot> Bytes Tokens 220B Token ScaleInterpolationExtrapolation Figure 12: The figure shows our scaling curves for different data scales and learning rates for a fixed parameter count of approximately 1 billion parameter transformer models(see Table 9for details). The distilled models are trained using Llama 3-8B as a teacher model. The validation BPB of the teacher model is 0.85511. 27 Table 11: Power-law equationsy=b·x a +cfitted to BPB vs. training FLOPs. LRScenarioEquation 1×10 −3 Token Supervisedy= 3.013×10 8 ·x −0.4724 + 0.9559 Token Distilledy= 5.677×10 8 ·x −0.4911 + 0.9452 Bytes Supervisedy= 7.253×10 7 ·x −0.4274 + 0.9124 Marginalize-It Distilledy= 1.503×10 8 ·x −0.4449 + 0.9092 Bytes w/ <eot> Supervisedy= 6.329×10 7 ·x −0.4210 + 0.9074 End-Of-Token Distilledy= 3.383×10 8 ·x −0.4593 + 0.9074 4×10 −3 Token Supervisedy= 1.075×10 9 ·x −0.5004 + 0.9568 Token Distilledy= 2.803×10 8 ·x −0.4754 + 0.9407 Bytes Supervisedy= 4.467×10 6 ·x −0.3658 + 0.8967 Marginalize-It Distilledy= 2.706×10 7 ·x −0.4074 + 0.9016 Bytes w/ <eot> Supervisedy= 3.080×10 6 ·x −0.3546 + 0.8891 End-Of-Token Distilledy= 2.734×10 7 ·x −0.4047 + 0.8983 8×10 −3 Token Supervisedy= 2.927×10 9 ·x −0.5214 + 0.9575 Token Distilledy= 3.692×10 9 ·x −0.5314 + 0.9441 Bytes Supervisedy= 8.176×10 7 ·x −0.4289 + 0.9080 Marginalize-It Distilledy= 2.271×10 9 ·x −0.5028 + 0.9133 Bytes w/ <eot> Supervisedy= 3.627×10 9 ·x −0.5068 + 0.9165 End-Of-Token Distilledy= 6.619×10 8 ·x −0.4729 + 0.9073 Table 12: We demonstrate that the amount of FLOPs spent on a fixed amount of data (one BPE token in this case) affects the asymptotic BPB. Overall:Bytes w/ <eot> SupervisedBPB<Bytes Supervised BPB<Token SupervisedBPB. andEnd-Of-Token DistilledBPB<Marginalize-It Distilled) BPB<Token DistilledBPB. We also note that for all learning rates, within the groups (based on training objective), the supervised and distilled models exhibit small differences in the asymp- totic value. The FLOPs/Unit values are obtained from Table9. MethodFLOPs / Unit Units Total FLOPs Asymptotic BPB Per Token 1e-3 4e-3 8e-3 Token Supervised8.18×10 9 1.08.18×10 9 0.9559 0.9568 0.9575 Bytes Supervised8.81×10 9 4.539.65×10 9 0.9124 0.8967 0.9080 Bytes w/ <eot> Supervised9.44×10 9 5.551.92×10 9 0.9074 0.8891 0.9165 Token Distilled8.18×10 9 1.08.18×10 9 0.9452 0.9407 0.9441 Marginalize-It Distilled8.81×10 9 4.539.65×10 9 0.9092 0.9016 0.9133 End-Of-Token Distilled9.44×10 9 5.551.92×10 9 0.9074 0.8983 0.9073 Table 13: Scaling-law goodness of fit. Each entry is the coeﬀicient of determination R 2 = 1 − S res / S tot for the fity=b·x a +cof BPB versus training FLOPs. ScenarioR 2 (LR=1e-3)R 2 (LR=4e-3)R 2 (LR=8e-3) Token Supervised0.99990.99991.0000 Token Distilled0.99961.00000.9997 Bytes Supervised0.99990.99800.9996 Marginalize-It Distilled0.99740.99890.9985 Bytes w/ <eot> Supervised0.99820.99950.9984 End-Of-Token Distilled0.99800.99900.9987 28 10 20 10 21 10 22 40 45 50 55 60 Choice Average Accuracy (%) LR=1e-3 10 20 10 21 10 22 40 45 50 55 60 LR=4e-3 10 20 10 21 10 22 40 45 50 55 60 LR=8e-3 10 20 10 21 10 22 25 30 35 ARC-Challenge Accuracy (%) 10 20 10 21 10 22 25 30 35 10 20 10 21 10 22 25 30 35 10 20 10 21 10 22 50 60 70 ARC-Easy Accuracy (%) 10 20 10 21 10 22 50 60 70 10 20 10 21 10 22 50 60 70 10 20 10 21 10 22 40 50 60 Hellaswag Accuracy (%) 10 20 10 21 10 22 40 50 60 10 20 10 21 10 22 40 50 60 10 20 10 21 10 22 Training FLOPs (log scale) 65 70 75 PIQA Accuracy (%) 10 20 10 21 10 22 Training FLOPs (log scale) 65 70 75 10 20 10 21 10 22 Training FLOPs (log scale) 65 70 75 Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled 220B Token Supervised Bytes w/ <eot> Supervised End-Of-Token Distilled Figure 13: We plot Accuracy vs training FLOP curves for all the six scenarios from section3.2. For all of the Multiple Choice Question Answering Benchmarks, we observe that the token models show better performance at low compute budgets and eventually plateau. Byte models, start worse yet improve at a much faster rate. 29 10 20 10 21 10 22 0 5 10 Generation Average Accuracy (%) LR=1e-3 10 20 10 21 10 22 0 5 10 LR=4e-3 10 20 10 21 10 22 0 5 10 LR=8e-3 10 20 10 21 10 22 0 5 10 MBPP (Pass@1) (%) 10 20 10 21 10 22 0 5 10 10 20 10 21 10 22 0 5 10 10 20 10 21 10 22 Training FLOPs (log scale) 0 5 10 NQ Exact Match (%) 10 20 10 21 10 22 Training FLOPs (log scale) 0 5 10 10 20 10 21 10 22 Training FLOPs (log scale) 0 5 10 Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled 220B Token Supervised Bytes w/ <eot> Supervised End-Of-Token Distilled Figure 14: We visualize % Accuracy vs training FLOPs curves for language generation tasks. On gener- ative tasks, token models achieve better performance at lower compute budgets and appear to plateau with the training FLOPs.Bytes-1BandEnd-Of-Token-1Bmodels show a different scaling behavior, with much worse performance at a lower compute budget which continues to improve with FLOPs with no sign of saturation. 30 10 20 10 21 10 22 0 10 20 30 MT Average BLEU LR=1e-3 10 20 10 21 10 22 0 10 20 30 LR=4e-3 10 20 10 21 10 22 0 10 20 30 LR=8e-3 10 20 10 21 10 22 0 5 10 15 20 English --> German BLEU 10 20 10 21 10 22 0 5 10 15 20 10 20 10 21 10 22 0 5 10 15 20 10 20 10 21 10 22 Training FLOPs (log scale) 0 10 20 30 German --> English BLEU 10 20 10 21 10 22 Training FLOPs (log scale) 0 10 20 30 10 20 10 21 10 22 Training FLOPs (log scale) 0 10 20 30 Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled 220B Token Supervised Bytes w/ <eot> Supervised End-Of-Token Distilled Figure 15: We plot BLEU score for the translation tasks for different training budgets. We observe clearly different scaling behaviors for token and byte models. Token models can achieve significantly better performance at a smaller compute budgets. The BLEU score appears to saturate with overtraining. Byte models both with and without the <eot> demonstrate extremely low BLEU scores at lower compute budgets yet the rate of improvement with overtraining is better compared to token models. 31 0.961.021.081.14 0.40 0.50 0.60 Choice Average Error (1 - % Accuracy) LR=1e-3 0.961.021.081.14 0.40 0.50 0.60 LR=4e-3 0.961.021.081.14 0.40 0.50 0.60 LR=8e-3 0.961.021.081.14 0.70 0.80 ARC-Challenge Error (1 - % Accuracy) 0.961.021.081.14 0.70 0.80 0.961.021.081.14 0.70 0.80 0.961.021.081.14 0.35 0.40 0.50 0.60 ARC-Easy Error (1 - % Accuracy) 0.961.021.081.14 0.35 0.40 0.50 0.60 0.961.021.081.14 0.35 0.40 0.50 0.60 0.961.021.081.14 0.35 0.40 0.50 0.60 0.70 Hellaswag Error (1 - % Accuracy) 0.961.021.081.14 0.35 0.40 0.50 0.60 0.70 0.961.021.081.14 0.35 0.40 0.50 0.60 0.70 0.961.021.081.14 Validation BPB (descending -->) 0.25 0.30 0.35 0.40 PIQA Error (1 - % Accuracy) 0.961.021.081.14 Validation BPB (descending -->) 0.25 0.30 0.35 0.40 0.961.021.081.14 Validation BPB (descending -->) 0.25 0.30 0.35 0.40 Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled Figure 16: We visualize Average-top-1 error vs the validation BPB curves for the six scenarios for differ- ent learning rates and Multiple Choice Question Answering benchmarks. All of these curves including the averaged metric appear to follow an exponential decay. HellaSwag demonstrates smoothest scaling behavior. 32 0.961.021.081.14 0.88 0.90 0.92 0.94 0.96 0.98 1.00 Generation Average Error (1 - % Accuracy) LR=1e-3 0.961.021.081.14 0.88 0.90 0.92 0.94 0.96 0.98 1.00 LR=4e-3 0.961.021.081.14 0.88 0.90 0.92 0.94 0.96 0.98 1.00 LR=8e-3 0.961.021.081.14 0.88 0.90 0.92 0.94 0.96 0.98 1.00 MBPP (Pass@1) Error (1 - % Accuracy) 0.961.021.081.14 0.88 0.90 0.92 0.94 0.96 0.98 1.00 0.961.021.081.14 0.88 0.90 0.92 0.94 0.96 0.98 1.00 0.961.021.081.14 Validation BPB (descending -->) 0.88 0.90 0.92 0.94 0.96 0.98 1.00 NQ Error (1 - % Accuracy) 0.961.021.081.14 Validation BPB (descending -->) 0.88 0.90 0.92 0.94 0.96 0.98 1.00 0.961.021.081.14 Validation BPB (descending -->) 0.88 0.90 0.92 0.94 0.96 0.98 1.00 Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled Figure 17: We plot the Average-top-1 error vs BPB curves for the MBPP and the NQ benchmarks. We notice an exponential decay for all three scenarios, with significantly different trends for the token and byte models. MBPP shows noisier behavior compared to NQ. We deduce that the same validation Bits-Per-Byte imply different downstream performace for different training objectives(distillation vs cross- entropy) tokenization schemes(tokens vs bytes). Distilled models as well demonstrate slightly different scaling behavior compared to the supervised counterparts. 33 0.961.021.081.14 0 6 12 18 24 30 MT Average BLEU LR=1e-3 0.961.021.081.14 0 6 12 18 24 30 LR=4e-3 0.961.021.081.14 0 6 12 18 24 30 LR=8e-3 0.961.021.081.14 0 5 10 15 20 English -> German BLEU 0.961.021.081.14 0 5 10 15 20 0.961.021.081.14 0 5 10 15 20 0.961.021.081.14 Validation BPB (descending -->) 0 8 16 24 32 German -> English BLEU 0.961.021.081.14 Validation BPB (descending -->) 0 8 16 24 32 0.961.021.081.14 Validation BPB (descending -->) 0 8 16 24 32 Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled Figure 18: We plot BLEU score vs Validation BPB performance for English to German translation, German to English translation, and on the averaged BLEU performance for the six scenarios for three learning rates. We notice the differences in the scaling behavior across tokenization schemes as well as the training objectives. In some cases distillation appears to exhibit monotonic trend compared to the noisy supervised curves especially for English to German translation. 34 0.880.961.041.12 Validation BPB (descending -->) 0.46 0.50 0.60 0.70 Average Error (1 - % Accuracy) LR=1e-3 0.880.961.041.12 Validation BPB (descending -->) 0.46 0.50 0.60 0.70 LR=4e-3 0.880.961.041.12 Validation BPB (descending -->) 0.46 0.50 0.60 0.70 LR=8e-3 0.880.961.041.12 Validation BPB (descending -->) 0.46 0.50 0.60 0.70 Average Error (1 - % Accuracy) LR=1e-3 0.880.961.041.12 Validation BPB (descending -->) 0.46 0.50 0.60 0.70 LR=4e-3 0.880.961.041.12 Validation BPB (descending -->) 0.46 0.50 0.60 0.70 LR=8e-3 Downstream Error: Scaling Laws Downstream Error: Scaling Laws (curves only) Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled Bytes + EoT Supervised End-Of-Token Distilled Asymptotic point (BPB, Err) Figure 19: Downstream Error Scaling laws Table 14: Fitted downstream error scaling laws z ( y ) = ε − k · e −γy (equation 2) for each model variant and learning rate, whereyis the validation BPB. ModelLR= 1×10 −3 LR= 4×10 −3 LR= 8×10 −3 Token Supervised0.9239−7.3780e −3.1423y 0.7501−934.3212e −8.7782y 0.7600−492.4601e −8.0652y Token Distilled0.7430−398.4377e −7.9888y 0.7383−1292.0124e −9.2132y 1.0076−5.6152e −2.6394y Bytes Supervised0.7726−1950.1906e −9.7822y 0.7686−1231.1241e −9.3532y 0.7872−367.2256e −7.9602y Marginalize-It Distilled0.7805−958.5881e −8.9880y 0.7721−1137.3004e −9.2265y 0.7825−454.9215e −8.2080y Bytes w/ <eot> Supervised0.7947−267.8536e − 7 . 5664 y 0.7972−185.6303e − 7 . 1822 y 0.7776−650.0335e − 8 . 5998 y End-Of-Token Distilled0.7915−378.9535e −7.9129y 0.7776−827.6076e −8.8125y 0.7799−773.4793e −8.7411y 35 Table 15: Asymptotic validation BPB (c) and the corresponding downstream average accuracya ⋆ = 1−%z ⋆ wherez ⋆ is the Asymptotic Average top-1 error predicted by the scaling law equation:2, z ⋆ =ε−k·e −γc , wherecis obtained from the power law equations in Table4for each model variant and learning rate. LR= 1×10 −3 LR= 4×10 −3 LR= 8×10 −3 Methodca ⋆ (%) ca ⋆ (%) ca ⋆ (%) Token Supervised0.9559 44.2 0.9568 46.0 0.9575 45.8 Token Distilled0.9452 46.6 0.9407 48.4 0.9441 45.7 Bytes Supervised0.9124 48.7 0.8967 51.2 0.9080 47.9 Marginalize-It Distilled0.9092 49.0 0.9016 50.5 0.9133 47.0 Bytes w/ <eot> Supervised0.9074 48.5 0.8891 51.6 0.9165 46.8 End-Of-Token Distilled0.9074 49.7 0.8983 52.4 0.9073 49.8 0.880.961.041.12 Validation BPB (descending -->) 0.50 0.60 0.70 Average Error (1 - % Accuracy) LR=1e-3 0.880.961.041.12 Validation BPB (descending -->) 0.50 0.60 0.70 LR=4e-3 0.880.961.041.12 Validation BPB (descending -->) 0.50 0.60 0.70 LR=8e-3 10 20 10 22 10 24 10 26 10 28 10 30 Training FLOPs (isoFLOP lines) Downstream Error: IsoFLOP Lines Token DistilledEnd-Of-Token DistilledMarginalize-It Distilled Asymptotic point (BPB, Err)Crossover point: reaches Token Distilled asymptotic error Figure 20: Feather plots for three learning rates on validation dataset 1 36 Table 16: Training compute at whichMarginalize-It DistilledandEnd-Of-Token Distilled match the asymptotic downstream performance of Token Distilled. We take Token Distilledś asymptotic BPB (thecterm of its fitted BPB–FLOPs power law; columnToken Dist. BPB) and map it through Token Distilledś benchmark scaling law to obtain its asymptotic downstream accuracy (columnAccu- racy). For each byte method we then invert its own benchmark scaling law at that accuracy to get the BPB it must attain (columnMethod BPB), and invert its BPB–FLOPs power law at that BPB to get the required training compute (columnTraining FLOPs). The accuracy column is shared within each learning rate because both methods are evaluated against the same Token Distilled asymptote. LRMethodToken Dist. BPB Accuracy (%) Method BPB Method Training FLOPs 10 −3 Marginalize-It Distilled0.945246.6%0.91957.05×10 22 End-Of-Token Distilled0.945246.6%0.92163.91×10 22 4×10 −3 Marginalize-It Distilled0.940748.4%0.91022.04×10 23 End-Of-Token Distilled0.940748.4%0.91456.33×10 22 8 × 10 −3 Marginalize-It Distilled0.944145.7%0.91979.27×10 22 End-Of-Token Distilled0.944145.7%0.92562.13×10 22 Table 17: Effective training-data volume at the compute budget required to achieve theTokendistillation asymptotic performance from Table5. Here FPU (FLOPs Per Unit) is the training FLOPs consumed per sequence unit, where a unit is one token forToken Distilledand one byte for the byte-level methods; it is obtained from Table9.Byte Units=FLOPs/FPU bytes is the number of byte units the method processes at its crossover compute.Bytes=Byte Units×Multiplier converts byte units into actual UTF-8 bytes of unique text, accounting for the fraction of each unit that is genuine content (Multiplier = 1for Marginalize-It Distilled;= 4.5/5.5forEnd-Of-Token Distilled, whose sequences carry an extra<eot>per4.5content bytes).Tokens=FLOPs/FPU tokens is the token countToken Distilled would process at the same compute.Token to Byte=Tokens×4.5expresses that token count in bytes using the average of4.5bytes per token.Data Savings=Token to Byte/Bytes is how much less unique text the byte method needs to reach the same performance. To calculate storage savings we use our storage cost calculations from Table10where we equate the storage cost of oneTokenwith 4.5Bytes to obtain k (for top-k).Storage Savings= (Tokens×4.5)/Byte Units compares the byte footprint of Token Distilledapproach against the raw byte-unit count. LRMethodTraining FLOPs FPU bytes FPU tokens Byte Units Bytes Tokens Token to Byte Data Savings Storage Savings 10 −3 Marginalize-It Distilled7.05×10 22 8.81×10 9 8.18×10 9 8.00×10 12 8.00×10 12 8.61×10 12 3.87×10 13 4.85×4.85× End-Of-Token Distilled3.91×10 22 9.44×10 9 8.18×10 9 4.14×10 12 3.39×10 12 4.77×10 12 2.15×10 13 6.35×5.19× 4×10 −3 Marginalize-It Distilled2.04×10 23 8.81×10 9 8.18×10 9 2.32×10 13 2.32×10 13 2.50×10 13 1.12×10 14 4.85×4.85× End-Of-Token Distilled6.33×10 22 9.44×10 9 8.18×10 9 6.70×10 12 5.48×10 12 7.73×10 12 3.48×10 13 6.35×5.19× 8×10 −3 Marginalize-It Distilled9.27×10 22 8.81×10 9 8.18×10 9 1.05×10 13 1.05×10 13 1.13×10 13 5.10×10 13 4.85×4.85× End-Of-Token Distilled2.13×10 22 9.44×10 9 8.18×10 9 2.25×10 12 1.84×10 12 2.60×10 12 1.17×10 13 6.35×5.19× C Comparisons with open-weight models: Additional details Table 18: Downstream benchmark accuracy (%) for the4e−3learning-rate runs at their maximum available data scale, compared against the Llama and Gemma 1B baselines. Avg. / Asympt. Avg. is the mean over the six benchmarks. ModelData Scale HellaSwag PIQA ARC-Easy ARC-Challenge MBPP NQ Avg. Llama-3.2-1B9T64.375.366.136.627.0 5.7 45.9 Gemma-3-1B-pt2T62.174.772.238.49.0 9.5 44.3 Gemma 2b3T71.44 78.07 72.1841.8928.0 10.53 50.3 Token Supervised500B63.474.565.136.18.6 10.2 43.0 Token Distilled220B64.275.766.436.310.6 13.0 44.4 Bytes Supervised880B63.973.162.633.87.0 10.1 41.8 Marginalize-It Distilled880B64.974.563.636.69.4 10.0 43.2 Bytes w/ <eot> Supervised880B65.672.066.335.15.6 11.1 42.6 End-Of-Token Distilled880B67.171.768.336.911.2 12.7 44.6 37 D On Scaling Laws Sensitivity We now characterize the sensitivity of downstream error scaling laws. D.1 Leave One Out Error Analysis We analyze the Leave-One-Out sensitivity of the power laws. 0.900.910.920.930.940.950.960.970.98 Irreducible-loss asymptote c (BPB) Token Supervised Token Distilled Byte Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled c=0.0048 c=0.0151 c=0.0028 c=0.0213 c=0.0185 c=0.0167 LR = 1e-3 0.880.900.920.940.960.98 Irreducible-loss asymptote c (BPB) Token Supervised Token Distilled Byte Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled c=0.0050 c=0.0025 c=0.0181 c=0.0129 c=0.0065 c=0.0098 LR = 4e-3 0.900.910.920.930.940.950.960.970.98 Irreducible-loss asymptote c (BPB) Token Supervised Token Distilled Byte Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled c=0.0039 c=0.0100 c=0.0069 c=0.0112 c=0.0188 c=0.0125 LR = 8e-3 Fit on all pointsEach leave-one-out fitLOO range of c (supervised)LOO range of c (distilled) Figure 21: We visualize the sensitivity of the scaling law fits via leave-one-out analysis. For each learning rate and for each experiment, we drop one of the datapoints and fit the power law using the rest and visualize the range of c. Tokencurves on average demonstrate little spread compared toBytesand Bytes w/ <eot>methods. 38 D.2 Residual Weighting Scheme Sensitivity Analysis 0.910.920.930.940.950.960.97 Irreducible-loss asymptote c (BPB) Token Supervised Token Distilled Byte Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled c=0.0031 c=0.0067 c=0.0004 c=0.0137 c=0.0181 c=0.0108 LR = 1e-3 0.890.900.910.920.930.940.950.960.97 Irreducible-loss asymptote c (BPB) Token Supervised Token Distilled Byte Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled c=0.0034 c=0.0004 c=0.0240 c=0.0180 c=0.0130 c=0.0036 LR = 4e-3 0.880.900.920.940.96 Irreducible-loss asymptote c (BPB) Token Supervised Token Distilled Byte Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled c=0.0014 c=0.0042 c=0.0067 c=0.0052 c=0.0376 c=0.0093 LR = 8e-3 unweighted weighted_y inv_xc range (supervised)c range (distilled) Figure 22: We visualize the sensitivity of the residual weighting scheme on the power laws. Three weighting schemes, each a choice of theσvector incurve_fit’s objective ∑ i ( (ˆy(x i )−y i )/σ i ) 2 : •unweighted: ordinary least squares. •weighted y: minimizes relative error(ˆy−y)/y, down-weighting high-BPB low-compute runs. •inverse x: shrinksσfor large-compute points so they dominate the extrapolation to high compute points. We useunweightedmethod for all of our plots. We note that other weighting schemes are more sensitive to BytesandBytes w/ <eot>tokenizations. 39 D.3 Choice of the validation data We observe that the choice of the validation dataset can alter the cross-over points of the training curves and as a result the perception of a better model. Consider for example in Figure23. In Validation Dataset 2, the token curves have not intersected the byte curves yet, while in Validation Dataset 1 they cross them quickly. The nature of theFeather Plotsalso change with changes in the validation data. Moreover, the amount of overtraining required to reliably predict the asymptotic behavior could also change with change in the validation dataset. Hence, taking performance on multiple validation sets into account, when fitting scaling laws might be helpful in predicting the final model performance. 10 20 10 21 10 22 10 23 0.900 0.925 0.950 0.975 1.000 1.025 1.050 1.075 Bits per Byte (BPB) Validation Dataset 1 Token Supervised (c=0.9568) Token Distilled (c=0.9407) Byte Supervised (c=0.8967) Marginalize-It Distilled (c=0.9016) Bytes w/ <eot> Supervised (c=0.8891) End-Of-Token Distilled (c=0.8983) 10 20 10 21 10 22 10 23 Training FLOPs 0.70 0.75 0.80 0.85 0.90 Bits per Byte (BPB) Validation Dataset 2 Token Supervised (c=0.7263) Token Distilled (c=0.7272) Byte Supervised (c=0.7038) Marginalize-It Distilled (c=0.7155) Bytes w/ <eot> Supervised (c=0.7057) End-Of-Token Distilled (c=0.7142) 220B Token ScaleInterpolationExtrapolation Figure 23: We demonstrate how changes in the validation dataset can affect the validation BPB vs training FLOPs scaling plots by comparing power laws for LR=4e-3. At the 220B Token Supervised FLOP budget, all of the other five models have a lower BPB for validation 1. On the other hand, for validation dataset 2, only the distilled token model has a lower BPB. The two datasets also have different asymptotic Bits-Per-Byte values for all six curves. 40 10 20 10 21 10 22 10 23 0.75 0.80 0.85 0.90 Bits per Byte (BPB) LR = 1e-3 Token Supervised (c=0.7456) Token Distilled (c=0.7119) Byte Supervised (c=0.7258) Marginalize-It Distilled (c=0.7214) Bytes w/ <eot> Supervised (c=0.7180) End-Of-Token Distilled (c=0.7223) 10 20 10 21 10 22 10 23 0.70 0.75 0.80 0.85 0.90 Bits per Byte (BPB) LR = 4e-3 Token Supervised (c=0.7263) Token Distilled (c=0.7272) Byte Supervised (c=0.7038) Marginalize-It Distilled (c=0.7155) Bytes w/ <eot> Supervised (c=0.7057) End-Of-Token Distilled (c=0.7142) 10 20 10 21 10 22 10 23 Training FLOPs 0.75 0.80 0.85 0.90 0.95 Bits per Byte (BPB) LR = 8e-3 Token Supervised (c=0.7267) Token Distilled (c=0.7316) Byte Supervised (c=0.7157) Marginalize-It Distilled (c=0.7264) Bytes w/ <eot> Supervised (c=0.7253) End-Of-Token Distilled (c=0.7181) Data Scales 20B40B68B102B149B203B298B678B1.2T 16B33B55B82B121B165B242B550B968B 15B30B50B110B220B500B Bytes w/ <eot> Bytes Tokens 220B Token ScaleInterpolationExtrapolation Figure 24: BPB vs FLOPs for validation dataset 2. 41 E Additional Results using Validation Dataset 2 E.1 BPB vs Training FLOPs power law equations Table 19: Fitted BPB-vs-FLOPs power laws of the formy=bx a +c, wherexis training FLOPs andy is bits-per-byte on validation dataset 2. LRScenarioEquation 1×10 −3 Token Supervisedy= 9.993×10 9 x −0.5514 + 0.7456 Token Distilledy= 3.251×10 5 x −0.3261 + 0.7119 Bytes Supervisedy= 7.753×10 8 x −0.4759 + 0.7258 Marginalize-It Distilledy= 1.036×10 9 x −0.4842 + 0.7214 Bytes w/ <eot> Supervisedy= 3.312×10 8 x −0.4541 + 0.7180 End-Of-Token Distilledy= 6.752×10 9 x −0.5218 + 0.7223 4×10 −3 Token Supervisedy = 3 . 404 × 10 6 x −0.3748 + 0 . 7263 Token Distilledy= 7.849×10 7 x −0.4494 + 0.7272 Bytes Supervisedy= 1.629×10 7 x −0.3907 + 0.7038 Marginalize-It Distilledy= 2.437×10 8 x −0.4526 + 0.7155 Bytes w/ <eot> Supervisedy= 3.929×10 7 x −0.4073 + 0.7057 End-Of-Token Distilledy= 5.648×10 8 x −0.4679 + 0.7142 8×10 −3 Token Supervisedy= 2.413×10 7 x −0.4163 + 0.7267 Token Distilledy= 1.237×10 9 x −0.5097 + 0.7316 Bytes Supervisedy= 2.294×10 8 x −0.4480 + 0.7157 Marginalize-It Distilledy= 1.520×10 10 x −0.5414 + 0.7264 Bytes w/ <eot> Supervisedy= 1.336×10 10 x −0.5318 + 0.7253 End-Of-Token Distilledy= 3.570×10 9 x −0.5068 + 0.7181 E.2 Average top-1 Error Scaling Laws Table 20: Fitted downstream error scaling lawsz(y) =ε−k·e −γy for each model variant and learning rate, whereyis the validation BPB calibrated using Validation dataset 2 BPB values. ModelLR= 1×10 −3 LR= 4×10 −3 LR= 8×10 −3 Token Supervised0.8410−9.8211e −4.7851y 0.7793−55.2335e −7.3843y 0.7780−50.7658e −7.2916y Token Distilled0.8419−11.1499e −4.9268y 0.7521−193.7935e −9.2079y 0.8640−8.9141e −4.5266y Bytes Supervised0.7635−216.7748e −9.3897y 0.7641−103.2172e −8.4542y 0.7814−45.6536e −7.2404y Marginalize-It Distilled0.7709−129.1220e −8.6465y 0.7705−95.2408e −8.2516y 0.7723−84.2661e −8.0843y Bytes w/ <eot> Supervised0.7843−47.9190e −7.2569y 0.7850−39.1641e −6.9986y 0.7729−69.5249e −7.8363y End-Of-Token Distilled0.7726−110.6994e −8.3872y 0.7673−133.6625e −8.6766y 0.7728−102.4737e −8.3038y 42 0.720.800.880.96 Validation BPB (descending -->) 0.46 0.50 0.60 0.70 Average Error (1 - % Accuracy) LR=1e-3 0.720.800.880.96 Validation BPB (descending -->) 0.46 0.50 0.60 0.70 LR=4e-3 0.720.800.880.96 Validation BPB (descending -->) 0.46 0.50 0.60 0.70 LR=8e-3 0.720.800.880.96 Validation BPB (descending -->) 0.46 0.50 0.60 0.70 Average Error (1 - % Accuracy) LR=1e-3 0.720.800.880.96 Validation BPB (descending -->) 0.46 0.50 0.60 0.70 LR=4e-3 0.720.800.880.96 Validation BPB (descending -->) 0.46 0.50 0.60 0.70 LR=8e-3 Downstream Error: Scaling Laws Downstream Error: Scaling Laws (curves only) Token Supervised Token Distilled Bytes Supervised Marginalize-It Distilled Bytes w/ <eot> Supervised End-Of-Token Distilled Asymptotic point (BPB, Err) Figure 25: Downstream error scaling laws calibrated using Validation dataset 2 BPB values. Table 21: Asymptotic validation BPB (BPB ⋆ ) and the corresponding downstream average accuracy predicted by the scalingz(y) =ε−k·e −γy aty=c, for each model variant and learning rate calibrated using Validation dataset 2 BPB values. LR= 1×10 −3 LR= 4×10 −3 LR= 8×10 −3 Model c a ⋆ c a ⋆ c a ⋆ Token Supervised0.7456 43.6 0.7263 47.9 0.7267 47.6 Token Distilled0.7119 49.2 0.7272 48.7 0.7316 46.1 Bytes Supervised0.7258 47.4 0.7038 50.5 0.7157 47.5 Marginalize-It Distilled0.7214 48.1 0.7155 48.9 0.7264 46.5 Bytes w/ <eot> Supervised0.7180 47.7 0.7057 49.6 0.7253 46.4 End-Of-Token Distilled0.7223 48.6 0.7142 50.5 0.7181 49.1 43 E.3Feather Plots: Comparisons using validation dataset 2 0.720.800.880.96 Validation BPB (descending -->) 0.50 0.60 0.70 Average Error (1 - % Accuracy) LR=1e-3 0.720.800.880.96 Validation BPB (descending -->) 0.50 0.60 0.70 LR=4e-3 0.720.800.880.96 Validation BPB (descending -->) 0.50 0.60 0.70 LR=8e-3 10 20 10 22 10 24 10 26 10 28 10 30 Training FLOPs (isoFLOP lines) Downstream Error: IsoFLOP Lines Token DistilledEnd-Of-Token DistilledMarginalize-It Distilled Asymptotic point (BPB, Err)Crossover point: reaches Token Distilled asymptotic error Figure 26: Feather plots calibrated using Validation dataset 2 BPB values E.4 Predicting the asymptotic Average Downstream task performance Table 22: Training compute at whichMarginalize-It DistilledandEnd-Of-Token Distilled match the asymptotic downstream performance of Token Distilled using Validation Dataset 2. We take Token Distilledś asymptotic BPB (thecterm of its fitted BPB–FLOPs power law; columnToken Dist. BPB) and map it through Token Distilledś benchmark scaling law to obtain its asymptotic downstream accuracy (columnAccuracy). For each byte method we then invert its own benchmark scaling law at that accuracy to get the BPB it must attain (columnMethod BPB), and invert its BPB–FLOPs power law at that BPB to get the required training compute (columnTraining FLOPs). The accuracy column is shared within each learning rate because both methods are evaluated against the same Token Distilled asymptote. LRMethodToken Dist. BPB Accuracy (%) Method BPB Method Training FLOPs 10 −3 Marginalize-It Distilled— End-Of-Token Distilled— 4×10 −3 Marginalize-It Distilled0.727248.7%0.71651.60×10 25 End-Of-Token Distilled0.727248.7%0.72191.69×10 23 8×10 −3 Marginalize-It Distilled0.731646.1%0.72865.17×10 23 End-Of-Token Distilled0.731646.1%0.73263.01×10 22 44 Table 23: Effective training-data volume at the compute budget of Table22using Validation Dataset 2. Here FPU (FLOPs Per Unit) is the training FLOPs consumed per sequence unit, where a unit is one token for Token Distilled and one byte for the byte-level methods; it is read directly from each run and used to convert a compute budget into a data count.Byte Units=FLOPs/FPU bytes is the number of byte units the method processes at its crossover compute.Bytes=Byte Units×Multiplier converts byte units into actual UTF-8 bytes of unique text, accounting for the fraction of each unit that is genuine content (Multiplier= 1forMarginalize-It Distilled;= 4.5/5.5forEnd-Of-Token Distilled, whose sequences carry an extra end-of-token symbol per4.5content bytes).Tokens= FLOPs/FPU tokens is the token count Token Distilled would process at the same compute.Token to Byte=Tokens×4.5expresses that token count in bytes using the average of4.5bytes per token.Data Savings=Token to Byte/Bytes is how much less unique text the byte method needs to reach the same performance.Storage Savings= (Tokens×4.5)/Byte Units compares the token pipelineś byte footprint against the raw byte-unit count. LRMethodTraining FLOPs FPU bytes FPU tokens Byte Units Bytes Tokens Token to Byte Data Savings Storage Savings 10 −3 Marginalize-It Distilled— End-Of-Token Distilled— 4×10 −3 Marginalize-It Distilled1.60×10 25 8.81×10 9 8.18×10 9 1.82×10 15 1.82×10 15 1.96×10 15 8.82×10 15 4.85×4.85× End-Of-Token Distilled1.69×10 23 9.44×10 9 8.18×10 9 1.79×10 13 1.47×10 13 2.07×10 13 9.30×10 13 6.35×5.19× 8×10 −3 Marginalize-It Distilled5.17×10 23 8.81×10 9 8.18×10 9 5.87×10 13 5.87×10 13 6.32×10 13 2.84×10 14 4.85×4.85× End-Of-Token Distilled3.01×10 22 9.44×10 9 8.18×10 9 3.19×10 12 2.61×10 12 3.68×10 12 1.66×10 13 6.35×5.19× 45 F Distillation Methods: Additional Details F.1Marginalize-ItandEnd-Of-TokenPseudocode importnumpyasnp EOT = 256# end-of-token symbol (index 256 when use_eot=True) defbyte_probabilities(probs, token_to_bytes, lengths, gold, use_eot=False): """Byte distributions P(b | gold[:k], context) for k = 0..len(gold)-1. probs (vocab_size):token probabilities. token_to_bytes (vocab_size, max_len): per-token bytes, -1 padded. lengths (vocab_size):true byte length. gold:byte sequence to condition on. use_eot:if True, alphabet is 0..255 U EOT (size 257) else 256. """ n_bytes = 257ifuse_eotelse256 live = np.ones(len(probs), dtype=bool) dist = [] forkin range(len(gold)): active = live & (lengths > k) bytes_at_k = token_to_bytes[:, k] accum = np.bincount(bytes_at_k[active], probs[active], minlength=n_bytes) total = accum.sum() dist.append(accum / totaliftotal > 0elseaccum) live &= (bytes_at_k == gold[k]) & (lengths > k + 1) returndist F.2 ComparingMarginalize-Itmethod withEnd-Of-Token In this section we discuss four special cases of byte logit conversion: Case I: Predicting token’s first byte distribution using token level teacher distribution (Figure 27): Predicting First Byte Marginalize-It Ground Truth Tokenization: [‘iram’, ‘isu’, ...] End-Of-Token Ground Truth Tokenization: [‘iram<eot>’,’isu<eot>’, ...] B1B1 i iiiiimmn? i iiiiimmn? sssmmniaes s smmniae u kpaotxwuk <eot> paotxw lp <eot> <eot> lp <eot><eot><eot> <eot><eot> (a) B1 is predicted directly by marginalization using next token distribution of tokeniramand is exact. (b) B1 is predicted directly by marginalization using next token distribution of tokeniram<eot>and is exact. Figure 27: We demonstrate the first byte prediction of the next token. BothMarginalize-Itand End-Of-Tokenmethods exactly preserve teacher distributions in this case Case I: Predicting token’s intermediate byte distribution using token level teacher distribution when 46 the ground truth token length matches with all other alternative tokens with non-zero probability in the vocabulary (Figure28): : Predicting Intermediate Byte: l(ground truth) = l(alternative paths) Marginalize-It Ground Truth Tokenization: [‘isu’, ...] End-Of-Token Ground Truth Tokenization: [‘isu<eot>’, ...] B3B3B4 i i i i s u kmt ? ukmt ? u <eot> <eot><eot><eot> ? (a) B3 prediction is based on tokens with prefixiswith a length of at least 3 bytes. Other alternative paths (with non zero probability) have exactly 3 bytes length and can be marginalized exactly. (b) B3 prediction is based on tokens with the prefixis and a length of at least 3 bytes. All alternative paths with non zero probabilities have byte length 4 and can be marginalized exactly. Similarly, for B4 prediction, all paths have length of 4 and can be marginalized exactly. Figure 28: We demonstrate the intermediate byte prediction case when all other alternative paths with non zero probability in the vocabulary have exactly the same length as that of the ground truth token. In this case, bothMarginalize-ItandEnd-Of-Tokenmethods behave equally and give exact byte probabilities. Case I: Predicting token’s intermediate byte distribution using token level teacher distribution when there exist other alternative tokens with non zero probabilities in the vocabulary sharing the same prefix but having shorter byte sequence length compared to the ground truth token (Figure 29):. 47 Predicting Intermediate Byte: l(ground truth) > l(alternative paths) Marginalize-It Ground Truth Tokenization: [‘isu’, ...] End-Of-Token Ground Truth Tokenization: [‘isu<eot>’, ...] B3B3B4 i i i i i i s ukoou ? uk <eot> ? u oou <eot > <eot > oou ? ouoou o <eot> u <eot><eot> (a) B3 prediction is based on tokens with prefixiswith a length of at least 3 bytes. Therefore it misses other tokenization paths: [is,o] , [is,o], [is,u] (b) B3 prediction is based on tokens with the prefixis and a length of at least 3 bytes. Adding<eot>preserves the distribution, as the byte length ofis<eot>is now 3 and no alternative token paths are missed. B4 predic- tion is conditioned on tokens starting with prefixisu and with a length of at least 4. Since only the token isu<eot>satisfies that condition, the teacher distribu- tion is preserved exactly. Figure 29: We demonstrate the intermediate byte prediction when there exits other tokens in the vocabu- lary with shorter byte length with non zero probabilities. SinceMarginalize-Itcomputes probabilities based on the ground truth token it can miss these alternative paths.End-Of-Tokenmethod on the other hand circumvents this problem by adding the new<eot>token. Case IV: Predicting token’s intermediate byte distribution using token level teacher distribution when there exist other alternative tokens with non zero probabilities in the vocabulary sharing the same prefix but having longer byte sequence length compared to the ground truth token. (Figure 30): 48 Predicting Intermediate Byte: l(ground truth) < l(alternative paths) Marginalize-It Ground Truth Tokenization: [‘is’,’u’] End-Of-Token Ground Truth Tokenization: [‘is<eot>’, ‘u<eot>’] B3B3B1’ i i i i i s s s s s s uko ou ? uk <eot> ? <eot> oou <eot><eot> o ou ? o u oou o <eot> u <eot> <eot> (a) B3 prediction is based on the next token distribution ofisand therefore it misses contributions from other tokenization paths formed by tokensisuandisk. (b) B3 prediction is based on tokens with prefixisand has a length of at least 3 bytes. Adding<eot>pre- serves the distribution as now byte length ofis<eot>is 3 and no alternative token paths are missed. B4 predic- tion is based on the next token distribution ofis<eot> and preserves exact distribution as no other token paths have prefixis<eot>. Figure 30: We demonstrate intermediate byte prediction when there exist other tokens in the vocabulary with longer byte lengths and non-zero probabilities. SinceMarginalize-Itcomputes probabilities based on the ground-truth token, it can miss the contributions from these alternative paths. The End-Of-Token method, on the other hand, circumvents this problem by adding the new<eot>token. G On BPB vs Benchmark Performance Discrepancy G.1 Why the same BPB may imply different downstream performance? BPB is a function of only one number per position: the probability the model places on the target token, p correct . The per-token loss is−log 2 p correct , so oncep correct is fixed the loss is fixed—BPB is completely blind to how the remaining mass1−p correct is arranged among the other vocabulary items. Downstream accuracy, in contrast, depend on whether the target is the argmax, i.e. on the ranking of the full distri- bution. Since the ranking is governed by the residual mass that BPB ignores, two models can share an identical BPB while decoding completely different text. Example:Concretely, take the target “Where's my tiramisu?”, which the Llama3-8b tokenizer segments into7tokens spanning20bytes: Where|'s|␣my|␣tir|am|isu|? where␣denotes a leading space. Suppose both models place probabilityp correct = 0.4on the correct token at every position. Then each per-token loss is−log 2 0.4 = 1.32bits, the total is7×1.32 = 9.24 bits, and BPB= 9.24 20 = 0.462(7) for both models. The models differ only in where they place the remaining0.6of mass—which BPB never observes: In Model A the target is rank 1 at every step, so greedy decoding produces “Where's my tiramisu?”—a 7/7exact match. In Model B a single distractor sits at0.5, just above the target’s0.4, so the target is rank 2 at every step and the concatenated argmax tokens spell “There'd be calzone!”—0/7. The loss never observes the0.5distractor greater the0.4target; it considers only the0.4. Both models therefore 49 Table 24: An illustration of how the relative ranking of tokens in the vocabulary can lead to worse generation performance while maintaining the same BPB value. Pos Targetp correct Model AModel B distractorsargmax distractorsargmax 1Where0.4There= 0.35,When= 0.25Where3There= 0.5,When= 0.1There7 2's0.4're= 0.3,'d= 0.3's3'd= 0.5,'re= 0.1'd7 3␣my0.4␣your= 0.35,␣the= 0.25␣my3␣be= 0.5,␣the= 0.1␣be7 4␣tir0.4␣bur= 0.35,␣piz= 0.25␣tir3␣cal= 0.5,␣bur= 0.1␣cal7 5am0.4av= 0.3,ig= 0.3am3z= 0.5,av= 0.1z7 6isu0.4ami= 0.35,osa= 0.25isu3one= 0.5,osa= 0.1one7 7?0.4.= 0.35,!= 0.25?3!= 0.5,.= 0.1!7 have an identical BPB of0.463while achieving100% versus0% token accuracy.Equal BPB constrains the probability of the target but does not guarantee a rank 1. References Al-Rfou, R., Choe, D., Constant, N., Guo, M., & Jones, L. (2018). Character-level language modeling with deeper self-attention. InAaai conference on artificial intelligence.Retrieved fromhttps:// api.semanticscholar.org/CorpusID:52004855 Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., ... others (2021). Program synthesis with large language models.arXiv preprint arXiv:2108.07732. Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U. (2024). Explaining neural scaling laws.Proceedings of the National Academy of Sciences , 121 (27), e2311878121. Bansal, Y., Ghorbani, B., Garg, A., Zhang, B., Cherry, C., Neyshabur, B., & Firat, O. (2022). Data scaling laws in nmt: The effect of noise and architecture. InInternational conference on machine learning(p. 1466–1482). Bao, Z., Leng, J., Wang, J., Peng, B., & Lu, Y. (2026). Distilling token-trained models into byte-level models.arXiv preprint arXiv:2602.01007. Beyer, L., Zhai, X., Royer, A., Markeeva, L., Anil, R., & Kolesnikov, A. (2022). Knowledge distillation: A good teacher is patient and consistent. InProceedings of the ieee/cvf conference on computer vision and pattern recognition(p. 10925–10934). Bhagia, A., Liu, J., Wettig, A., Heineman, D., Tafjord, O., Jha, A. H., ... others (2024). Establishing task scaling laws via compute-eﬀicient model ladders.arXiv preprint arXiv:2412.04403. Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. (2020). Piqa: Reasoning about physical commonsense in natural language. InProceedings of the aaai conference on artificial intelligence(Vol. 34, p. 7432–7439). Busbridge, D., Shidani, A., Weers, F., Ramapuram, J., Littwin, E., & Webb, R. (2025). Distillation scaling laws.arXiv preprint arXiv:2502.08606. Choe, D., Al-Rfou, R., Guo, M., Lee, H., & Constant, N. (2019). Bridging the gap for tokenizer-free language models.arXiv e-prints, arXiv–1908. Clark, J. H., Garrette, D., Turc, I., & Wieting, J. (2022). Canine: Pre-training an eﬀicient tokenization- free encoder for language representation.Transactions of the Association for Computational Lin- guistics,10, 73–91. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018). Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In2009 ieee conference on computer vision and pattern recognition(p. 248-255). DOI: 10.1109/CVPR.2009.5206848 Dettmers, T., & Zettlemoyer, L. (2023). The case for 4-bit precision: k-bit inference scaling laws. In International conference on machine learning(p. 7750–7774). El Boukkouri, H., Ferret, O., Lavergne, T., Noji, H., Zweigenbaum, P., & Tsujii, J. (2020). Character- bert: Reconciling elmo and bert for word-level open-vocabulary representations from characters. In Proceedings of the 28th international conference on computational linguistics(p. 6903–6915). 50 Fang, L., Wang, Y., Liu, Z., Zhang, C., Jegelka, S., Gao, J., ... Wang, Y. (2025). What is wrong with per- plexity for long-context language modeling? InInternational conference on learning representations (Vol. 2025, p. 94541–94563). Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., ... others (2024). Language models scale reliably with over-training and on downstream tasks.arXiv preprint arXiv:2403.08540. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., ... Zou, A. (2024, 07).The lan- guage model evaluation harness.Zenodo. Retrieved fromhttps://zenodo.org/records/12608602 DOI: 10.5281/zenodo.12608602 Goyal, N., Gao, C., Chaudhary, V., Chen, P.-J., Wenzek, G., Ju, D., ... Fan, A. (2022). The flores-101 evaluation benchmark for low-resource and multilingual machine translation.Transactions of the Association for Computational Linguistics,10, 522–538. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., ... others (2024). The llama 3 herd of models.arXiv preprint arXiv:2407.21783. Hayase, J., Liu, A., Smith, N. A., & Oh, S. (2025). Sampling from your language model one byte at a time.arXiv preprint arXiv:2506.14123. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. InProceedings of the ieee conference on computer vision and pattern recognition(p. 770–778). Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., ... others (2020). Scaling laws for autoregressive generative modeling.arXiv preprint arXiv:2010.14701. Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., ... Zhou, Y. (2017). Deep learning scaling is predictable, empirically.arXiv preprint arXiv:1712.00409. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., ... others (2022). Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,10. Hwang, S., Wang, B., & Gu, A. (2026). Dynamic chunking for end-to-end hierarchical sequence modeling. InInternational conference on learning representations(Vol. 2026, p. 149273–149313). Isik, B., Ponomareva, N., Hazimeh, H., Paparas, D., Vassilvitskii, S., & Koyejo, S. (2025). Scaling laws for downstream task performance in machine translation. InInternational conference on learning representations(Vol. 2025, p. 88769–88790). Kamath, G. T. A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., ... Hussenot, L. (2025). Gemma 3 technical report.ArXiv,abs/2503.19786. Retrieved from https://api .semanticscholar.org/CorpusID:277313563 Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., ... Amodei, D. (2020). Scaling laws for neural language models.arXiv preprint arXiv:2001.08361. Kosson, A., Welborn, J., Liu, Y., Jaggi, M., & Chen, X. (2025). Weight decay may matter more than mup for learning rate transfer in practice.arXiv preprint arXiv:2510.19093. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., ... Petrov, S. (2019). Nat- ural questions: A benchmark for question answering research.Transactions of the Association for Computational Linguistics,7, 452–466. Retrieved fromhttps://aclanthology.org/Q19-1026/ Lee, C., Yan, J. N., Liang, C., Shi, J., Zhang, Y., Liu, J., ... others (2026). The eﬀiciency gap in byte modeling.arXiv preprint arXiv:2605.12928. Limisiewicz, T., Pagnoni, A., Iyer, S., Lewis, M., Mehta, S., Liu, A., ... Zettlemoyer, L. (2026). Compute optimal tokenization.arXiv preprint arXiv:2605.01188. Liu, H., Xie, S. M., Li, Z., & Ma, T. (2023). Same pre-training loss, better downstream: Implicit bias matters for language models. InInternational conference on machine learning(p. 22188–22214). Loshchilov, I., & Hutter, F. (2017). Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101. Lourie, N., Hu, M. Y., & Cho, K. (2025). Scaling laws are unreliable for downstream tasks: A reality check.arXiv preprint arXiv:2507.00885. Magnusson, I., Tai, N., Bogin, B., Heineman, D., Hwang, J. D., Soldaini, L., ... others (2025). Datadecide: How to predict best pretraining data with small experiments.arXiv preprint arXiv:2504.11393. Meta AI. (2024).Llama 3.2: Revolutionizing edge ai and vision.Retrieved fromhttps://ai.meta.com/ blog/llama-3-2-connect-2024-vision-edge-mobile-devices/(Accessed: 2024-09-25) Minixhofer, B., Murray, T., Limisiewicz, T., Korhonen, A., Zettlemoyer, L., Smith, N. A., ... Hof- mann, V. (2025). Bolmo: Byteifying the next generation of language models.arXiv preprint arXiv:2512.15586. 51 Minixhofer, B., Vulić, I., & Ponti, E. M. (2026). Universal cross-tokenizer distillation via approximate likelihood matching.Advances in Neural Information Processing Systems,38, 79297–79326. Nawrot, P., Chorowski, J., Lancucki, A., & Ponti, E. M. (2023, July). Eﬀicient transformers with dynamic token pooling. In A. Rogers, J. Boyd-Graber, & N. Okazaki (Eds.),Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers)(p. 6403–6417). Toronto, Canada: Association for Computational Linguistics. Retrieved fromhttps:// aclanthology.org/2023.acl-long.353/DOI: 10.18653/v1/2023.acl-long.353 Owen, D. (2024). How predictable is language model benchmark performance?arXiv preprint arXiv:2401.04757. Pagnoni, A., Pasunuru, R., Rodriguez, P., Nguyen, J., Muller, B., Li, M., ... others (2025). Byte latent transformer: Patches scale better than tokens. InProceedings of the 63rd annual meeting of the association for computational linguistics (volume 1: Long papers)(p. 9238–9258). Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting on association for computational linguistics(p. 311–318). USA: Association for Computational Linguistics. Retrieved fromhttps:// doi.org/10.3115/1073083.1073135DOI: 10.3115/1073083.1073135 Peng, H., Lv, X., Bai, Y., Yao, Z., Zhang, J., Hou, L., & Li, J. (2025). Pre-training distillation for large language models: A design space exploration. InProceedings of the 63rd annual meeting of the association for computational linguistics (volume 1: Long papers)(p. 3603–3618). Phan, B., Amos, B., Gat, I., Havasi, M., Muckley, M., & Ullrich, K. (2024). Exact byte-level prob- abilities from tokenized language models for fim-tasks and model ensembles.arXiv preprint arXiv:2410.09303. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., ... Dean, J. (2023). Eﬀiciently scaling transformer inference.Proceedings of machine learning and systems,5, 606–624. Sardana, N., Portes, J., Doubov, S., & Frankle, J. (2023). Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.arXiv preprint arXiv:2401.00448. Schaeffer, R., Miranda, B., & Koyejo, S. (2023). Are emergent abilities of large language models a mirage? Advances in neural information processing systems,36, 55565–55581. Shazeer, N. (2020). Glu variants improve transformer.arXiv preprint arXiv:2002.05202. Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., ... others (2024). Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., ... others (2023). Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... Polosukhin, I. (2017). Attention is all you need.Advances in neural information processing systems,30. Veličković, P., Barbero, F., Perivolaropoulos, C., Osindero, S., & Pascanu, R. (2026). Perplexity cannot always tell right from wrong.arXiv preprint arXiv:2601.22950. Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., ... Raffel, C. (2022). Byt5: Towards a token-free future with pre-trained byte-to-byte models.Transactions of the Association for Computational Linguistics,10, 291–306. Yu, L., Simig, D., Flaherty, C., Aghajanyan, A., Zettlemoyer, L., & Lewis, M. (2023). Megabyte: Predicting million-byte sequences with multiscale transformers.Advances in Neural Information Processing Systems,36, 78808–78823. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019). Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th annual meeting of the association for computational linguistics(p. 4791–4800). Zhang, T. T., Shah, A., Zhang, Y., Zhang, V., Matni, N., & Simchowitz, M. (2026). Double pre- conditioning (dopr): Optimization for test-time performance, not validation loss.arXiv preprint arXiv:2606.06418. Zheng, L., Zhao, X., Wang, G., Wu, C., Dong, D., Wang, A., ... Kong, L. (2025).Evabyte: Eﬀicient byte- level language models at scale.Retrieved fromhttps://hkunlp.github.io/blog/2025/evabyte 52

Instant research discovery