Paper deep dive
Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs
Deniz Bayazit, Badr AlKhamissi, Antoine Bosselut
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/2/2026, 4:20:12 AM
Summary
This paper investigates whether latent language identification (LLID) probes in multilingual LLMs measure a single internal 'lingua franca' or distinct aspects of multilingual computation. By comparing GMM-based representation probes (relying on hidden state geometry) and decoding-based probes (relying on output-space decodability) across 12 models and 27 languages, the authors find systematic disagreements. Representation probes show earlier cross-lingual mixing, while decoding probes retain sharper, more English-biased signals. The study concludes that current probes expose different facets of processing rather than revealing a unified internal language.
Entities (10)
Relation Signals (7)
LogitLens → istypeof → Decoding-based probe
confidence 95% · Applying the model’s unmodified output head directly to an intermediate hidden state is known as the logitlens
Tuned Lens → istypeof → Decoding-based probe
confidence 95% · we also consider the tuned lens... before applying the output head
Decoding-based probe → usesevidencefrom → output-space decodability
confidence 95% · decoding-based probes, which rely on output-space decodability
GMM-based representation probe → usesevidencefrom → hidden state geometry
confidence 95% · the GMM-based representation probe, which draws evidence from hidden state geometry
Decoding-based probe → exhibits → English-biased signals
confidence 90% · decoding-based probes... retain sharper language-specific and more English-biased signals
GMM-based representation probe → exhibits → earlier cross-lingual mixing
confidence 90% · the GMM-based representation probe... shows earlier cross-lingual mixing
Llama-2-7b → exhibits → English-centered bias
confidence 90% · Llama-2 frequently maintains relatively high probability mass on English lexicalizations... suggesting a stronger English-centered bias
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Latent language identification is often used to argue that multilingual language models route computation through language-specific states, such as English pivots. However, existing probes infer latent language from different signals, such as the geometry of hidden states or what can be decoded from intermediate representations. Since such claims shape conclusions about how models share and route information across languages, we ask whether these probes measure the same phenomenon or expose distinct aspects of multilingual computation. We study this question across model families, training regimes, domains, tasks, checkpoints, and up to 27 languages. We find that identification probes systematically disagree: the GMM-based representation probe, which draws evidence from hidden state geometry, shows earlier cross-lingual mixing, whereas decoding-based probes, which rely on output-space decodability, retain sharper language-specific and more English-biased signals. These differences track model multilinguality and training progression, but are comparatively stable across domains. Our results suggest a more cautious interpretation of latent language identification, where current probes expose different aspects of multilingual processing, rather than directly revealing a single internal lingua franca.
Tags
Links
- Source: https://arxiv.org/abs/2609.00155v1
- Canonical: https://arxiv.org/abs/2609.00155v1
Trouble viewing inline? Open PDF directly →
Full Text
75,861 characters extracted from source content.
Expand or collapse full text
Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs Deniz Bayazit, Badr AlKhamissi, Antoine Bosselut EPFL Correspondence: deniz.bayazit,badr.alkhamissi,antoine.bosselut@epfl.ch Abstract Latent language identification is often used to argue that multilingual language models route computation through language-specific states, such as English pivots. However, existing probes infer latent language from different sig- nals, such as the geometry of hidden states or what can be decoded from intermediate rep- resentations. Since such claims shape conclu- sions about how models share and route infor- mation across languages, we ask whether these probes measure the same phenomenon or ex- pose distinct aspects of multilingual computa- tion. We study this question across model fam- ilies, training regimes, domains, tasks, check- points, and up to 27 languages.We find that identification probes systematically dis- agree: the GMM-based representation probe, which draws evidence from hidden state ge- ometry, shows earlier cross-lingual mixing, whereas decoding-based probes, which rely on output-space decodability, retain sharper language-specific and more English-biased sig- nals. These differences track model multilin- guality and training progression, but are com- paratively stable across domains. Our results suggest a more cautious interpretation of latent language identification, where current probes expose different aspects of multilingual pro- cessing, rather than directly revealing a single internal lingua franca. 1 1 Introduction Multilingual language models are often assumed to share information across languages by map- ping them into common or partially language- neutral representations (Foroutan et al., 2022; Du- mas et al., 2025). Yet the form of this sharing remains poorly understood. One influential hypoth- esis is that multilingual models rely on a latent language: an intermediate language-like regime, 1 The code is available at: https://github.com/bayazitdeniz/latent-lid Decoder Layer Decoder Layer + + + Model EN FR TR EN FR TR TR: Kedi atlıyor. FR: Le chat ... MethodLLID EN FR TR Latent Language Distribution Logit Lens Tuned Lens GMM Latent Language Identification Conflicts Figure 1: Do multilingual LLMs work in English? It depends on how you ask. We probe the same inter- mediate hidden states of a multilingual LLM with three probes: logitlens, tuned lens, and a representation-based GMM, and recover three different latent language dis- tributions over the task’s languages (TR, FR), and other candidate languages like English (EN). We study when and why these probes disagree across models, training regimes, and tasks. sometimes described as an internal pivot or lingua franca, through which the model routes meaning before producing an output. Wendler et al. (2024) provide evidence for such behavior in English- dominated models like LLaMA-2 (Touvron et al., 2023), but newer models trained on more balanced multilingual data may behave differently (Üstün et al., 2024; Foroutan et al., 2025; Schut et al., 2025). This phenomenon raises a broader question: what part of a latent language estimate reflects the model’s internal computation, and what part reflects the assumptions of the diagnostic method? Interpreting such probes is challenging because latent language identification (LLID) is inferred rather than directly observed. As illustrated in Fig. 1, a latent language estimate depends on what kind of evidence is extracted from the model. Prior work relies on two broad families (Fig. 2): representation-based probes, which infer language arXiv:2609.00155v1 [cs.CL] 31 Aug 2026 LLID Across Model Layers ModelLanguage Prior Modeling h EN English Bucket PCA GMM: one component per lang Posterior Dist. P(EN | ) = 0.72 P(FR | ) = 0.20 h h Evidence 2: Representation-based h FR h FR French Bucket ... Decoder Layer Decoder Layer + + + ModelLogitLens Softmax LayerNorm Unembed jumps saute Output Dist. Evidence 1: Decoding-based Decoder Layer Decoder Layer + + + h TurkishEnglishArabicFrench TR: Kedi atlıyor. FR: Le chat ... ... saute Figure 2: Two families of latent language identifica- tion (LLID) probes. Given an intermediate hidden state h, decoding-based methods (left) projecththrough the model’s unembedding (optionally via a tuned lens) and derive language scores from the resulting vocabulary distribution. Representation-based methods (right) in- stead comparehwith language-conditioned activation distributions, here a per-layer GMM with one compo- nent per candidate language. The two families operate on the samehbut extract different evidence: output- space decodability versus activation-space geometry. from the geometry of hidden states (Shani and Basirat, 2025), and decoding-based probes, which infer language by projecting intermediate hidden states through the model’s output head (Wendler et al., 2024; Zhong et al., 2025). These families make different assumptions about what it means for a model to internally use a language, and it is unclear whether they identify a common mecha- nism or expose different processes in multilingual encoding and decoding. In this work, we treat LLID as a measurement problem. Instead of asking whether multilingual LLMs “think in English,” we ask whether latent language estimates reflect model properties, such as multilinguality and training progression, or di- agnostic choices, such as representation geometry versus output-space decodability. To answer this, we compare a GMM-based representation probe and decoding probes across controlled and open- ended multilingual tasks. We first test the internal consistency and generalization of decoding-based LLID across controlled and open-ended settings (§4–5), before asking whether representation-based LLID recovers the same patterns under matched conditions (§6). We then trace how these esti- mates change across domains, training regimes, and checkpoints. Our analysis shows that LLID probes should not be treated as interchangeable measurements of a single phenomenon. The GMM represen- tation probe reveals earlier cross-lingual mixing and weaker English dominance, while decoding- based probes retain sharper language-specific and more English-biased signals. This pattern recurs across our analyses, but the layerwise behavior of each probe is model-dependent. For a given probe, where these effects emerge in the layer stack and how they change across layers vary with model multilinguality and over the course of model pre- training, although multilinguality alone does not determine them. By contrast, domain and language- inventory changes shift LLID estimates only mod- estly. Rather than supporting strong claims about an internal lingua franca, each diagnostic provides a partial view of multilingual computation, shaped by the evidence it extracts from hidden states or decoded outputs. 2 A Framework for Comparing Latent Language Probes 2.1 Task Definition: Latent Language Identification LetLbe a finite set of candidate languages, and leth ∈ R d denote a hidden state extracted from a given model layer and sequence positions. We represent latent language with a variableZ ∈ L and estimate its conditional distribution givenhas: q(ℓ)≈ Pr(Z = ℓ| h), ℓ∈L As the variableZis not directly observed, we treat qas a diagnostic estimate, not as ground-truth ev- idence of an internal language variable. Different LLID estimators may therefore produce different distributions for the same prompt and layer. In practice, each estimator constructsqby selecting one or more token positions and aggregating the resulting language evidence (§3). 2.2 Two Kinds of Evidence for LLID and Their Limits Existing LLID methods differ mainly in the evi- dence they use to constructq. We distinguish two families. Representation-based methods infer lan- guage from where a hidden state lies in activation space relative to language-conditioned structure learned from multilingual data. Decoding-based methods infer language from what can be read out from that hidden state through the model’s output vocabulary. Representation-based LLID. This approach compares a hidden statehto language-conditioned structure learned from multilingual data, such as GMM posteriors (Shani and Basirat, 2025), lan- guage subspaces (Zhao et al., 2025), or neuron- level indicators (Zeng et al., 2025). This compar- ison yields per-language scores, which are then aggregated into a layerwise distribution q. In our experiments, we fit a per-layer GMM with one language-conditioned component per candi- date language using hidden states from examples in each language, and use the resulting component posteriors as the layerwise language distributionq. A common concern for this approach is that learned structure may conflate language identity with correlated factors such as script, tokenization, or domain. In our experiments, however, we find consistent behavior across QA domains such as STEM and Arts & Humanities (§6). Decoding-based LLID. These approaches esti- mate latent language by asking what language can be decoded from an intermediate layer. LetVbe the model’s vocabulary and letW U ∈ R |V|×d be the model’s unembedding matrix. Given any hid- den stateh ∈ R d , after final layer normalization, we form an intermediate vocabulary distribution: p := softmax(W U h) Applying the model’s unmodified output head di- rectly to an intermediate hidden state is known as the logitlens (nostalgebraist, 2020), and we refer to this direct projection as the raw logitlens. Because intermediate representations may not align with those expected by the final output head, we also consider the tuned lens (Belrose et al., 2025), which learns a per-layer affine transformation to predict the final-layer hidden state from an intermediate one before applying the output head. Prior work mapspto language scoresqin several ways, such as summing probability over valid first- token prefixes of language-specific target words (Wendler et al., 2024; Dumas et al., 2025), scoring the probability of multi-token language-specific an- swer sequences (Zhong et al., 2025), or decoding one or more tokens frompand classifying the re- sult with an external language identification (LID) model (Ozaki et al., 2025). In our experiments, for tasks where the completion is known, we sum the probability over valid first-tokens, whereas if the task is open-ended, we use two rollout-based vari- ants (argmax and top-p) where short continuations decoded from intermediate layers are passed to a LID classifier (Kargaran et al., 2023). Decoding-based LLID methods assume that the final unembedding remains meaningful at interme- diate layers, which may not hold in earlier layers. A further concern is that token-level heuristics such as target-string matching require non-trivial design choices (e.g., gathering valid completions) to yield a meaningful distribution overL. 2.3 Metrics to Compare LLID Estimates LLID estimates vary along several dimensions that cannot be captured by a single metric: distributions can be confident or diffuse, while their dominant language can match the task-relevant language or pivot to another one. Furthermore, two estimators can agree or disagree. We therefore use a small set of complementary metrics. Unless otherwise stated, each is computed per estimator and averaged over prompts at each layer. Entropy. Entropy,H = − P ℓ∈L q(ℓ) logq(ℓ), measures how diffuse the LLID distribution is; lower values indicate a peaky estimate, while higher values indicate mixing across languages. Dominance. We measure dominance withD = max ℓ∈L q(ℓ) , which is high when most probability mass is assigned to a single language. Pivot rate. Pivot rate is the fraction of prompts whose dominant estimated language differs from the task-relevant languageℓ x (e.g., the source or target language in translation): P x = I arg max ℓ∈L q(ℓ)̸= ℓ x Agreement. Given two LLID estimatorsaand b, agreement measures how often the estimators assign the same dominant language: A x = I arg max ℓ∈L q a (ℓ) = arg max ℓ∈L q b (ℓ) In our experiments, this is used to compare decoding-based and representation-based LLIDs. 3 Experimental Setup 3.1 Models & Measuring Multilinguality We evaluate up to 12 autoregressive language mod- els spanning English-centric baselines, multilin- gual base models, and instruction-tuned variants. Because training mixtures are not fully available, we consider the following multilinguality prox- ies: zero-shot multilingual QA accuracy, cross- lingual perplexity, and byte-normalized likelihood. Unless otherwise stated, cross-model plots order models by increasing zero-shot macro-accuracy on INCLUDE(Romanou et al., 2025). Model details, scores, and rankings are in Appendix B. 3.2 Evaluation Regimes We evaluate LLID in two settings: (1) a con- trolled regime where ground-truth completions are available, and (2) an open-ended regime that tests whether the same patterns hold in natural text. The exact language inventory, i.e., the set of candidate languages included in each setting, is listed in Ta- ble 1 in Appendix A. Controlled Target-String Evaluation. We use three synthetic task formats: copy, cloze, and trans- lation. Copy asks the model to reproduce a concept word in the same language, cloze asks the model to complete a masked context, and translation asks for the concept word in a specified target language. These examples build on prior target-word settings for latent language analysis (Wendler et al., 2024; Dumas et al., 2025; Zhong et al., 2025). For each promptxand candidate languageℓ, letw ℓ,x be the language-ℓanswer. Following Wendler et al., let START τ (w ℓ,x ) ⊆ Vdenote the set of vocabulary tokens under tokenizerτthat can beginw ℓ,x . For simplicity, we fix x, ℓ, and τ and write START(w). The controlled decoding-based language score is then defined as: s x (ℓ) = X t∈START(w) P θ (x n+1 = t| h n ), whereh n is the hidden state at the final prompt token. We refer to this summed mass as the START(w)probability. When a language distri- bution is needed, we normalizes x over the can- didate setLto obtainq x . Construction details for START(w)and candidate language completions are in Appendix A.2. Open-Ended Evaluation. For natural-language prompts without constrained completions, we sum- marize both representation-based and decoding- based LLID at the midpoint of the prompt and aggregate over a forward window of five positions. We evaluate on PUD sentence data (Zeman et al., 2017) and INCLUDE question-answering prompts (Romanou et al., 2025), using PUD variants to vary the language inventory and INCLUDE domains to test domain sensitivity. Exact dataset compo- sition and additional evaluation details are in Ap- pendix A.3. 3.3 LLID Estimators For open-ended evaluation, all estimators produce a layerwise distributionqover candidate languages. We compare: • repr: a representation-based GMM posterior fit to multilingual hidden states; • dec-rmax: decoding-based LLID using deter- ministic argmax rollouts from intermediate lay- ers; • dec-topp: decoding-based LLID using top-p rollouts. Fordec-rmaxanddec-topp, we generate short continuations from intermediate layers, pass them through GlotLID (Kargaran et al., 2023), and ag- gregate the resulting per-language scores intoq. We evaluate both the raw logitlens and tuned lens defined in §2.2, separating the rollout procedure from the projection used to map intermediate hid- den states into the output vocabulary. Fitting details for GMMs, tuned lenses, and GlotLID are in Ap- pendix C. 4 The Controlled Case: Latent Language Under Known Completions We begin by testing decoding-based LLID in the controlled setting, where the set of valid comple- tions is known across languages and we can di- rectly track when language-specific completions become decodable from intermediate layers. We track the summedSTART(w)probability assigned to language-specific tokens of the same concept across layers for three models ordered by increas- ing multilingual capability: Llama-2-7B, Aya-23- 8B, and Apertus-8B. Consistent with prior observa- tions on Llama-2-7B, we find that across all model families the decoded language signal is weak in early layers and becomes most visible in later lay- ers across translation (Fig. 3), copy (Fig. 7), and cloze tasks (Fig. 8). However, the sharpness of this signal differs substantially across models with dif- ferent degrees of multilinguality. The effect is also not necessarily uniform across tasks and languages within a model. Across models, the copy task yields sharper and higher late-layer probabilities for the observed lan- guage. This is expected: copying can rely on re- producing a surface form that is already present in the context. Cloze, on the other hand, requires Lang Entrp French Vocab Entrp 0102030 0.0 0.5 1.0 Llama-2-7B Turkish 0102030 Russian 0102030 Arabic 0102030 Hindi 0102030 Chinese 0102030 Lang Entrp Vocab Entrp 0102030 0.0 0.5 1.0 Aya-23-8B 01020300102030010203001020300102030 Lang Entrp Vocab Entrp 0102030 0.0 0.5 1.0 Apertus-8B 01020300102030010203001020300102030 0 1 2 3 Lang Entrp 0.0 2.5 5.0 7.5 10.0 12.5 Vocab Entrp Start(w) probability per latent language Layer Candidate Latent Languages: Arabic Bulgarian Czech German English Spanish Persian Finnish French Galician Hindi Indonesian Icelandic Italian Japanese Korean Marathi Polish Portuguese (Brazil) Russian Serbian Swedish Thai Turkish Ukrainian Urdu Chinese Candidate Latent Languages: Arabic Bulgarian Czech German English Spanish Persian Finnish French Galician Hindi Indonesian Icelandic Italian Japanese Korean Marathi Polish Portuguese (Brazil) Russian Serbian Swedish Thai Turkish Ukrainian Urdu Chinese latent lang == translation targettop-3 peak langs if not column lang Figure 3: Latent language probability rises in later layers in controlled translation; English is a stronger competing latent language in Llama-2 than in Aya-23 and Apertus. For each translation prompt and model layer, we measure the summed next-token probability of all tokens that can begin the corresponding answer in each candidate language (START(w)probability). Rows are models and columns group prompts with the same translation targets. English is excluded as a source or target, but retained as a candidate latent language. Upper strips show average latent language distribution entropy and average first-token vocabulary entropy. retrieving a concept and lexicalizing it in the appro- priate language 2 . Compared with copy, the cloze plots show lower peaks, more competing languages, and more diffuse probability mass, especially for Llama-2 and Aya-23. Even when English is excluded from the trans- lation direction (Fig. 3), Llama-2 frequently main- tains relatively high probability mass on English lexicalizations across layers, suggesting a stronger English-centered bias in its internal representations. This effect is weaker in Aya and further reduced in Apertus, where the target language probability rises relatively earlier and becomes more consistently dominant in later layers. Taken together,these results show that START(w)recovers a meaningful late-layer lan- guage signal in the controlled setting. The diagnos- tic is strongest when the task makes the relevant surface form directly available, as in copy prompts, and weaker when the model must retrieve and lex- icalize a concept, as in cloze prompts. The diag- nosis also varies systematically with model mul- tilinguality: less multilingual models more often retain high-probability English alternatives, while the more multilingual Apertus-8B shifts probability 2 Expressing an underlying concept using the words/forms of a particular language. toward the intended non-English target earlier. 5 Generalizing Decoding-based Estimates to Open-Ended Generation The controlled setting above provides a favorable testbed for theSTART(w)measurement as the rel- evant cross-lingual completions are known in ad- vance, so the diagnostic can compare a fixed set of lexical alternatives. However, forSTART(w)to support claims about latent language or multilin- gual pivoting, its signal should generalize beyond controlled settings with predefined candidate an- swers. We therefore ask whether similar language preferences can be recovered in open-ended next- token prediction, where the model generates con- tinuations in context rather than selecting among predefined translations. We extend the decoding diagnostic to open- ended generation by generating short continuations from intermediate hidden states under either the raw logitlens or tuned lens, using argmax or top- pdecoding, and then classifying the continuation language. Fig. 4 shows that the projection method changes how probability mass is allocated across languages. Raw logitlens decoding assigns sub- stantially more mass to English than the controlled START(w)setting, suggesting that English lexical- 0.0 0.2 0.4 0.6 Layers 50%-75% Start(w) 0.0 0.2 0.4 0.6 0.8 Raw Logitlens Top-pTunedLens Top-pRepr-GMM Llama2 Aya-23 EuroLLM EuroLLM-Instr Llama3.1 Nemo-Instr Llama3.1-Instr Apertus Apertus-Instr 0.0 0.2 0.4 0.6 Layers 75%-100% Llama2 Aya-23 EuroLLM EuroLLM-Instr Llama3.1 Nemo-Instr Llama3.1-Instr Apertus Apertus-Instr 0.0 0.2 0.4 0.6 0.8 Llama2 Aya-23 EuroLLM EuroLLM-Instr Llama3.1 Nemo-Instr Llama3.1-Instr Apertus Apertus-Instr Llama2 Aya-23 EuroLLM EuroLLM-Instr Llama3.1 Nemo-Instr Llama3.1-Instr Apertus Apertus-Instr Avg. Lang Prob across Prompts and Layers Controlled: translationOpen-ended: INCLUDE Avg. P(L=task-relevant)P(L=English)Avg. P(L=other)Avg. Max. P(L) Figure 4: Later-layer latent language estimates depend strongly on the probe: raw logitlens assigns more probability to English, while tuned lens and GMM estimates favor task-relevant languages. Rows aggregate the 50–75% and 75–100% layer windows. Left: translation prompts using normalizedSTART(w)probabilities. Right: open-ended INCLUDE prompts using raw logitlens top-p, tuned lens top-p, and representation-based GMM LLID. Bars pool prompt–layer–language probabilities within each category before averaging; diamonds show the average maximum probability within the task-relevant or other-language category. Error bars show SE. Cases with English as source or target language are excluded. izations remain accessible when the model is not restricted to known translations. For some mod- els, raw logitlens outputs also become difficult to interpret, given the representational drift between intermediate and final-layer hidden states (Belrose et al., 2025). The tuned lens shifts average probabil- ity toward the task-relevant language while average probability assigned to other languages remains low. This suggests that the tuned lens may partly collapse intermediate language evidence toward the final decoded language, consistent with the concern raised by Wendler et al. (2024). 6 Representation vs. Decoding: Different Evidence, Different LLID Behaviors Having established how decoding-based estimates vary across projections, rollouts, and task settings, we now compare them with representation-based LLID on the same hidden states. Representation- based LLID provides a complementary view of the same intermediate states examined by decoding- based probes. Instead of asking what language can be decoded from a hidden state, it asks which language-conditioned region of activation space the hidden state resembles. This distinction matters be- cause a hidden state may be geometrically closer to one language’s cluster without being decodable as that language, and conversely, an intermediate state may decode into a language whose representation- space signature is not dominant. Disagreement between the two estimators should therefore be treated as measurement information rather than as immediate diagnostic failure. Such disagreement provides evidence that the probes capture different aspects of multilingual processing. Comparing the rightmost column of Fig. 4 (Repr- GMM) with the decoding-based columns on the same open-ended INCLUDE prompts makes this contrast visible. The representation probe assigns substantially more probability mass to the task- relevant language and correspondingly less to En- glish than raw logitlens top-pdecoding does, par- ticularly in the 50–75% layer window. This gap is most pronounced for models with moderate multi- linguality (e.g., Llama-3.1), where decoding-based estimates still show a strong English bar while the representation-based estimate already favors the task languages. For the most multilingual models (e.g., Apertus, Apertus-Instruct), the average prob- abilities suggest closer agreement as both probes assign relatively little probability to English and substantial probability to the task-relevant language in late layers. However, the maximum-language estimates also reveal that, under the representation probe, a single non-task, non-English language can still receive more probability than the task-relevant language. These patterns suggest that the two esti- mator families are sensitive to different aspects of the same hidden states. The GMM representation probe reflects the geometric neighborhood of the 0102030 0.0 0.5 1.0 Repr-GMM Llama-2-7B 0102030 OLMo-2-1124-7B 0102030 Aya-23-8B 0102030 Apertus-8B 0102030 0.0 0.5 1.0 Raw LogitLens Top-p 010203001020300102030 Layer Pivot Rate / Dominance DominanceEnglish pivotOther-language pivotTotal pivot Figure 5: In open-ended INCLUDE, latent language dominance and pivoting vary across models and probes. Representation-based estimates exhibit model-specific changes across the layer stack, whereas decoding-based estimates generally shift later, depending on model multilinguality. For each prompt and layer, dominance is the highest probability assigned to a candidate latent language; a pivot occurs when the dominant language differs from the prompt language. Rows are LLID estimators and columns are models. Pivot rates are separated into English and other languages; the dashed line shows their sum. INCLUDE contains no English prompts. input language’s activation, while decoding-based estimators reflect the accessibility of language- specific tokens through the unembedding matrix, which retains a stronger English bias even when the representation probe assigns substantially less probability to English. The precise layerwise patterns are also model- dependent (Fig. 5). For the representation probe, Llama-2 exhibits a mixture of English- and other- language pivots, Aya-23 shows a pronounced but temporary increase in English-directed piv- ots, while Apertus-8B pivots are almost entirely to non-English languages. OLMo-2 shows another distinct pattern, maintaining high representation- probe dominance across nearly the entire layer stack. By contrast, dominance drops sharply near the start and recovers through the middle-to-late layers for Llama-2 and Aya-23, while Apertus-8B remains low until a late-layer rise. These patterns reflect again the different evi- dence used by the two probe families. The rep- resentation probe is fit directly to input-language activation distributions, so it is most confident where hidden states preserve a strong language- conditioned geometric signature. As states become more mixed or task-oriented, they are less cleanly assigned to a single fitted language distribution, producing the early dominance drop in Llama-2 and Aya-23 and the low early-to-middle dominance in Apertus. Their later recovery indicates that the states become concentrated again in a language- conditioned region. OLMo-2 is the exception, as its high dominance suggests a persistent separation of language-conditioned activation regions rather than a pronounced early-to-mid layer mixing phase. Decoding-based estimators, on the other hand, gen- erally change much later in the network. Across Domains.We next ask whether the model and estimator-specific trajectories in Fig. 5 persist across language inventories and QA subject do- mains. Fig. 16 repeats the dominance and pivot- rate analyses across PUD language-set variants and INCLUDE domains. For a given model and LLID estimator, the curves largely preserve the same lay- erwise shape across settings. Thus, domain and language inventory changes affect absolute levels less than model or diagnostic choice. The clear- est systematic exception is candidate-inventory size: PUD21 generally yields lower dominance and higher pivot rates than PUD9, as expected when probability mass is distributed over more possible languages. Across Pretraining. We ask whether latent lan- guage estimates appear suddenly, consolidate grad- ually, or drift as the model trains. Fig. 6 compares OLMo-2-7B and Apertus-8B across checkpoints (see Table 3 for checkpoint alignment), tracking representation and decoding dominance, confident pivot rate, and estimator agreement in early, mid- dle, and late layer bins. We use confident pivot rate to avoid treating small fluctuations in diffuse 0.0 0.2 0.4 0.6 0.8 1.0 OLMo-2-7B DominanceConfident pivot rateRepresentation-decoding agreement 0.00.20.40.60.81.0 0.0 0.2 0.4 0.6 0.8 1.0 Apertus-8B 0.00.20.40.60.81.00.00.20.40.60.81.0 Training Dynamics of LLID Methods on PUD21 Rows = model families, columns = diagnostics; confident pivot = pivot and dominance >= 0.6 Normalized training progress Layer bin:EarlyMiddleLateEstimator:Representation probeRaw LogitLens Top-p Figure 6: Training dynamics of latent language estimates across pretraining on PUD21. Rows are model families and columns are metrics. Colors denote early, middle, and late layer bins; solid and dashed lines denote repranddec-topp, respectively. Dominance is the average top-language probability; confident pivot rate is the fraction of prompts where a non-task language is dominant with probability at least 0.6; agreement measures whether the two probes select the same dominant language. Thex-axis is normalized training progress, spanning∼210B to 4T training tokens. Layer bins use normalized layer indexj/J: earlyj/J < 0.25, middle0.25≤ j/J < 0.75, and latej/J ≥ 0.75. Most checkpoint-dependent changes occur in late layers, while representation–decoding agreement remains low across training. LLID distributions as meaningful pivots, retain- ing only cases where a non-task language is dom- inant with probability at least 0.6. The two mod- els show qualitatively different trajectories. For OLMo, representation-based dominance is high and stable from the earliest checkpoint, while de- coding dominance consolidates over the first∼20– 30% of training; confident pivoting is low forrepr but moderate and slowly increasing fordec-topp, consistent with English-centric training where the representation space separates languages early but the decoding pathway retains an English bias. For Apertus, both estimators undergo a sharp transi- tion between roughly 20% and 40% of training, after which late-layer dominance reaches∼0.9. Representation–decoding agreement remains low throughout training for both models, and is espe- cially low for Apertus. These dynamics suggest that most of the checkpoint-dependent changes oc- cur in the late-layer bin for both models. Overall, the checkpoint analysis shows that representation- and decoding-based estimates of LLID behavior can change at different points in training. Base vs. Instruct. Finally, we ask whether in- struction tuning increases English dominance in LLID estimates. This hypothesis is plausible be- cause instruction data is often less multilingual than pretraining data, so instruction tuning might reori- ent intermediate states or decoded continuations toward English. In Fig. 4, base and instruction- tuned variants show only modest differences in their average LLID probabilities. The layerwise comparisons in the Appendix (Figs. 20–21) like- wise show broadly similar pivot and entropy tra- jectories, although some model-specific deviations remain. Contrary to our initial expectation, instruc- tion tuning produces little systematic shift toward English in LLID estimates, and its overall effects are considerably smaller than the changes observed across pretraining checkpoints. 7 Related Work Shared and language-specific multilingual structure. Multilingual models maintain both shared and language-specific structure in hidden states, through related but distinguishable sub- spaces (Chang et al., 2022), language-neutral sub- networks (Foroutan et al., 2022), and language- specific neurons or components (Zeng et al., 2025; Zhao et al., 2025). These findings provide a ba- sis for representation-based LLID, while highlight- ing that hidden states can simultaneously encode shared and language-specific structure. Latent pivots and output language. A second line of work studies whether multilingual models internally route computation through a pivot lan- guage. Decoding-based probes find English-pivot behavior in English-dominated models and dif- ferent pivoting patterns under non-English-centric training (Wendler et al., 2024; Zhong et al., 2025). Causal interventions further separate output lan- guage from conceptual content, suggesting that language form and meaning can be partially disen- tangled in the residual stream (Dumas et al., 2025). Our work differs by asking whether the probes used to infer such latent language behavior agree on the same hidden states. Training dynamics of multilingual representa- tions.Recent work studies how multilingual abil- ities emerge during training rather than only at the final checkpoint, showing that cross-lingual abil- ities (Blevins et al., 2022) and internal linguistic features (Bayazit et al., 2026) develop, stabilize, or change over pretraining. Accordingly, we ex- amine LLID across training checkpoints: if latent language estimates reflect model properties rather than diagnostic artifacts, they should vary system- atically with training progression. 8 Conclusion We reframed latent language identification as a measurement problem, comparing representation- based and decoding-based probes across models, training regimes, and tasks. The GMM represen- tation probe and decoding-based probes disagree systematically: the former identifies language struc- ture earlier and with weaker English bias, while decoding pathways retain sharper, more English- favored signals. These differences vary system- atically with multilinguality and training progres- sion, but their exact layerwise trajectories remain model and lens-dependent and cannot be predicted from multilinguality alone. Domain and language- inventory changes, by contrast, shift LLID esti- mates only modestly. Rather than confirming a sin- gle internal lingua franca, existing probes expose complementary aspects of multilingual computa- tion, showing that the choice of probe is itself part of the claim. Acknowledgements We thank Clara Meister, Negar Foroutan, Ange- lika Romanou, and Ayush K. Tarun for their help- ful discussions and feedback on our manuscript. We also gratefully acknowledge the support of the Swiss National Science Foundation (No. 215390), the AI2050 program at Schmidt Sciences (Grant #G-25-69783), Sony Group Corporation, and the Swiss National Supercomputing Center (CSCS) in the form of an infrastructure engineering and devel- opment project. This research is with support from Google.org and the Google Cloud Research Credits program for the Gemini Academic Program. This project is also funded by the European Union (ERC, RESPECT-LM, 101222478). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. Limitations Our findings come with several limitations. We evaluate decoder-only models in the 7–9B param- eter range, and our checkpoint analysis is neces- sarily restricted to OLMo-2 and Apertus, the two model families that release intermediate training checkpoints publicly. Whether probe disagreement persists at other scales, or for non-decoder architec- tures, remains an open question. The precise layer- wise LLID patterns also vary across model families, so the trajectories observed here should not be as- sumed to generalize unchanged to other models. We compare only two probe families, of which the representation-based family is instantiated only with the GMM estimator (Shani and Basirat, 2025), and do not examine causal interventions (Dumas et al., 2025) or neuron-level attribution (Zeng et al., 2025), which may align with either family or ex- pose a third aspect of multilingual processing. Be- cause latent language is inferred rather than ob- served (i.e., no ground truth), our work documents probe disagreement but cannot adjudicate which (if any) best reflects internal computation. Probe- specific choices (e.g., GMM components, the tuned lens fitting corpus, the open-ended anchor position, and the concept-word construction) may shift ab- solute LLID values, though we expect qualitative trends to be robust. Finally, our 27-language in- ventory excludes very low-resource languages and formal languages such as code and mathematics. We also leave open whether probe disagreement predicts downstream failure modes in cross-lingual transfer or generation. Ethics Statement This work uses publicly available pretrained lan- guage models and multilingual evaluation datasets (PUD/UD, INCLUDE, and Fineweb2), used in ac- cordance with their respective licenses. Our anal- ysis is diagnostic in nature and does not involve training new models, collecting human data, or de- ploying systems in user-facing settings. We see no direct ethical concerns arising from this work. References Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Kelly Marchisio, Se- bastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024. Aya 23: Open weight re- leases to further multilingual progress. Preprint, arXiv:2405.15032. Deniz Bayazit, Aaron Mueller, and Antoine Bosselut. 2026. Crosscoding through time: Tracking emer- gence & consolidation of linguistic representations throughout LLM pretraining. In Proceedings of the 64th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1353–1377, San Diego, California, United States. As- sociation for Computational Linguistics. Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Fur- man, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. 2025. Eliciting latent predic- tions from transformers with the tuned lens. Preprint, arXiv:2303.08112. Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2022. Analyzing the mono- and cross-lingual pretraining dynamics of multilingual language models. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 3575– 3590, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Tyler A. Chang, Zhuowen Tu, and Benjamin K. Bergen. 2022. The geometry of multilingual language model representations. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 119–136, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2025. Separat- ing tongue from thought: Activation patching reveals language-agnostic concept representations in trans- formers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 31822–31841, Vienna, Austria. Association for Computational Linguistics. Negar Foroutan, Mohammadreza Banaei, Rémi Lebret, Antoine Bosselut, and Karl Aberer. 2022. Discov- ering language-neutral sub-networks in multilingual language models. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 7560–7575, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics. Negar Foroutan, Paul Teiletche, Ayush Kumar Tarun, and Antoine Bosselut. 2025. Revisiting multilin- gual data mixtures in language model pretraining. Preprint, arXiv:2510.25947. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mi- tra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. AlejandroHernández-Cano,AlexanderHägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pásztor, Bettina Messmer, Dhia Garbaya, Eduard Frank ˇ Durech, Ido Hakimi, Juan Garcia Giraldo, Mete Ismayilzada, Negar Foroutan,Skander Moalla,Tiancheng Chen, Vinko Sabol ˇ cec, Yixuan Xu, Michael Aerni, Badr AlKhamissi, and 83 others. 2026. Apertus: Democratizing open and compliant LLMs for global language environments. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 46877–46955, San Diego, California, United States. Association for Computational Linguistics. Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. 2023. GlotLID: Language identification for low-resource languages. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2023, pages 6155–6218, Singapore. Association for Computational Linguistics. Pedro Henrique Martins, João Alves, Patrick Fernan- des, Nuno M. Guerreiro, Ricardo Rei, Amin Fara- jian, Mateusz Klimaszewski, Duarte M. Alves, José Pombal, Nicolas Boizard, Manuel Faysse, Pierre Colombo, François Yvon, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Mar- tins. 2025. Eurollm-9b: Technical report. Preprint, arXiv:2506.04079. nostalgebraist. 2020. Interpreting GPT: The logit lens. lesswrong, 2020. Shintaro Ozaki, Tatsuya Hiraoka, Hiroto Otake, Hiroki Ouchi, Masaru Isonuma, Benjamin Heinzerling, Ken- taro Inui, Taro Watanabe, Yusuke Miyao, Yohei Os- eki, and Yu Takagi. 2025. Do llms need to think in one language? correlation between latent language and task performance. Preprint, arXiv:2505.21458. Guilherme Penedo, Hynek Kydlí ˇ cek, Vinko Sabol ˇ cec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. Fineweb2: One pipeline to scale them all — adapting pre-training data processing to every language. In Second Confer- ence on Language Modeling. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learn- ers. OpenAI blog, 1(8):9. Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Ma- heshwary, Micol Altomare, Zeming Chen, Mohamed Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya K Dalmia, Abraham Diress, Sharad Duwal, and 38 others. 2025. Include: Eval- uating multilingual language understanding with regional knowledge. In International Conference on Learning Representations, volume 2025, pages 83291–83322. Lisa Schut, Yarin Gal, and Sebastian Farquhar. 2025. Do multilingual LLMs think in english? In ICLR 2025 Workshop on Building Trust in Language Mod- els and Applications. Nadav Shani and Ali Basirat. 2025. Language dom- inance in multilingual large language models. In Proceedings of the 8th BlackboxNLP Workshop: An- alyzing and Interpreting Neural Networks for NLP, pages 137–148, Suzhou, China. Association for Com- putational Linguistics. MistralAIteam.2024.Mistralnemo. https://mistral.ai/news/mistral-nemo.Accessed: 2026. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 oth- ers. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288. Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhan- dari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Fred- die Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction fine- tuned open-access multilingual language model. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 15894–15939, Bangkok, Thai- land. Association for Computational Linguistics. Evan Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Allyson Ettinger, and 23 others. 2025. 2 OLMo 2 furious (COLM’s ver- sion). In Second Conference on Language Modeling. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in English? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15366–15394, Bangkok, Thai- land. Association for Computational Linguistics. Daniel Zeman, Martin Popel, Milan Straka, Jan Ha- ji ˇ c, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Nedoluzhko, Silvie Cinková, Jan Haji ˇ c jr., Jaroslava Hlavá ˇ cová, Václava Kettnerová, Zde ˇ nka Urešová, and 43 others. 2017. CoNLL 2017 shared task: Mul- tilingual parsing from raw text to Universal Depen- dencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Univer- sal Dependencies, pages 1–19, Vancouver, Canada. Association for Computational Linguistics. Hongchuan Zeng, Senyu Han, Lu Chen, and Kai Yu. 2025. Converging to a lingua franca: Evolution of linguistic regions and semantics alignment in mul- tilingual large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10602–10617, Abu Dhabi, UAE. Association for Computational Linguistics. Weixiang Zhao, Jiahe Guo, Yang Deng, Tongtong Wu, Wenxuan Zhang, Yulin Hu, Xingyu Sui, Yanyan Zhao, Wanxiang Che, Bing Qin, Tat-Seng Chua, and Ting Liu. 2025. When less language is more: Language-reasoning disentanglement makes llms bet- ter multilingual reasoners. In Advances in Neural Information Processing Systems, volume 38, Main Conference, pages 38608–38642. Curran Associates, Inc. Chengzhi Zhong, Qianying Liu, Fei Cheng, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki, and Sadao Kurohashi. 2025. What language do non- English-centric large language models think in? In Findings of the Association for Computational Lin- guistics: ACL 2025, pages 26333–26346, Vienna, Austria. Association for Computational Linguistics. Setting# Langs# EntriesLanguages PUD99900 train + 900 held-outArabic (ar), Czech (cs), English (en), French (fr), Hindi (hi), Icelandic (is), Indonesian (id), Portuguese (pt), Spanish (es) PUD2121 2,100 train + 2,100 held-outArabic (ar), Russian (ru), Hindi (hi), Chinese (zh), Korean (ko), Japanese (ja), Indonesian (id), German (de), English (en), Icelandic (is), Swedish (sv), Spanish (es), French (fr), Galician (gl), Italian (it), Portuguese (pt), Czech (cs), Polish (pl), Turkish (tr), Finnish (fi), Thai (th) UD6 extension6600 train + 600 held-outUkrainian (uk), Bulgarian (bg), Serbian (sr), Urdu (ur), Persian (fa), Marathi (mr) INCLUDE-1010882 promptsArabic (90), Spanish (90), Finnish (90), French (81), Hindi (90), Indonesian (90), Portuguese (90), Russian (90), Turkish (90), Chinese (81) Copy/cloze6414 prompts per taskArabic (ar), Hindi (hi), Chinese (zh), Russian (ru), English (en), French (fr) Translation916,146 promptsArabic (ar), Spanish (es), Finnish (fi), French (fr), Hindi (hi), Indonesian (id), Russian (ru), Turkish (tr), Chinese (zh) Table 1: Language inventories used in the experiments. PUD and UD rows are balanced at 100 train and 100 held-out prompts per language. PUD9+UD6 and PUD21+UD6 add the UD6 extension to the corresponding PUD inventory. The INCLUDE subset is capped at 30 prompts per language-domain cell, but French and Chinese have fewer total prompts because their STEM cells contain only 21 selected examples. Target-string copy and cloze counts are for the six-language main-plot subset; translation counts aggregate the nine target-specific runs. A Dataset Details In this section we describe the datasets and the pre- processing steps. Note that we did not collect these datasets ourselves; they may contain personally identifying or offensive content. A.1 Language Inventory Table 1 gives the language inventories used across the controlled target-string runs, PUD runs, and INCLUDE runs. In the main text, we use the short- hand PUD9, PUD21, PUD9+UD6, PUD21+UD6 to avoid repeatedly listing these languages. Entry counts refer to prepared prompt rows; for PUD and UD settings, we report the deterministic train and held-out splits separately. The PUD9+UD6 and PUD21+UD6 settings are formed by adding the UD6 extension row to PUD9 or PUD21. A.2 Controlled Target-String Construction The controlled copy, cloze, and translation prompts are derived from the target-word setup of prior la- tent language studies (Wendler et al., 2024; Dumas et al., 2025; Zhong et al., 2025). Each example has a known concept completion in multiple languages, and the logitlens query is made at the final task token, before the answer begins. For instruction- tuned models, formatting may append assistant- control tokens, so we anchor scoring to the end of the original task prefix rather than the final format- ted token. We first build a common-concept grid and then derive copy, cloze, and directed translation prompts from that grid. Copy and translation prompts use four completed in-context examples followed by a query line; cloze prompts use two completed cloze demonstrations followed by the query context. For translation and cloze settings, word translations and cloze contexts are constructed jointly so that the context disambiguates homonyms. We filter examples where the translated cloze answer does not align with the independently translated concept word. We do not remove examples solely because the English and target language forms share a tok- enizer prefix. For each remaining promptx, candidate lan- guageℓ, and model tokenizerτ, we construct START(w), the set of vocabulary tokens that can begin the language-ℓanswerw ℓ,x , as defined in §3.2. This set includes vocabulary tokens with tokenizer-specific prefixes and whitespace variants when applicable. For tokenizers with byte-fallback behavior, we also include valid byte-level starting tokens. The corresponding scores x (ℓ), also defined in §3.2, is the summed next-token probability as- signed to these valid starting tokens. We use this raw summed mass for the probability curves. When computing distributional metrics such as entropy, dominance, or pivot rate, we normalize these scores over the candidate language set: q x (ℓ) = s x (ℓ) P ℓ ′ ∈L s x (ℓ ′ ) The language entropy is computed from this nor- malized distributionq x , while vocabulary entropy is computed over the model’s next-token vocabu- lary distribution. A.3 Open-Ended Evaluation Details For natural-language data without constrained com- pletions, both representation-based and decoding- based LLID are evaluated at a matched prompt anchor. Unless otherwise stated, we use the mid- point of the prompt and aggregate over a forward window of five positions. Decoding-based open- ended LLID uses either deterministic argmax roll- outs or top-prollouts withp = 0.9and five sam- ples. We score each decoded continuation with GlotLID (Kargaran et al., 2023), collapse its labels to the candidate-language inventory, and normalize the resulting distribution. For top-prollouts, we average these distributions uniformly across sam- ples to obtainq. We use PUD as sentence-level natural-language data. PUD21 is the broadest set- ting, while PUD9 follows the language subset used by Shani and Basirat (2025). We also evaluate PUD9+UD6 and PUD21+UD6 by adding six Uni- versal Dependencies languages selected for script or family overlap. For domain analyses, we use INCLUDE question-answering prompts in 10 lan- guages and three domains: Social Science, Arts & Humanities, and STEM. B Model Details Table 2 reports all evaluated models and multilin- guality proxies. Since direct pretraining mixtures are not fully available, we use zero-shot QA accu- racy on INCLUDE, cross-lingual perplexity, and byte-normalized likelihood as external proxies for multilingual capability. For the training dynamics analysis, we align OLMo-2 and Apertus check- points as shown in Table 3. The INCLUDE score is zero-shot accu- racy fromlm-evaluation-harnessruns on include-base-44, macro-averaged over 44 lan- guage groups. We use it to sort the models because it directly measures multilingual task performance. The PUD21 line NLL is a likelihood-based aux- iliary score on the PUD21 parallel sentence set. We compute the autoregressive negative log-likelihood of each teacher-forced line, convert it to bits, and average across PUD-only parallel rows in the 21 languages. Lower is better. The PUD21+UD6 BPB score is a byte- Model INCLUDE 0-shot macro (↑) PUD21 line NLL (↓) PUD21+UD6 BPB (↓) Apertus-8B-Instruct55.0169.81.19 Apertus-8B (Hernández-Cano et al., 2026)53.1138.30.97 Llama-3.1-8B-Instruct52.8152.91.09 Mistral-Nemo-Instruct-2407 (team, 2024)52.2166.71.20 Llama-3.1-8B (Grattafiori et al., 2024)48.9148.81.05 EuroLLM-9B-Instruct47.8156.01.17 EuroLLM-9B (Martins et al., 2025)43.3149.21.11 Aya-23-8B (Aryabumi et al., 2024)40.1172.01.26 OLMo-2-1124-7B (Walsh et al., 2025)33.8181.11.30 Llama-2-7B (Touvron et al., 2023)27.7170.01.22 GPT-2 (Radford et al., 2019)26.2365.42.62 GPT-2-XL25.9302.32.20 Table 2: Multilingual Performance. Models are sorted by decreasing zero-shot macro accuracy on INCLUDE. Line NLL is an auxiliary likelihood score computed on a parallel corpus across languages. BPB is likeli- hood normalized by UTF-8 byte count, but it remains script-sensitive because byte counts differ across writing systems. Best is bold; second-best is underlined. Alignment IdxOLMo-2 tokensApertus tokens 1210B210B 2625B630B 31.04T1.05T 41.46T1.47T 51.87T1.89T 62.29T2.31T 72.70T2.73T 83.12T3.15T 93.53T3.57T 103.95T3.99T Table 3: Token-aligned checkpoints, used in the OLMo-2 and Apertus training dynamics comparison. normalized likelihood on the broader PUD21+UD6 corpus. We sum token-level negative log-likelihood over all evaluated lines, convert from nats to bits, and divide by the total number of UTF-8 bytes: BPB = P x − log 2 p θ (x) P x |utf8(x)| This reduces, but does not eliminate, tokenization and script effects; lower BPB is better. C Training Details Hardware, Packages & Artifacts. We run ex- periments on NVIDIA A100 80GB GPUs, using one GPU at a time. Each run lasts between one to six hours depending on the dataset size. We use Python 3.10 and use transformers 3 , nnsight 4 , and datasets 5 packages. We used AI-assisted tools to support code development, and we reviewed and tested the resulting code. For writing, AI tools were used to rephrase the wording of the manuscript. 3 https://huggingface.co/docs/transformers 4 https://nnsight.net/ 5 https://huggingface.co/datasets Lang Entrp English Vocab Entrp 0102030 0.0 0.5 1.0 Llama-2-7B French 0102030 Turkish 0102030 Russian 0102030 Arabic 0102030 Hindi 0102030 Chinese 0102030 Lang Entrp Vocab Entrp 0102030 0.0 0.5 1.0 Aya-23-8B 010203001020300102030010203001020300102030 Lang Entrp Vocab Entrp 0102030 0.0 0.5 1.0 Apertus-8B 010203001020300102030010203001020300102030 0 1 2 3 Lang Entrp 0.0 2.5 5.0 7.5 10.0 12.5 Vocab Entrp [Copy task] Start(w) probability across model layer Rows = models, Columns = prompt language Start(w) probability per latent language Layer Candidate Latent Languages: Arabic Bulgarian Czech German English Spanish Persian Finnish French Galician Hindi Indonesian Icelandic Italian Japanese Korean Marathi Polish Portuguese (Brazil) Russian Serbian Swedish Thai Turkish Ukrainian Urdu Chinese Candidate Latent Languages: Arabic Bulgarian Czech German English Spanish Persian Finnish French Galician Hindi Indonesian Icelandic Italian Japanese Korean Marathi Polish Portuguese (Brazil) Russian Serbian Swedish Thai Turkish Ukrainian Urdu Chinese latent lang == prompt langtop-3 peak langs if not column lang Figure 7: Copy prompts: latent language probability across model layers. For each copy prompt and model layer, we measure the summed next-token probability of all tokens that can begin the corresponding answer in each candidate language (START(w)probability). Rows are models and columns group prompts with the same prompt language. Upper strips show average latent language distribution entropy and average first-token vocabulary entropy. GMM Fitting. Representation-based LLID re- quires a language-conditioned model of activation space. We fit one per-layer Gaussian model per can- didate language from hidden states collected on the train split. Each GMM setup uses at most 100 train- ing prompts per language, uniform language priors, layerwise PCA retaining 98% explained variance, and diagonal covariance. Unless otherwise stated, we fit token-level hidden states and use the surface- token alignment from the dataset preprocessing to select comparable positions at evaluation time. We fit separate GMMs per candidate-language in- ventory:pud9,pud21,pud9_ud6,pud21_ud6, and include_10lang. For INCLUDE plots that con- tain English as a possible latent language, we fit with the same PUD calibration source but add En- glish to the candidate-language set. Tuned Lens Fitting. For decoding-based LLID, we compare raw logitlens projections with tuned lens projections (Belrose et al., 2025). Tuned lenses are fit on multilingual FineWeb2 data (Penedo et al., 2025), keeping the fitting corpus indepen- dent of PUD, INCLUDE, and controlled target- string evaluation prompts. Our main tuned lens artifacts use the 27-language inventory. For each model, FineWeb2 rows are tokenized into win- dows of length up to 2048 tokens, subject to the model’s supported context length. Training exam- ples are balanced by strict round-robin over lan- guages, dropping excess rows from higher-resource languages. We train for one epoch with at most 1000 optimization steps, AdamW with learning rate 10 −3 and weight decay 0, temperature 1.0, identity and bias regularization weights10 −4 , and bfloat16 autocast. Validation uses a lightweight balanced subset, and checkpoints are written periodically. We do not train a tuned lens translator for the final transformer layer; at runtime, the final layer uses the raw model-head projection. D Additional Results The appendix figures below are organized by anal- ysis group. For the controlled synthetic tasks, Figs. 7–11 show layerwiseSTART(w)probability curves, while Figs. 12–14 summarize the same task family into language-probability categories across layer windows. For open-ended language probabil- ities, Fig. 15 gives the PUD21+UD6 counterpart. For training analyses, Figs. 17–19 show checkpoint dynamics and full-layer trajectories, while Figs. 20– 21 compare base and instruction-tuned model vari- ants. Lang Entrp English Vocab Entrp 0102030 0.0 0.5 1.0 Llama-2-7B French 0102030 Turkish 0102030 Russian 0102030 Arabic 0102030 Hindi 0102030 Chinese 0102030 Lang Entrp Vocab Entrp 0102030 0.0 0.5 1.0 Aya-23-8B 010203001020300102030010203001020300102030 Lang Entrp Vocab Entrp 0102030 0.0 0.5 1.0 Apertus-8B 010203001020300102030010203001020300102030 0 1 2 3 Lang Entrp 0.0 2.5 5.0 7.5 10.0 12.5 Vocab Entrp [Cloze task] Start(w) probability across model layer Rows = models, Columns = prompt language Start(w) probability per latent language Layer Candidate Latent Languages: Arabic Bulgarian Czech German English Spanish Persian Finnish French Galician Hindi Indonesian Icelandic Italian Japanese Korean Marathi Polish Portuguese (Brazil) Russian Serbian Swedish Thai Turkish Ukrainian Urdu Chinese Candidate Latent Languages: Arabic Bulgarian Czech German English Spanish Persian Finnish French Galician Hindi Indonesian Icelandic Italian Japanese Korean Marathi Polish Portuguese (Brazil) Russian Serbian Swedish Thai Turkish Ukrainian Urdu Chinese latent lang == prompt langtop-3 peak langs if not column lang Figure 8: Cloze prompts: latent language probability across model layers. For each cloze prompt and model layer, we measure the summed next-token probability of all tokens that can begin the corresponding answer in each candidate language (START(w)probability). Rows are models and columns group prompts with the same prompt language. Upper strips show average latent language distribution entropy and average first-token vocabulary entropy. Lang Entrp English Vocab Entrp 0102030 0.0 0.5 1.0 Llama-2-7B French 0102030 Turkish 0102030 Russian 0102030 Arabic 0102030 Hindi 0102030 Chinese 0102030 Lang Entrp Vocab Entrp 0102030 0.0 0.5 1.0 Aya-23-8B 010203001020300102030010203001020300102030 Lang Entrp Vocab Entrp 0102030 0.0 0.5 1.0 Apertus-8B 010203001020300102030010203001020300102030 0 1 2 3 Lang Entrp 0.0 2.5 5.0 7.5 10.0 12.5 Vocab Entrp [Translation task] Start(w) probability across model layer Rows = models, Columns = target translation language (all cases) Start(w) probability per latent language Layer Candidate Latent Languages: Arabic Bulgarian Czech German English Spanish Persian Finnish French Galician Hindi Indonesian Icelandic Italian Japanese Korean Marathi Polish Portuguese (Brazil) Russian Serbian Swedish Thai Turkish Ukrainian Urdu Chinese Candidate Latent Languages: Arabic Bulgarian Czech German English Spanish Persian Finnish French Galician Hindi Indonesian Icelandic Italian Japanese Korean Marathi Polish Portuguese (Brazil) Russian Serbian Swedish Thai Turkish Ukrainian Urdu Chinese latent lang == translation targettop-3 peak langs if not column lang Figure 9: Translation prompts: latent language probability across model layers, grouped by translation target. For each translation prompt and model layer, we measure the summed next-token probability of all tokens that can begin the corresponding answer in each candidate language (START(w)probability). Rows are models and columns group prompts with the same translation target. English source and target cases are included. Upper strips show average latent language distribution entropy and average first-token vocabulary entropy. Lang Entrp English Vocab Entrp 0102030 0.0 0.5 1.0 Llama-2-7B French 0102030 Turkish 0102030 Russian 0102030 Arabic 0102030 Hindi 0102030 Chinese 0102030 Lang Entrp Vocab Entrp 0102030 0.0 0.5 1.0 Aya-23-8B 010203001020300102030010203001020300102030 Lang Entrp Vocab Entrp 0102030 0.0 0.5 1.0 Apertus-8B 010203001020300102030010203001020300102030 0 1 2 3 Lang Entrp 0.0 2.5 5.0 7.5 10.0 12.5 Vocab Entrp [Translation task] Start(w) probability across model layer Rows = models, Columns = source language (all cases) Start(w) probability per latent language Layer Candidate Latent Languages: Arabic Bulgarian Czech German English Spanish Persian Finnish French Galician Hindi Indonesian Icelandic Italian Japanese Korean Marathi Polish Portuguese (Brazil) Russian Serbian Swedish Thai Turkish Ukrainian Urdu Chinese Candidate Latent Languages: Arabic Bulgarian Czech German English Spanish Persian Finnish French Galician Hindi Indonesian Icelandic Italian Japanese Korean Marathi Polish Portuguese (Brazil) Russian Serbian Swedish Thai Turkish Ukrainian Urdu Chinese latent lang == translation sourcetop-3 peak langs if not column lang Figure 10: Translation prompts: latent language probability across model layers, grouped by translation source. For each translation prompt and model layer, we measure the summed next-token probability of all tokens that can begin the corresponding answer in each candidate language (START(w)probability). Rows are models and columns group prompts with the same translation source. English source and target cases are included. Upper strips show average latent language distribution entropy and average first-token vocabulary entropy. Lang Entrp French Vocab Entrp 0102030 0.0 0.5 1.0 Llama-2-7B Turkish 0102030 Russian 0102030 Arabic 0102030 Hindi 0102030 Chinese 0102030 Lang Entrp Vocab Entrp 0102030 0.0 0.5 1.0 Aya-23-8B 01020300102030010203001020300102030 Lang Entrp Vocab Entrp 0102030 0.0 0.5 1.0 Apertus-8B 01020300102030010203001020300102030 0 1 2 3 Lang Entrp 0.0 2.5 5.0 7.5 10.0 12.5 Vocab Entrp [Translation task] Start(w) probability across model layer Rows = models, Columns = source language (English cases excluded) Start(w) probability per latent language Layer Candidate Latent Languages: Arabic Bulgarian Czech German English Spanish Persian Finnish French Galician Hindi Indonesian Icelandic Italian Japanese Korean Marathi Polish Portuguese (Brazil) Russian Serbian Swedish Thai Turkish Ukrainian Urdu Chinese Candidate Latent Languages: Arabic Bulgarian Czech German English Spanish Persian Finnish French Galician Hindi Indonesian Icelandic Italian Japanese Korean Marathi Polish Portuguese (Brazil) Russian Serbian Swedish Thai Turkish Ukrainian Urdu Chinese latent lang == translation sourcetop-3 peak langs if not column lang Figure 11: Translation prompts: latent language probability across model layers, grouped by translation source. For each translation prompt and model layer, we measure the summed next-token probability of all tokens that can begin the corresponding answer in each candidate language (START(w)probability). Rows are models and columns group prompts with the same translation source. English is excluded as a source or target, but retained as a candidate latent language. Upper strips show average latent language distribution entropy and average first-token vocabulary entropy. Llama-2-7B Aya-23-8B EuroLLM-9B EuroLLM-9B-Instruct Llama-3.1-8B Mistral-Nemo-Instruct Llama-3.1-8B-Instruct Apertus-8B Apertus-8B-Instruct 0.0 0.2 0.4 0.6 0.8 First 50% Layers Llama-2-7B Aya-23-8B EuroLLM-9B EuroLLM-9B-Instruct Llama-3.1-8B Mistral-Nemo-Instruct Llama-3.1-8B-Instruct Apertus-8B Apertus-8B-Instruct 50%-75% Layers Llama-2-7B Aya-23-8B EuroLLM-9B EuroLLM-9B-Instruct Llama-3.1-8B Mistral-Nemo-Instruct Llama-3.1-8B-Instruct Apertus-8B Apertus-8B-Instruct 75%-100% Layers [Copy task] Language Probabilities Across Layer Windows English prompt cases are excluded. Bars pool prompt-layer-language probabilities within each category; diamonds average prompt-layer category maxima; error bars show SE for both. Avg. Lang Prob across Prompts and Layers Avg. P(L=task-relevant)P(L=English)Avg. P(L=other)Avg. Max. P(L) Figure 12: Copy prompts: latent language probabilities across layer windows. Columns show the first 50%, 50–75%, and 75–100% of layers. Bars pool individual prompt–layer–language probabilities within each category before averaging; diamonds show the average maximum probability within the task-relevant or other-language category. English prompts are excluded, but English remains a candidate latent language. Error bars show SE. Llama-2-7B Aya-23-8B EuroLLM-9B EuroLLM-9B-Instruct Llama-3.1-8B Mistral-Nemo-Instruct Llama-3.1-8B-Instruct Apertus-8B Apertus-8B-Instruct 0.0 0.2 0.4 0.6 First 50% Layers Llama-2-7B Aya-23-8B EuroLLM-9B EuroLLM-9B-Instruct Llama-3.1-8B Mistral-Nemo-Instruct Llama-3.1-8B-Instruct Apertus-8B Apertus-8B-Instruct 50%-75% Layers Llama-2-7B Aya-23-8B EuroLLM-9B EuroLLM-9B-Instruct Llama-3.1-8B Mistral-Nemo-Instruct Llama-3.1-8B-Instruct Apertus-8B Apertus-8B-Instruct 75%-100% Layers [Cloze task] Language Probabilities Across Layer Windows English prompt cases are excluded. Bars pool prompt-layer-language probabilities within each category; diamonds average prompt-layer category maxima; error bars show SE for both. Avg. Lang Prob across Prompts and Layers Avg. P(L=task-relevant)P(L=English)Avg. P(L=other)Avg. Max. P(L) Figure 13: Cloze prompts: latent language probabilities across layer windows. Same layout and aggregation as Fig. 12, evaluated on cloze prompts. English prompt cases are excluded. Llama-2-7B Aya-23-8B EuroLLM-9B EuroLLM-9B-Instruct Llama-3.1-8B Mistral-Nemo-Instruct Llama-3.1-8B-Instruct Apertus-8B Apertus-8B-Instruct 0.0 0.2 0.4 0.6 First 50% Layers Llama-2-7B Aya-23-8B EuroLLM-9B EuroLLM-9B-Instruct Llama-3.1-8B Mistral-Nemo-Instruct Llama-3.1-8B-Instruct Apertus-8B Apertus-8B-Instruct 50%-75% Layers Llama-2-7B Aya-23-8B EuroLLM-9B EuroLLM-9B-Instruct Llama-3.1-8B Mistral-Nemo-Instruct Llama-3.1-8B-Instruct Apertus-8B Apertus-8B-Instruct 75%-100% Layers [Translation task] Language Probabilities Across Layer Windows Cases with English as a source or target language are excluded. Bars pool prompt-layer-language probabilities within each category; diamonds average prompt-layer category maxima; error bars show SE for both. Avg. Lang Prob across Prompts and Layers Avg. P(L=task-relevant)P(L=English)Avg. P(L=other)Avg. Max. P(L) Figure 14: Translation prompts: latent language probabilities across layer windows. Same layout and aggregation as Fig. 12. For translation, the task-relevant category contains both the translation source and target language. English is excluded as a source or target, but retained as a candidate latent language. 0.00 0.25 0.50 0.75 1.00 raw-rmax First 50% Layers50%-75% Layers75%-100% Layers 0.00 0.25 0.50 0.75 1.00 raw-rtopp 0.00 0.25 0.50 0.75 1.00 tuned-rmax 0.00 0.25 0.50 0.75 1.00 tuned-rtopp GPT-2 XL GPT-2 Llama-2-7B OLMo-2-1124-7B Aya-23-8B EuroLLM-9B EuroLLM-9B-Instruct Llama-3.1-8B Mistral-Nemo-Instruct Llama-3.1-8B-Instruct Apertus-8B Apertus-8B-Instruct 0.00 0.25 0.50 0.75 1.00 repr GPT-2 XL GPT-2 Llama-2-7B OLMo-2-1124-7B Aya-23-8B EuroLLM-9B EuroLLM-9B-Instruct Llama-3.1-8B Mistral-Nemo-Instruct Llama-3.1-8B-Instruct Apertus-8B Apertus-8B-Instruct GPT-2 XL GPT-2 Llama-2-7B OLMo-2-1124-7B Aya-23-8B EuroLLM-9B EuroLLM-9B-Instruct Llama-3.1-8B Mistral-Nemo-Instruct Llama-3.1-8B-Instruct Apertus-8B Apertus-8B-Instruct [PUD21 + UD6] Language Probabilities Across Layer Windows Rows = LLID estimator, columns = layer window. English prompt cases are excluded; bars pool prompt-layer-language probabilities within each category; diamonds average prompt-layer category maxima; error bars show SE for both. Avg. Lang Prob across Prompts and Layers Avg. P(L=task-relevant)P(L=English)Avg. P(L=other)Avg. Max. P(L) Figure 15: Open-ended PUD21+UD6: latent language probabilities across layer windows. Rows are LLID estimators and columns are layer windows. Bars, diamonds, and error bars use the same aggregation as Fig. 12. The task-relevant category is the prompt language. English prompts are excluded, but English remains a candidate latent language. 0.0 0.5 1.0 Llama-2-7B reprraw-rmaxraw-rtopp 0.0 0.5 1.0 OLMo-2-1124-7B 0.0 0.5 1.0 Aya-23-8B 020 0.0 0.5 1.0 Apertus-8B 020020 Layer Dominance PUD:PUD9PUD9 + UD6PUD21PUD21 + UD6 INCLUDE:Arts/HumanitiesSocial SciencesSTEM (a) Dominance 0.0 0.5 1.0 Llama-2-7B reprraw-rmaxraw-rtopp 0.0 0.5 1.0 OLMo-2-1124-7B 0.0 0.5 1.0 Aya-23-8B 020 0.0 0.5 1.0 Apertus-8B 020020 Layer Pivot Rate PUD:PUD9PUD9 + UD6PUD21PUD21 + UD6 INCLUDE:Arts/HumanitiesSocial SciencesSTEM (b) Pivot rate Figure 16: Latent language dominance and pivoting are comparatively stable across domains and language inventories. Rows are language models and columns are LLID methods. For each prompt and layer, dominance is the highest probability assigned to a candidate latent language; a pivot occurs when the dominant language differs from the prompt language. Curves largely cluster across PUD variants (PUD9, PUD9+UD6, PUD21, PUD21+UD6) and INCLUDE domains (Arts/Humanities, Social Sciences, and STEM) within each model–estimator panel, while the layerwise trajectories can differ substantially across models. 0.0 0.2 0.4 0.6 0.8 1.0 OLMo-2-7B DominanceConfident pivot rateRepresentation-decoding agreement 0.00.20.40.60.81.0 0.0 0.2 0.4 0.6 0.8 1.0 Apertus-8B 0.00.20.40.60.81.00.00.20.40.60.81.0 Training Dynamics of LLID Methods on PUD21 Rows = model families, columns = diagnostics; confident pivot = pivot and dominance >= 0.6 Normalized training progress Layer bin:Q1Q2Q3Q4Estimator:Representation probeRaw LogitLens Top-p Figure 17: Training dynamics of latent language estimates across pretraining on PUD21 with four layer bins. This is the quartile-binned counterpart to Fig. 6, using the same rows, metrics, estimator overlays, and prompt-level definitions. Colors divide the normalized layer index j/J into quartiles: Q1 has j/J < 0.25, Q2 has 0.25≤ j/J < 0.50, Q3 has 0.50≤ j/J < 0.75, and Q4 has j/J ≥ 0.75. 0.0 0.2 0.4 0.6 0.8 1.0 OLMo-2-7B Pivot Rate 0.0 0.2 0.4 0.6 0.8 1.0 Dominance 0.0 0.1 0.2 0.3 0.4 Entropy 0102030 0.0 0.2 0.4 0.6 0.8 1.0 Apertus-8B 0102030 0.0 0.2 0.4 0.6 0.8 1.0 0102030 0 1 2 3 210B 1.2T 2.1T 3T 4T Training tokens [PUD21 checkpoint dynamics: GMM representation] LLID metrics across model layers Rows = models, Columns = metrics Layer Figure 18: Representation-based LLID checkpoint trajectories across layers on PUD21. Rows are model families and columns are layerwise metrics. Each curve is one token-aligned checkpoint, colored by its number of training tokens. Metrics are averaged over prompts at each layer. Unlike Fig. 6, this view keeps the full layer axis instead of aggregating layers into bins. 0.0 0.2 0.4 0.6 0.8 1.0 OLMo-2-7B Pivot Rate 0.0 0.2 0.4 0.6 0.8 1.0 Dominance 0.0 0.2 0.4 0.6 Entropy 0102030 0.0 0.2 0.4 0.6 0.8 1.0 Apertus-8B 0102030 0.0 0.2 0.4 0.6 0.8 1.0 0102030 0.0 0.5 1.0 1.5 2.0 210B 1.2T 2.1T 3T 4T Training tokens [PUD21 checkpoint dynamics: Raw logit-lens argmax decoding] LLID metrics across model layers Rows = models, Columns = metrics Layer Figure 19: Decoding-based LLID checkpoint trajectories across layers on PUD21. Same layout as Fig. 18, using the raw logitlens argmax decoding probe instead of the representation probe. 02040 0.0 0.5 1.0 EuroLLM Repr-GMM 02040 Raw LogitLens Top-p 02040 0.0 0.5 1.0 Llama-3.1 02040 02040 0.0 0.5 1.0 Apertus 02040 02040 0 2 Repr-GMM 02040 Raw LogitLens Top-p 02040 0 2 02040 02040 0 2 02040 02040 0.0 0.5 1.0 Repr-Decoding 02040 0.0 0.5 1.0 02040 0.0 0.5 1.0 Pivot rate Layer Entropy Layer Agreement Layer BaseInstruct Figure 20: Base-vs-instruct LLID trajectories on PUD21. Rows are model families. Columns show pivot rate underreprand raw logitlensdec-topp, entropy under the same two estimators, and their dominant language agreement. Blue solid and orange dashed curves denote base and instruction-tuned variants, respectively. Agreement is the fraction of matched prompt–layer pairs for which the two probes select the same dominant language. 02040 0.0 0.5 1.0 EuroLLM Repr-GMM 02040 Raw LogitLens Top-p 02040 0.0 0.5 1.0 Llama-3.1 02040 02040 0.0 0.5 1.0 Apertus 02040 02040 0 1 2 Repr-GMM 02040 Raw LogitLens Top-p 02040 0 1 2 02040 02040 0 1 2 02040 02040 0.0 0.5 1.0 Repr-Decoding 02040 0.0 0.5 1.0 02040 0.0 0.5 1.0 Pivot rate Layer Entropy Layer Agreement Layer BaseInstruct Figure 21: Base-vs-instruct LLID trajectories on INCLUDE-10 with English as a candidate language. Same layout as Fig. 20, evaluated on INCLUDE-10. INCLUDE contains no English prompts, but English is retained as a candidate latent language.