Paper deep dive
MixLID: Token-Level Language Identification in Mixed-Language Text
Seongtae Hong, Youngjoon Jang, Seungyoon Lee, Jungseob Lee, Heuiseok Lim
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Processing multilingual text requires both identifying a broad range of languages and distinguishing the language of each token in inputs containing multiple languages. We introduce MixLID, a token-level language identification model supporting 111 language-script combinations. MixLID follows a two-stage training process: it first learns to distinguish individual languages using the language labels of monolingual examples, then continues training on mixed-language data. The second stage uses concatenated examples from existing code-switching data alongside augmented examples created through word substitution based on correspondences between English source words and loanwords. For monolingual identification, MixLID outperforms all baselines in both accuracy and macro F1 on four benchmarks within the language coverage shared with each baseline. For mixed-language identification, it outperforms the baselines in identifying the language of individual tokens on every evaluated language pair. MixLID achieves high performance across a broad range of languages in both monolingual and mixed-language settings. The model is publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2610.16601v1
- Canonical: https://arxiv.org/abs/2610.16601v1
Trouble viewing inline? Open PDF directly â
Full Text
48,880 characters extracted from source content.
Expand or collapse full text
MixLID: Token-Level Language Identification in Mixed-Language Text Seongtae Hong, Youngjoon Jang, Seungyoon Lee, Jungseob Lee, Heuiseok Lim * Department of Computer Science and Engineering, Korea University ghdchlwls123@korea.ac.kr Abstract Processing multilingual text requires both iden- tifying a broad range of languages and distin- guishing the language of each token in inputs containing multiple languages. We introduce MixLID, a token-level language identification model supporting 111 languageâscript com- binations. MixLID follows a two-stage train- ing process: it first learns to distinguish indi- vidual languages using the language labels of monolingual examples, then continues train- ing on mixed-language data. The second stage uses concatenated examples from existing code- switching data alongside augmented examples created through word substitution based on cor- respondences between English source words and loanwords. For monolingual identification, MixLID outperforms all baselines in both accu- racy and macro F1 on four benchmarks within the language coverage shared with each base- line. For mixed-language identification, it out- performs the baselines in identifying the lan- guage of individual tokens on every evalu- ated language pair. MixLID achieves high per- formance across a broad range of languages in both monolingual and mixed-language set- tings. The model is publicly available athttps: //huggingface.co/nlpai-lab/MixLID. 1 Introduction Language identification (LID) determines the lan- guage used in a text, providing language infor- mation for constructing and using multilingual datasets (Burchell et al., 2025). In large-scale cor- pus construction, LID is used to categorize text by language and filter for a target language (Wenzek et al., 2020; Nguyen et al., 2024). Errors in lan- guage identification can lead to the inclusion of text in other languages or the exclusion of target- language text, affecting the composition and quality of language-specific training data (Penedo et al., 2025; Kreutzer et al., 2022). Beyond data curation, * Corresponding authors LID is also used to evaluate whether generated text is in the requested language (Chollampatt et al., 2025). In reinforcement learning, language match- ing is incorporated into the reward to encourage models to respond in the target language (Liu et al., 2026; Zhang et al., 2026). Treating an input as a single unit for language identification yields one language label for the en- tire text. Real-world text, however, may exhibit code-switching across sentences or within a sen- tence through the insertion of words or expressions from another language (Elfardy and Diab, 2012; Burchell et al., 2024a). A single label cannot distin- guish the languages used in different spans of such an input (Kargaran et al., 2024). This motivates token-level identification in addition to identifying the predominant language of the input (Nguyen and Do Ě gruĂśz, 2013; Solorio et al., 2014). Prior work has expanded the language coverage of datasets and models to include low-resource lan- guages (Burchell et al., 2023; Kargaran et al., 2023; Adebara et al., 2022). For token-level LID in mixed- language inputs, supervised sequence labeling ap- proaches use naturally occurring code-switched data with token-level language annotations to learn from word and character features and context (Chit- taranjan et al., 2014; Samih et al., 2016). To address the scarcity of token-level annotations, synthetic data have also been constructed by randomly con- catenating or mixing phrases from different lan- guages and used for training alongside monolingual and naturally occurring code-switched data (Zhang et al., 2018). Training for mixed-language inputs thus requires considering both how expressions from different languages are combined and how monolingual and mixed-language data are used to train the model. We introduce MixLID, a token-level language identification model supporting 111 languageâ script combinations. The model first learns to distin- guish individual languages using the language label arXiv:2610.16601v1 [cs.CL] 12 Oct 2026 of each monolingual example as supervision for its tokens. Training then continues on inputs formed by concatenating code-switched examples and in- puts augmented through word substitutions based on English source words and their corresponding loanwords, to identify the language of individual to- kens in mixed-language contexts. For monolingual identification, MixLID outperforms each baseline in accuracy and macro F1 on all four benchmarks, with each comparison restricted to the languageâ script combinations supported by both models. For mixed-language identification, it outperforms the baselines on every evaluated language pair. Anal- ysis of the training stages shows that subsequent mixed-language training improves the identifica- tion of individual tokens while largely preserving performance on monolingual inputs. 2 MixLID MixLID is trained in two stages to identify the language of each token in both monolingual and multilingual inputs. It first learns to distinguish lan- guages from monolingual examples in a multilin- gual corpus, then continues training on existing and synthetic examples containing multiple languages. 2.1 Training Data Construction Monolingual Data We use monolingual exam- ples with sentence-level language labels to con- struct token-level training data. Specifically, we use OpenLID-v3 (Fedorova et al., 2026) data covering 111 languageâscript combinations and assign each sentenceâs language label to its constituent words. However, the language label of a sentence may not be appropriate for all expressions it contains. We therefore assign separate labels to numbers, punc- tuation, and symbols. Abbreviations found in the lists 1 receive a language label that covers the en- tire expression, including periods. Abbreviations in the English list receive the English label regard- less of the source language, whereas those found only in the source-language list retain the source- language label. We further apply a heuristic filter based on English letter patterns to examples in non- Latin scripts to reduce incorrect assignment of the source-language label to expressions from other languages, while retaining these abbreviations. Mixed-Language DataTo identify the language of each token in mixed-language contexts, we 1 We use the abbreviation lists provided by NLTK Punkt. use both code-switched data and data augmented through word substitution. We concatenate exam- ples from the ACS Corpus into longer training sequences while preserving their token-level lan- guage labels. For augmentation, we apply word substitutions to sentences and documents from Smol (Caswell et al., 2025) and passages from Bele- bele (Bandarkar et al., 2024), retaining each aug- mented text as a separate training example. We con- struct the substitution dictionary from Wiktionary etymological data extracted with Wiktextract (Ylo- nen, 2022), retaining direct borrowing relationships between English source words and loanwords in other languages. We apply these dictionary map- pings to each text independently, replacing English source words with corresponding loanwords or vice versa. Each replacement is assigned the label of the selected target language, while the remaining ex- pressions are assigned the language label of the original text by default. All data and substitution candidates are restricted to supported languages. We retain the monolingual labeling rules for ab- breviations, numbers, punctuation, and symbols, but omit the exclusion rules based on Latin char- acter patterns to preserve mixed-language inputs. Detailed preprocessing and labeling rules for both data types are provided in Appendix B. 2.2 Learning Objective We align the labels in the constructed data with sub- word tokens using character offsets in the input text. When a word is split into multiple subwords, all resulting subwords are assigned the same language label as the original word. A multilingual encoder produces a contextual representation for each sub- word, and a classification head predicts its label. Letx = (x 1 ,...,x T )denote the input subword sequence andy i âYthe label associated with each labeled subwordx i . The setIcontains the posi- tions of labeled subwords used for training. The label setYincludes the supported language labels and separate labels for numbers, punctuation, and symbols, and is shared across both training stages. Letp θ (c| x,i)denote the predicted probability of labelc â Yat positioni, whereθcomprises the parameters of the encoder and classification head. We train the model to predict the corresponding token labels by minimizing the cross-entropy loss: L(θ;x,y) =â 1 |I| X iâI logp θ (y i | x,i).(1) BenchmarkMetric OpenLID-v3GlotLID-v3NLLB-LIDfastText lid.176CLD3CommonLinguaCONLIDPolyglot Tagger BaselineMixLID Baseline MixLIDBaselineMixLID Baseline MixLIDBaselineMixLID Baseline MixLIDBaselineMixLID Baseline MixLID FLORES+ #L11011010670749611070 Acc99.1999.7799.3899.7798.9799.7595.5999.9497.3799.9195.4099.8998.8999.7799.4699.91 Macro F199.4299.7699.5799.7699.1599.7595.4799.9497.3199.9196.1899.8999.3299.7699.5899.92 WiLI-2018 #L 8282807070818267 Acc97.5898.4397.7298.4197.5298.3795.8898.9195.7698.5997.0598.4197.6498.4198.5898.84 Macro F198.1698.5898.2198.5897.9898.5496.3498.9896.3498.7197.8298.5898.2098.5898.8898.92 UDHR #L1011019867729110167 Acc 98.8299.3898.1499.3898.3699.3494.9999.8797.6099.8493.4599.8097.1099.3899.3399.84 Macro F198.7599.3299.1499.3298.4099.2995.3099.8697.2399.8494.6399.8198.3699.3299.5499.84 CommonLID #L5858543945535841 Acc94.0496.9594.5696.7593.9396.6695.9397.7490.2897.2587.0796.8190.4396.7595.4297.74 Macro F1 81.8585.7983.4785.8083.2886.2884.9690.1775.7288.5178.3086.5078.5685.8085.0788.51 Table 1: Sentence-level language identification performance (%) on the languageâscript intersection between MixLID and each baseline. #L denotes the intersection size. Bold indicates the higher score in each paired comparison. TrainingDatasetSourceAugmentation#L#Samples MonoOpenLID-v3Monoâ111123.4M Mixed ACS CorpusCSconcatenation10509.8K SmolSentMonoSubstitution8746.8K SmolDocMonoSubstitution9220.7K BelebeleMonoSubstitution11156.6K Table 2: Training data composition and statistics. The same objective is used in both stages, with training on mixed-language data continuing from the model trained on monolingual data. 3 Experiments 3.1 Setup TrainingWe use mmBERT-small (Marone et al., 2025) as the backbone of MixLID with a 114- class token classification head for 111 languageâ script combinations and three auxiliary labels (NUM, PUNCT, andSYM). Table 2 summarizes the training data. Training proceeds in two stages, beginning with approximately 123.4M monolingual examples from OpenLID-v3. The resulting model is then further trained on approximately 634K examples constructed from the ACS Corpus, Smol, and Bele- bele for mixed-language identification. ACS exam- ples are shuffled and concatenated in groups of up to four. For Smol and Belebele, loanword-based substitution is applied independently to each eligi- ble expression with a probability of 70%. Detailed training settings for both stages and hardware spec- ifications are provided in the Appendix B. Evaluation For sentence-level evaluation, we compare MixLID with OpenLID-v3 (Fedorova et al., 2026), GlotLID-v3 (Kargaran et al., 2023), NLLB-LID (Costa-JussĂ et al., 2022), fastText lid.176 (Joulin et al., 2017), CLD3 2 , Common- 2 https://github.com/google/cld3 Lingua (PleIAs, 2026), CONLID (Foroutan et al., 2026), and Polyglot Tagger 3 on four bench- marks: FLORES+ (Burchell et al., 2024b), WiLI- 2018 (Thoma, 2018), UDHR (Kargaran et al., 2023) 4 , and CommonLID (Suarez et al., 2026). Each comparison is restricted to languages jointly supported by MixLID and the baseline. MixLID and Polyglot Tagger determine the input language by majority voting over token predictions. We re- port overall accuracy and macro F1, the mean of per-language F1 scores. For token-level eval- uation, we compare against MaskLID (Kargaran et al., 2024), Polyglot Tagger, and CLD3 on ACS (Sterner and Teufel, 2025) and the LID subset of CodeMixBench (Yang and Chai, 2025). CLD3 provides language predictions over text spans, which we align with the reference tokens. We retain language combinations supported by four models and evaluate the same eligible tokens. Token accu- racy and macro F1 are computed per example and averaged within each language combination. 3.2 Results Monolingual LID at the Sentence Level Ta- ble 1 shows that MixLID outperforms all eight baselines in both accuracy and macro F1 across the four benchmarks. This advantage holds over both sentence-level identification models and Polyglot Tagger, which supports token-level identification. Although MixLID is trained to predict language la- bels for individual tokens, these results show that its predictions also support accurate identification of the predominant language of an input. The higher accuracy and macro F1 indicate an advantage in 3 https://huggingface.co/polyglot-tagger/ language-identification 4 https://huggingface.co/datasets/cis-lmu/ udhr-lid Afrikaans Amharic Bengali Bulgarian Catalan Czech German Greek English Finnish French Gujarati Hebrew Hindi Indonesian Italian Javanese Japanese Kannada Korean Malayalam Marathi Dutch Punjabi Polish Portuguese Russian Spanish Tamil Telugu Thai Turkish Ukrainian Urdu Vietnamese Yoruba MixLID GlotLID-v3 Polyglot Tagger OpenLID-v3 NLLB-LID CONLID fastText lid.176 CommonLingua CLD3 94.499.999.997.797.598.898.889.395.799.598.510010099.497.898.79699.910098.41009898.310075.39899.698.699.510010099.48299.999.697.4 92.399.699.697.69498.297.297.29597.797.799.999.797.397.396.996.499.599.88399.998.197.899.875.196.699.597.699.399.999.99980.295.599.696.4 86.999.499.995.489.898.498.480.395.898.598.199.999.899.293.498.294.599.999.998.499.997.999.199.975.397.499.197.599.299.999.998.678.299.599.597.4 89.595.899.796.489.298.397.49594.897.696.110010097.994.19695.999.710089.410098.297.399.975.196.799.395.599.510099.999.178.198.399.697.5 91.599.399.896.989.898.197.593.694.998.697.799.999.998.996.797.996.996.199.974.399.99898.999.975.197.399.297.999.499.999.99977.698.499.396.4 88.799.199.19593.598.296.180.893.3989799.899.498.593.795.694.699.199.583.499.897.897.599.675.296.298.596.497.999.999.898.476.79499.595.8 86.182.399.898.185.798.597.983.787.598.697.699.910099.491.397.684.699.399.984.310098.293.699.975.397.69997.299.410099.999.27793.999.463.5 85.899.599.190.988.39696.477.193.397.395.999.999.197.478.295.289.799.796.586.699.996.695.399.774.994.197.994.799.410099.597.176.895.175.895.1 8399.599.484.282.697.397.476.69498.296.799.499.496.482.197.290.598.799.680.299.894.697.799.575.295.398.496.999.399.398.798.474.697.378.934.1 F1 (%) 34 90 95 98 100 Figure 1: Language-wise F1 scores for the 36 languages shared by all models and all four sentence-level benchmarks. Each cell shows the average score across these datasets. PairCLD3MaskLIDPolyglotMixLID ACS daâen 56.1 / 32.638.7 / 20.762.5 / 37.983.3 / 73.5 deâen 56.4 / 33.935.9 / 22.978.2 / 62.098.0 / 96.9 esâen48.2 / 28.222.9 / 15.374.6 / 56.996.1 / 94.1 frâen54.8 / 32.124.4 / 15.571.0 / 47.793.1 / 90.4 idâen31.7 / 18.419.3 / 12.138.0 / 24.292.1 / 89.1 itâen56.3 / 33.329.6 / 19.374.3 / 55.795.6 / 93.6 nlâen52.2 / 30.935.8 / 21.966.1 / 45.192.6 / 90.0 svâen56.6 / 32.941.7 / 24.971.7 / 48.792.3 / 87.1 trâde58.1 / 35.153.1 / 37.286.6 / 80.593.9 / 81.5 trâen44.6 / 25.831.2 / 18.969.9 / 45.691.6 / 87.4 zhâen 86.2 / 69.537.5 / 25.294.7 / 89.297.0 / 93.0 Avg54.6 / 33.933.6 / 21.371.6 / 53.993.2 / 88.8 CodeMixBench deâen 86.3 / 61.155.3 / 29.290.6 / 65.895.2 / 82.8 esâen55.9 / 47.325.8 / 17.682.3 / 71.292.8 / 88.7 zhâen 80.5 / 46.061.6 / 31.083.9 / 55.395.5 / 90.3 Avg74.2 / 51.547.6 / 25.985.6 / 64.194.5 / 87.3 Table 3: Token-level LID results (Acc / Macro F1). both overall prediction accuracy and performance averaged across languages. The results demonstrate consistent performance advantages across evalua- tion data from diverse sources, including web text. Mixed-Language LID at the Token Level Ta- ble 3 shows that MixLID outperforms MaskLID, Polyglot Tagger, and CLD3 in both accuracy and macro F1 across all evaluated subsets of ACS and CodeMixBench. Averaged across language pairs, MixLID achieves an accuracy of 93.23% and a macro F1 of 88.78% on ACS, and 94.50% and 87.29%, respectively, on CodeMixBench. These re- sults indicate that MixLID can distinguish expres- sions from different languages within the same con- text, beyond identifying the predominant language of an input. Its advantage across the evaluated lan- guage combinations suggests that this ability is not limited to a particular language pair. 4 Analysis 4.1 Language-wise Performance To examine whether the overall performance advan- tage of MixLID extends to individual languages, we compare performance on the 36 languages shared by all models and all four benchmarks, as shown in Figure 1. MixLID achieves the highest F1 on 26 of these languages. Its mean F1 is 97.38%, exceeding the strongest baseline, GlotLID-v3, by Stage 1Stage 2Stage 1+2 BenchmarkAccMacro F1AccMacro F1AccMacro F1 FLORES+99.6799.6497.3997.2199.7799.76 WiLI-201898.5298.6294.0594.7498.4398.58 UDHR98.6998.3496.3295.8699.3899.32 CommonLID93.6481.4290.9174.6796.9585.79 ACS53.7534.3187.3280.7893.2388.78 CodeMixBench77.3654.4178.5569.6594.5087.29 Table 4: Effect of training stages on LID performance at the sentence and token levels across six benchmarks. 0.99 percentage points. It outperforms GlotLID-v3 and Polyglot Tagger on 33 and 31 languages, re- spectively, and achieves F1 scores above 95% on 32 of the 36 languages. These results indicate that the overall performance advantage of MixLID is reflected across a broad range of languages. 4.2 Effect of Training Stages Table 4 compares training on monolingual data alone (Stage 1), mixed-language data alone (Stage 2), and both stages in sequence (Stage 1+2). Stage1 outperforms Stage2 on all four sentence-level benchmarks, whereas Stage2 performs better at token identification in mixed-language inputs. The model trained through both stages outperforms ei- ther individual stage on the mixed-language bench- marks ACS and CodeMixBench. It also surpasses Stage1 in sentence-level identification, except for a slight decrease on WiLI-2018. These results sug- gest that the two stages serve complementary roles, improving mixed-language identification while largely preserving monolingual performance. 5 Conclusion We introduced MixLID, a token-level language identification model covering 111 language la- bels and designed to distinguish languages within mixed-language inputs. Trained on monolingual data followed by existing and augmented mixed- language data, MixLID outperformed the baselines in both monolingual and mixed-language identifi- cation. Our analysis showed that mixed-language training improved individual token identification while largely preserving performance on monolin- gual inputs. We hope MixLID will serve as a useful resource for research and applications requiring language identification within multilingual text. Limitations The linguistic scope of this study is limited to the 111 language labels supported by MixLID and does not encompass the full range of linguistic diversity. In addition, dictionary-based substitution and auto- matic labeling may not fully account for contextual usage and linguistic ambiguity, potentially intro- ducing unnatural expressions or labeling errors into the training data. Finally, mixed-language evalu- ation covers a limited set of code-switching lan- guage combinations, and generalization to a wider range of combinations requires further validation. Ethical Considerations This study uses publicly available datasets and lexi- cal resources to develop a model for language iden- tification in text. The loanword-based augmented data are constructed for training token-level lan- guage identification and do not represent the actual language use or code-switching practices of any particular linguistic community. To ensure trans- parency in data use and experimentation, we doc- ument the resource sources, data construction and preprocessing methods, and evaluation procedures. References Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Inciarte. 2022. Afrolid: A neural language identification tool for african lan- guages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1958â1981. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The belebele benchmark: a parallel reading comprehension dataset in 122 lan- guage variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 749â775. Laurie Burchell, Alexandra Birch, Nikolay Bogoychev, and Kenneth Heafield. 2023. An open dataset and model for language identification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 865â879. Laurie Burchell, Alexandra Birch, Robert Thompson, and Kenneth Heafield. 2024a. Code-switched lan- guage identification is harder than you think. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 646â658. Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Baùón, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Ha- jic, et al. 2025. An expanded massive multilingual dataset for high-performance language technologies (hplt). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 17452â17485. Laurie Burchell, Jean Maillard, Antonios Anastasopou- los, Christian Federmann, Philipp Koehn, and Skyler Wang. 2024b. Findings of the wmt 2024 shared task of the open language data initiative. In Proceed- ings of the Ninth Conference on Machine Translation, pages 110â117. Isaac Caswell, Elizabeth Nielsen, Jiaming Luo, Colin Cherry, Geza Kovacs, Hadar Shemtov, Partha Taluk- dar, Dinesh Tewari, Moussa Doumbouya, Djibrila Diane, et al. 2025. Smol: Professionally translated parallel data for 115 under-represented languages. In Proceedings of the Tenth Conference on Machine Translation, pages 1103â1123. Gokul Chittaranjan, Yogarshi Vyas, Kalika Bali, and Monojit Choudhury. 2014. Word-level language identification using crf: Code-switching shared task report of msr india system. In Proceedings of the first workshop on computational approaches to code switching, pages 73â79. Shamil Chollampatt, Minh Quang Pham, Sathish Reddy Indurthi, and Marco Turchi. 2025. Cross-lingual eval- uation of multilingual text generation. In Proceed- ings of the 31st International Conference on Compu- tational Linguistics, pages 7766â7777. Marta R Costa-JussĂ , James Cross, Onur Ăelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022.No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. Heba Elfardy and Mona Diab. 2012. Token level identi- fication of linguistic code switching. In Proceedings of COLING 2012: posters, pages 287â296. Mariia Fedorova, Nikolay Arefyev, Maja Buljan, Jind Ë rich Helcl, Stephan Oepen, Egil Rønningstad, and Yves Scherrer. 2026. Openlid-v3: Improving the precision of closely related language identificationâ an experience report. In Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 275â292. Negar Foroutan, Jakhongir Saydaliev, Grace Kim, and Antoine Bosselut. 2026. Conlid: Supervised con- trastive learning for low-resource language identifi- cation. In Proceedings of the 19th Conference of the European Chapter of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 6693â6708. Javier Iranzo-SĂĄnchez, Parnia Bahar, Alejandro PĂŠrez- GonzĂĄlez-de Martos, and Mattia Antonino Di Gangi. 2025.Word-level language identification using encoder-only and decoder-only models. In Proceed- ings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025), pages 82â86. Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timo- thy Baldwin, and Krister LindĂŠn. 2019. Automatic language identification in texts: A survey. Journal of Artificial Intelligence Research, 65:675â782. Armand Joulin, Edouard Grave, Piotr Bojanowski, and TomĂĄĹĄ Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th conference of the European chapter of the association for com- putational linguistics: volume 2, short papers, pages 427â431. Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. 2023. Glotlid: Language iden- tification for low-resource languages. arXiv preprint arXiv:2310.16248. Amir Hossein Kargaran, François Yvon, and Hinrich SchĂźtze. 2024. Masklid: Code-switching language identification through iterative masking. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 459â469. Ben King and Steven Abney. 2013. Labeling the lan- guages of words in mixed-language documents using weakly supervised methods. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1110â1119. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan Van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. 2022. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics, 10:50â 72. Yifeng Liu, Siqi Ouyang, Lei Li, et al. 2026. Mending the holes: Mitigating reward hacking in reinforce- ment learning for multilingual translation. In Find- ings of the Association for Computational Linguistics: ACL 2026, pages 33699â33723. Marc Marone, Orion Weller, William Fleshman, Eu- gene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmbert: A modern multilingual en- coder with annealed language learning. Preprint, arXiv:2509.06888. Dong Nguyen and A Seza Do Ě gruĂśz. 2013. Word level language identification in online multilingual com- munication. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 857â862. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. 2024. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. In Proceedings of the 2024 Joint International Conference on Compu- tational Linguistics, Language Resources and Evalu- ation (LREC-COLING 2024), pages 4226â4237. Guilherme Penedo, Hynek KydlĂ Ë cek, Vinko Sabol Ë cec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. Fineweb2: One pipeline to scale them allâadapting pre-training data processing to every language. arXiv preprint arXiv:2506.20920. PleIAs. 2026. Commonlingua: Byte-level language identification for 334 languages. Younes Samih, Suraj Maharjan, Mohammed Attia, Laura Kallmeyer, and Thamar Solorio. 2016. Mul- tilingual code-switching identification via lstm re- current neural networks. In Proceedings of the sec- ond workshop on computational approaches to code switching, pages 50â59. Edoardo Signoroni, Emma Bedna Ë rĂkovĂĄ, and Pavel Rychl ` y. 2026. Rebelòt: Datasets and token-level language identification for lombard-italian-english code-mixing. In SIGUL 2026 Joint Workshop with ELE, EURALI, and DCLRL: Towards Inclusivity and Equality: Language Resources and Technologies for Under-Resourced and Endangered Languages@ LREC 2026, page 253. Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, et al. 2014. Overview for the first shared task on language identification in code-switched data. In Proceedings of the first workshop on computational approaches to code switching, pages 62â72. Igor Sterner. 2024. Multilingual identification of en- glish code-switching. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 163â173. Igor Sterner and Simone Teufel. 2023. Tongueswitcher: Fine-grained identification of german-english code- switching. In Proceedings of the 6th Workshop on Computational Approaches to Linguistic Code- Switching, pages 1â13. Igor Sterner and Simone Teufel. 2025. Minimal pair- based evaluation of code-switching. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18575â18598. Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett, Rafael Mosquera, Sara HincapiĂŠ Monsalve, Thom Vaughan, Damian Stewart, Malte Ostendorff, Idris Abdulmumin, Vukosi Marivate, et al. 2026. Com- monlid: Re-evaluating state-of-the-art language iden- tification performance on web data. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 33063â33080. Martin Thoma. 2018. The wili benchmark dataset for written language identification. arXiv preprint arXiv:1801.07779. Rob Van Der Goot. 2025. Identifying open challenges in language identification. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18207â 18227. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco GuzmĂĄn, Ar- mand Joulin, and Edouard Grave. 2020. Ccnet: Ex- tracting high quality monolingual datasets from web crawl data. In Proceedings of the twelfth language resources and evaluation conference, pages 4003â 4012. Yilun Yang and Yekun Chai. 2025. Codemixbench: Evaluating code-mixing capabilities of llms across 18 languages. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2139â2169. Tatu Ylonen. 2022.Wiktextract: Wiktionary as machine-readable structured data. In Proceedings of the Thirteenth Language Resources and Evalua- tion Conference, pages 1317â1325. Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Kaiyu Huang, Yufeng Chen, Jinan Xu, and Jie Zhou. 2026. Think natively: Unlocking multilingual reasoning with consistency-enhanced reinforcement learning. In Proceedings of the 64th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 20766â20783. Yuan Zhang, Jason Riesa, Dan Gillick, Anton Bakalov, Jason Baldridge, and David Weiss. 2018. A fast, compact, accurate model for language identification of codemixed text. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 328â337. A Related Work Research on language identification at the sen- tence and document levels has explored statistical classification using character and word features as well as neural approaches (Jauhiainen et al., 2019). More recent work has expanded language coverage through the collection, curation, and re- lease of training data, addressing diverse scripts and low-resource languages (Burchell et al., 2023; Kargaran et al., 2023; Adebara et al., 2022). Efforts to better distinguish closely related languages and handle nonlinguistic noise have included enriching training data, merging selected language varieties, and treating noise as a separate class (Fedorova et al., 2026). Evaluations on web data have shown that high scores on established benchmarks do not necessarily translate into comparable performance in domains containing noise and diverse language varieties (Suarez et al., 2026). A systematic com- parison of factors including input length, training data size, and domain has likewise identified cross- domain generalization as a major challenge (Van Der Goot, 2025). To address domain differences, supervised contrastive learning has been used to learn language representations and improve identifi- cation of low-resource languages on out-of-domain data (Foroutan et al., 2026). Word-level identification in mixed-language text has been formulated as sequence labeling that accounts for context, with weak supervision de- rived from monolingual data (King and Abney, 2013). Synthetic mixed-language data have also been used to train token classifiers supporting mul- tiple languages (Zhang et al., 2018). Research has explored identification using lexicons, morpho- logical analysis, and contextual embeddings, as well as comparisons between encoder-based clas- sification and decoder-based generation (Sterner and Teufel, 2023; Iranzo-SĂĄnchez et al., 2025). Other work has harmonized annotations across language-pair datasets to train token-level iden- tification of English versus other languages and examined transfer to language pairs unseen dur- ing training (Sterner, 2024). Token identification models have also been fine-tuned on authentic code- switched data and synthetic examples constructed by inserting contiguous spans from another lan- guage into sentences (Signoroni et al., 2026). An openly available model similarly fine-tunes a pre- trained encoder to identify the language of each token using synthetic data formed by concatenat- ing sentences in different languages. Other ap- proaches predict the set of languages present in a mixed-language input through sentence-level mul- tilabel classification, or iteratively mask language- specific features to detect additional languages with an existing identifier (Burchell et al., 2024a; Kar- garan et al., 2024). Our work focuses on two-stage training from monolingual to mixed-language data, using both existing code-switched data and aug- mented data based on loanword correspondences for mixed-language training. This approach sup- ports token-level language identification in mixed- language inputs across a broad range of languages, while aggregating the predictions also enables iden- tification of the language of a monolingual input. B Training Details B.1 Language Inventory Table 5 lists the languages supported by MixLID and the 111 language labels used for training. We add three labels,NUM,PUNCT, andSYM, for numbers, punctuation, and symbols, respectively, yielding 114 output classes. These labels are assigned to the corresponding expressions instead of a language label. The same label inventory is used in both training stages. B.2 Shared Labeling and Token Alignment We distinguish numbers, punctuation, and symbols using Unicode categories, while treating decimal points and commas separating groups of three dig- its as part of a numeric expression. For example, 3.14and1,000each receive a singleNUMlabel, whereas3,14is split intoNUM,PUNCT, andNUM. Percent and per mille signs and their variants are labeledSYMregardless of their Unicode categories. Abbreviations matching the NLTK Punkt lists re- ceive a language label over the entire expression, including periods. Matches in the English list take precedence and receive the English label; those found only in the list for the original language re- tain that language label. Apostrophes within words receive the same lan- guage label as the word, whereas standalone quota- tion marks are labeledPUNCT. Ordinary spaces are left unlabeled, while newlines and tabs in mixed- language data are labeledPUNCT. The resulting la- beled spans are aligned with subwords using char- acter offsets. When a subword overlaps multiple spans, we assign the label of the span with the great- est character overlap, excluding ordinary spaces. Ties are resolved in favor of the earlier span. B.3 Monolingual Data Preparation To reduce the assignment of a sentenceâs language label to expressions from other languages, we apply a filter based on English letter patterns to mono- lingual examples in non-Latin scripts. Except for expressions in the abbreviation lists, examples con- taining a sequence of three or more English letters are excluded. This filter is not applied to examples in Latin scripts or to mixed-language data. B.4 Mixed-Language Data Preparation Code-Switched Data Preparation To train token-level language identification on longer in- puts containing multiple languages, we construct training sequences by concatenating existing code- switched examples. We use the ACS Corpus, which provides token-level language labels. Before con- catenation, we remove anonymized user mentions and relabel tokens without a specific language la- bel as numbers, punctuation, or symbols. Examples containing any remaining tokens that cannot be re- labeled are excluded. We shuffle the cleaned exam- ples and concatenate up to four with a single space between them, preserving their language labels and using each example once. Loanword-Based Augmentation To construct code-switched data, we apply loanword-based sub- stitution to datasets containing monolingual ex- amples in multiple languages: sentences and doc- uments from Smol and passages from Belebele. The substitution dictionary includes entries for bor- rowing, adapted borrowing, unadapted borrowing, learned borrowing, semi-learned borrowing, and orthographic borrowing. We exclude entries whose forms are identical to their English source words or differ only in case, as well as affix forms with a leading or trailing hyphen. For reverse substitu- tion, we also exclude loanwords associated with multiple English source words. Within each input, we select the longest non-overlapping matches at word boundaries and independently substitute each match with a probability of 70%. When an English source word has candidates in multiple languages, we select a candidate language uniformly at ran- dom. We do not perform contextual word sense disambiguation or infer inflected forms, and we re- tain examples even when no substitution is applied. B.5 Training Configuration Table 6 reports the hyperparameters for each train- ing stage. The second stage is initialized from the model trained in the first stage, with the op- timizer and learning rate scheduler reinitialized. Both stages update all model parameters using four NVIDIA RTX A6000 GPUs. C Evaluation Details C.1 Baseline Models The baselines include models that predict a sin- gle language for an entire input and models that LanguageLabelLanguageLabelLanguageLabelLanguageLabel Afrikaans afr_LatnHausa hau_LatnMarathi mar_DevaSpanish spa_Latn Amharic amh_EthiHebrew heb_HebrModern Greek ell_GrekStandard Estonian ekk_Latn Arabic ara_ArabHindi hin_DevaNepali npi_DevaStandard Latvian lvs_Latn Armenian hye_ArmnHungarian hun_LatnNigerian Fulfulde fuv_LatnStandard Malay zsm_Latn Assamese asm_BengIcelandic isl_LatnNorth Azerbaijani azj_LatnSundanese sun_Latn Bambara bam_LatnIgbo ibo_LatnNorthern Uzbek uzn_LatnSwahili swh_Latn Basque eus_LatnIloko ilo_LatnNorwegian BokmĂĽl nob_LatnSwati ssw_Latn Bengali ben_BengIndonesian ind_LatnNyanja nya_LatnSwedish swe_Latn Bulgarian bul_CyrlItalian ita_LatnOdia ory_OryaTajik tgk_Cyrl Burmese mya_MymrJapanese jpn_JpanPanjabi pan_GuruTamil tam_Taml Catalan cat_LatnJavanese jav_LatnParaguayan GuaranĂ gug_LatnTelugu tel_Telu Cebuano ceb_LatnKabuverdianu kea_LatnPedi nso_LatnThai tha_Thai Central Khmer khm_KhmrKachin kac_LatnPersian fas_ArabTibetan bod_Tibt Central Kurdish ckb_ArabKannada kan_KndaPlateau Malagasy plt_LatnTigrinya tir_Ethi Croatian hrv_LatnKazakh kaz_CyrlPolish pol_LatnTosk Albanian als_Latn Czech ces_LatnKinyarwanda kin_LatnPortuguese por_LatnTsonga tso_Latn Danish dan_LatnKirghiz kir_CyrlRomanian ron_LatnTswana tsn_Latn Dutch nld_LatnKorean kor_HangRussian rus_CyrlTurkish tur_Latn English eng_LatnLao lao_LaooSerbian srp_CyrlUkrainian ukr_Cyrl Filipino fil_LatnLingala lin_LatnShan shn_MymrUrdu urd_Arab Finnish fin_LatnLithuanian lit_LatnShona sna_LatnVietnamese vie_Latn French fra_LatnLuo luo_LatnSindhi snd_ArabWaray war_Latn Ganda lug_LatnMacedonian mkd_CyrlSinhala sin_SinhWest Central Oromo gaz_Latn Georgian kat_GeorMalayalam mal_MlymSlovak slk_LatnWolof wol_Latn German deu_LatnMaltese mlt_LatnSlovenian slv_LatnXhosa xho_Latn Gujarati guj_GujrMandarin Chinese cmn_HansSomali som_LatnYoruba yor_Latn Haitian hat_LatnMandarin Chinese cmn_HantSouthern Pashto pbt_ArabZulu zul_Latn Halh Mongolian khk_CyrlMaori mri_LatnSouthern Sotho sot_Latn Table 5: Supported languages and their training labels. SettingMonolingualMixed OptimizerAdamWAdamW LR scheduleLinearLinear Maximum length8,1928,192 Learning rate 5Ă 10 â5 1Ă 10 â5 Epochs11 Batch size per GPU512128 Gradient accumulation21 Effective batch size4,096512 Warmup ratio0.050.05 Weight decay00 PrecisionBF16BF16 Seed4242 Table 6: Training hyperparameters for both stages. provide token-level or span-level predictions. The coverage figures below refer to the label invento- ries of the models used in our experiments and are distinct from the numbers of shared evaluation languages reported in the main tables. ⢠OpenLID-v3 A fastText-based model trained on monolingual data from multiple sources, with a focus on improving identification precision for closely related languages. It provides 194 lan- guage labels and a non-language label. â˘GlotLID-v3 A fastText-based model designed to extend language identification coverage to low- resource languages. It provides 2,102 labels dis- tinguishing languages and scripts. â˘NLLB-LID A fastText-based language identifi- cation model used for data collection and filtering in multilingual machine translation. It predicts the language of an entire input, and the model used in our experiments provides 218 output la- bels. â˘fastText lid.176 A general-purpose language identification model trained on Wikipedia, Tatoeba, and SETimes. It supports 176 languages and predicts the language of an input text. ⢠CommonLingua A lightweight byte-level model designed for language identification in large cor- pora. Its training data include diverse documents and low-resource languages, and it provides 334 language labels. â˘CONLID A language identification model that uses contrastive learning and supports a broad range of languages. The model used in our exper- iments provides 2,099 output labels. ⢠CLD3 A neural language identification model us- ing charactern-gram features, with 109 language and script labels. It provides predictions for both entire inputs and text spans and is therefore in- cluded in both evaluations. â˘MaskLID A method that applies iterative mask- ing to an existing language identification model to identify expressions in different languages within mixed-language inputs, without additional training. We use GlotLID-v3, with its 2,102 la- bels, as the underlying model. TypeNameURL ModelOpenLID-v3 https://huggingface.co/HPLT/OpenLID-v3 GlotLID-v3 https://huggingface.co/cis-lmu/glotlid NLLB-LID https://huggingface.co/facebook/fasttext-language-identification fastText lid.176 https://fasttext.c/docs/en/language-identification.html CommonLingua https://huggingface.co/PleIAs/CommonLingua CONLID https://huggingface.co/epfl-nlp/ConLID CLD3 https://github.com/google/cld3 MaskLID https://github.com/cisnlp/MaskLID Polyglot Tagger https://huggingface.co/polyglot-tagger/language-identification BenchmarkFLORES+ https://huggingface.co/datasets/openlanguagedata/flores_plus WiLI-2018 https://doi.org/10.5281/zenodo.841984 UDHR https://huggingface.co/datasets/cis-lmu/udhr-lid CommonLID https://huggingface.co/datasets/commoncrawl/CommonLID ACS Benchmark https://huggingface.co/datasets/igorsterner/acs-benchmark CodeMixBench https://huggingface.co/datasets/CodeMixBench/CodeMixBench Table 7: Sources of baseline models and evaluation benchmarks. â˘Polyglot Tagger An XLM-R-based token clas- sification model trained on synthetic data con- structed by combining sentences in different lan- guages. The checkpoint used in our experiments supports 105 languages. We use its token pre- dictions directly for token-level evaluation and aggregate them for sentence-level evaluation. C.2 Evaluation Benchmarks Monolingual Benchmarks We use four bench- marks to evaluate the prediction of a single lan- guage for each input. FLORES+ contains parallel texts translated into multiple languages; we use the devtestsplit of v4.5. WiLI-2018 consists of para- graphs collected from Wikipedia, for which we use the official test split. UDHR is based on multilin- gual translations of the Universal Declaration of Human Rights; we use the cleaned version released with the GlotLID study. CommonLID contains web text with human-annotated language labels. Each sentence, paragraph, or line is treated as one input, and each comparison is restricted to the languages jointly supported by the models. Mixed-Language BenchmarksWe use the ACS Benchmark and the LID subset of CodeMixBench to evaluate token-level language identification in mixed-language inputs. ACS provides observed code-switched sentences and manipulated variants. We evaluate both types of input across 11 language pairs in its test split, using the provided automatic language labels. For CodeMixBench, we evaluate GermanâEnglish, SpanishâEnglish, and Chineseâ English, which are supported by all models in the comparison. C.3 Evaluation Protocol and Metrics Language MatchingWe map language codes to a common representation to reconcile differences in model label conventions. Sentence-level com- parisons include only examples in languages sup- ported by the benchmark and both models. Ara- bic and Persian varieties are mapped to the cor- responding macrolanguage labels when these are provided by a model; support for one variety is not taken to imply support for other varieties. Script constraints are also applied when specified by the model. We use the script metadata provided by FLORES+ and UDHR, and infer the predominant script from the distribution of Unicode scripts in the input for WiLI-2018 and CommonLID. For fast- Text lid.176, CommonLingua, and Polyglot Tagger, we apply language-level coverage and the script constraints of the comparison model. Token-level evaluation includes only subsets for which MixLID, MaskLID, Polyglot Tagger, and CLD3 all support both languages in the pair. Prediction Alignment For sentence-level evalu- ation, MixLID and Polyglot Tagger determine the input language by majority voting over subword language predictions, excluding non-language la- bels. Ties are resolved by lexicographic label order. For token-level evaluation, we use character off- sets in the original text to align model output spans with the reference tokens. For each reference to- ken, we sum the character overlap with predicted spans by language and select the language with the largest total overlap. CLD3 span offsets are con- verted from bytes to characters, and tokens with no corresponding prediction are treated as unde- termined. We exclude tokens with non-language, mixed, or ambiguous reference labels, as well as tokens consisting entirely of whitespace, numbers, punctuation, or symbols, consistently across mod- els. Undetermined predictions and predictions out- side the language pair are counted as errors for eligible tokens. Sentence-Level MetricsWe report accuracy and macro F1 to measure overall identification accuracy and performance across languages. Accuracy is the proportion of evaluation examples for which the language is correctly predicted. Acc = # correct samples # samples (2) For each languageâ, F1 is computed from its true positives, false positives, and false negatives. LetL denote the set of shared languages included in the evaluation. F 1,â = 2 TP â 2 TP â + FP â + FN â (3) MacroF1 = 1 |L| X ââL F 1,â (4) Thus, accuracy measures the proportion of correct predictions across all evaluation examples, whereas macro F1 is the arithmetic mean of the F1 scores for individual languages. Token-Level Metrics For mixed-language in- puts, we first compute accuracy and macro F1 over the eligible tokens in each example, then average the scores across examples within each language pair. Examples with no eligible tokens are excluded. LetA i andF i denote the accuracy and macro F1 for examplei, respectively, and letL i be the union of its reference and predicted labels over eligible tokens. A i = # correct tokens # tokens (5) F i = 1 |L i | X ââL i F (i) 1,â (6) Here,F (i) 1,â is computed using only the eligible to- kens in examplei. Avg in the main tables is the arithmetic mean of the language-pair scores within each benchmark. All scores are reported as percent- ages.