Paper deep dive
The Curse of Multilinguality in Lexical Normalization
Saman Rahbar
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/2/2026, 3:42:46 AM
Summary
This study investigates the 'curse of multilinguality' in lexical normalization, demonstrating that training a fixed-capacity character-level Transformer model on multiple languages simultaneously leads to a decline in per-language accuracy after an optimal point (typically 1-4 languages). The decline is attributed to capacity competition rather than data volume, as confirmed by a fixed-data ablation. Typological distance does not reliably predict the optimal number of co-training languages.
Entities (10)
Relation Signals (6)
MULTILEXNORM → contains → 12 Languages
confidence 95% · We use all twelve language datasets from MULTILEXNORM: Danish, German, English, Spanish, Croatian, Indonesian–English, Italian, Dutch, Slovenian, Serbian, Turkish, and Turkish–German
Lexical Normalization → exhibits → Curse of Multilinguality
confidence 95% · We find a clear curse of multilinguality: accuracy is highest when a language is trained with only a few others... and then falls steadily
Spanish → performspoorly → Lexical Normalization
confidence 92% · on both the model scores below zero at everyk, meaning it corrupts more than it repairs
Italian → performspoorly → Lexical Normalization
confidence 92% · on both the model scores below zero at everyk, meaning it corrupts more than it repairs
Curse of Multilinguality → causedby → Capacity Competition
confidence 90% · points to competition among the languages for one fixed-size model rather than to how much data is available.
Typological Distance → doesnotpredict → Optimal Co-training Count
confidence 88% · find no dependable rule: any apparent relationship rests on a couple of languages and does not hold up.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Lexical normalization rewrites the noisy, non-standard words that fill user-generated text (tmrw, u, gr8) into their standard forms. Because labelled data is scarce for most languages, a popular shortcut is to train a single model on many languages at once. We ask a simple question: how many languages should such a model be trained on? Using one fixed-capacity character-level model and twelve languages from a standard benchmark, we vary the number of jointly trained languages from one to twelve and measure per-language accuracy. We find a clear curse of multilinguality: accuracy is highest when a language is trained with only a few others, often just one to four, and then falls steadily and substantially, dropping by about forty percent as the rest are piled on. A control that holds the total amount of training data constant makes the decline arrive sooner and fall further, which points to competition among the languages for one fixed-size model rather than to how much data is available. We also test whether a language's typological distance from the others predicts its ideal number of co-training languages, and find no dependable rule: any apparent relationship rests on a couple of languages and does not hold up. For compact normalization models, less can be more: a few languages beat pooling everything into a single model.
Tags
Links
- Source: https://arxiv.org/abs/2609.00329v1
- Canonical: https://arxiv.org/abs/2609.00329v1
Trouble viewing inline? Open PDF directly →
Full Text
29,526 characters extracted from source content.
Expand or collapse full text
The Curse of Multilinguality in Lexical Normalization Saman Rahbar Independent Researcher info@srahbar.com Abstract Lexical normalization rewrites the noisy, non- standard words that fill user-generated text (tmrw, u, gr8) into their standard forms. Be- cause labelled data is scarce for most languages, a popular shortcut is to train a single model on many languages at once. We ask a simple question: how many languages should such a model be trained on? Using one fixed-capacity character-level model and twelve languages from a standard benchmark, we vary the num- ber of jointly trained languages from one to twelve and measure per-language accuracy. We find a clear curse of multilinguality: accuracy is highest when a language is trained with only a few others, often just one to four, and then falls steadily and substantially, dropping by about forty percent as the rest are piled on. A con- trol that holds the total amount of training data constant makes the decline arrive sooner and fall further, which points to competition among the languages for one fixed-size model rather than to how much data is available. We also test whether a language’s typological distance from the others predicts its ideal number of co- training languages, and find no dependable rule: any apparent relationship rests on a couple of languages and does not hold up. For compact normalization models, less can be more: a few languages beat pooling everything into a single model. 1 Introduction Text written by people online (social-media posts, comments, messages) is full of abbreviations, misspellings, phonetic spellings, and playful re- spellings. Lexical normalization is the task of turn- ing each such non-standard word into its canonical form, mapping tmrw to tomorrow or u to you, so that downstream tools built for clean text can cope. It is a long-standing preprocessing step for noisy- text natural language processing (NLP) (Han and Baldwin, 2011), and a shared task in its own right (Baldwin et al., 2015). The main obstacle is data. Hand-labelled normal- ization corpora are small and exist for only a hand- ful of languages. The usual way around this is to train one model on several languages at once, in the hope that they reinforce one another: a misspelling pattern learned in one language may transfer to another, and the pooled data is larger than any sin- gle language’s. The MULTILEXNORM shared task was organized around exactly this setting (van der Goot et al., 2021), and its strongest entry was a single multilingual byte-level model fine-tuned on all languages together (Samuel and Straka, 2021); the same pool-everything recipe underpins multi- lingual pretraining more broadly (Conneau et al., 2020). But is more always better? In machine trans- lation and in large language models, researchers have repeatedly observed a curse of multilinguality (Conneau et al., 2020; Chang et al., 2024): when a model of fixed size is trained on more and more languages, each individual language eventually gets worse, because the languages compete for the model’s limited capacity. Whether the same trade- off governs lexical normalization has not, to our knowledge, been studied. Normalization is a small, character-level task with tiny models and short out- puts, so it is not obvious that the crowded-capacity story carries over: the models may be far from their capacity limit, or the shared character patterns across languages may make extra languages purely helpful. We study this directly.We fix one small character-level model and, holding its size con- stant, train it on every number of languages from one to twelve, drawn from the MULTILEXNORM benchmark, and measure how well it normalizes each individual language. Our contributions are: •We map the accuracy-versus-number-of- languages frontier for lexical normalization and show that per-language accuracy peaks arXiv:2609.00329v1 [cs.CL] 27 Aug 2026 at a few languages and then declines steadily, and the curse of multilinguality holds (Sec- tion 4.1). •Through a control that holds total training data fixed, we show the decline is not explained by data volume but by many languages sharing one fixed-size model (Section 4.2). •We test, and fail to confirm, the intuitive idea that a language’s typological isolation predicts its ideal number of co-training languages: any apparent effect is fragile and rests on a couple of languages (Section 4.3). 2 Related Work Lexical normalization. Normalization of user- generated text goes back to work on text mes- sages and tweets (Han and Baldwin, 2011) and was shaped by the W-NUT shared tasks (Baldwin et al., 2015) and by broad-coverage systems such as MONOISE (van der Goot and van Noord, 2017). We build directly on MULTILEXNORM (van der Goot et al., 2021), which put twelve normalization datasets into one word-aligned format under one metric and made cross-language comparison possi- ble for the first time. Its strongest entry fine-tuned the byte-level model BYT5 (Samuel and Straka, 2021; Xue et al., 2022) on all languages jointly. That is the setting we take apart: the benchmark fixed the languages and the metric, but the number of languages was never treated as a variable. The curse of multilinguality.Adding languages to a fixed-capacity model helps for a while and then hurts each one. This has been documented for massively multilingual translation (Arivazhagan et al., 2019), for cross-lingual language models (Conneau and Lample, 2019; Conneau et al., 2020), and most directly in a study of language-model perplexity spanning 250 languages (Chang et al., 2024). The mechanism usually named is negative interference, where languages start to degrade one another once too little capacity is left to go around (Wang et al., 2020), and the usual remedy is to give each language a little private capacity, as in adapter modules (Pfeiffer et al., 2020). Every one of these results comes from large models doing word- or subword-level work. Normalization is neither: the models are tiny, the alphabet is shared, and the output is a handful of characters. Whether a curse appears at that scale is an open question, and a plausible case can be made either way, since a small model might be nowhere near its capacity limit, or shared character patterns might make every extra language a gift. Typological distance. If languages do compete, one might expect similar ones to compete less. Dis- tance between languages can be read off URIEL and itslang2vecinterface (Littell et al., 2017), which we use to ask whether a language’s typo- logical isolation says anything about how many partners it wants. 3 Experimental Setup Data and metric. We use all twelve language datasets from MULTILEXNORM: Danish, German, English, Spanish, Croatian, Indonesian–English, Italian, Dutch, Slovenian, Serbian, Turkish, and Turkish–German (two are code-switched pairs). The data is pre-tokenized and word-aligned: each input token is paired with its gold normalized form, and most tokens are already standard and map to themselves. Table 1 lists the sizes; the datasets differ widely both in size (from about 6k to 57k training tokens) and in how much normalization they need (from6.6%to37%of tokens changed), which matters for how much a language can gain from partners. We keep the official train/test splits so that our per-language numbers sit on the same footing as the published MULTILEXNORM results, and for the five languages without a development split we hold out10%of training as development. No data outside these training splits is used at any point: the model has no pretraining stage, and every run below starts from random initialization. We report the benchmark’s official Error Reduc- tion Rate (ERR), which measures how much of the achievable normalization a system performs relative to doing nothing: ERR = acc− LAI 1− LAI ,(1) whereaccis token accuracy andLAI(“leave-as- is”) is the accuracy of copying the input unchanged. ERR= 1is perfect; ERR= 0means no better than copying; ERR< 0means the system corrupts more than it fixes. Model. Every experiment uses the same model: a character-level encoder–decoder Transformer (Vaswani et al., 2017) that reads a raw token one character at a time and generates its normalized form one character at a time. Encoder and decoder LangTrainTest% changed Danish16,4483,7589.2 German15,0065,08217.2 English35,21629,4217.6 Spanish7,1896,6357.7 Croatian54,41615,6956.6 Indo.–Eng.13,9494,36614.7 Italian12,6451,9967.3 Dutch12,3815,41329.7 Slovenian44,94415,02114.8 Serbian56,82317,1197.4 Turkish6,4431,63937.0 Turkish–Ger.12,7733,73524.1 Table 1: The twelve MULTILEXNORM datasets: train- ing and test tokens, and the share of training tokens whose gold form differs from the input (“% changed”). SettingValue Architecturechar Transformer (enc–dec) Layers (enc / dec)3 / 3 Model width128 Attention heads4 Feed-forward width512 Parameters1.49M Max token length32 characters OptimizerAdamW, lr 3×10 −4 Batch size256 tokens Max steps8,000 (early stopping) Seeds3 Table 2: Model and training settings, identical across all conditions. each have three layers, a model width of128, four attention heads, and a feed-forward width of512; a single character vocabulary (the union of charac- ters over all languages, plus start, end, and padding symbols) is shared by every language. This comes to1.49M parameters in total (Table 2). Holding this model fixed across every setting is the crux of the study: any change in per-language accuracy as we add languages reflects how that fixed capacity is shared, not a larger or smaller model. We keep the model small and train it from scratch, rather than fine-tuning a pretrained byte-level model such as BYT5 (Xue et al., 2022), for one reason: a pre- trained model arrives with an unknown amount of multilingual capacity already inside it, and we would then be measuring how that inherited ca- pacity gets redistributed, not how a known, fixed capacity is shared. Training from scratch makes capacity a quantity we set, not one we inherit. The cost of that choice is absolute performance, and we return to it when two languages turn out to be normalized worse than not at all. Protocol.For each number of languageskfrom1 to12, we train the model jointly on random size-k subsets of the twelve languages. Subsets are drawn with a coverage constraint so that every language appears in a comparable number of them; other- wise, at smallk, some languages would never be trained. We repeat the whole sweep over three random seeds, giving390training runs. Training pools the chosen languages’ data, shuffles it, and optimizes with AdamW; we stop early when mean development ERR over the trained languages stops improving, and keep the best checkpoint. We then measure each trained model’s ERR on the test set of every language it was trained on. For a language, its mean ERR across all subsets of a given size, plotted againstk, is its frontier, and thekat which that mean is highest is its optimumk ∗ . Because greedy decoding of full test sets dominates runtime, during the sweep we estimate each language’s test ERR on a fixed sample of up to 2,000 test tokens. Data-controlled ablation. Adding languages does two things at once: it changes the mix the model must serve, and it increases the total train- ing data. To separate them, we repeat the entire protocol while capping the total number of training tokens at a fixed budget, so that a twelve-language model sees no more data than a one-language model. Any decline that survives this control can- not be a data-volume effect. Typology.For each language we compute its iso- lation: the mean URIEL syntactic distance to the other eleven. Turkish and Indonesian–English are the two clear outliers; the European languages clus- ter tightly. 4 Results 4.1 Normalization has a curse of multilinguality Figure 1 shows the frontier. Two languages, Span- ish and Italian, are set aside first: on both the model scores below zero at everyk, meaning it corrupts more than it repairs, so a best number of partners is not defined for them. We take them up in their own right below, and average here over the remaining ten. Across those ten, mean ERR peaks atk = 2 (0.316), barely above training alone (0.312), and then falls steadily to0.191atk = 12, about40% below the peak. The fall is not an artifact of aver- aging noisy curves. Every one of the ten languages ends lower atk = 12than at its own best, and nine of the ten are worse with eleven partners than with none. Six peak at an interiork ∗ ; the remaining four are best trained alone, and the mean optimum across all ten isk ∗ = 2.5. A few partners can help, but most only crowd. The per-language curves (Figure 2, Table 3) fill in the picture. English is the sharpest case: it does best on its own, at an ERR of0.384, and the trend runs downward from there to0.088atk = 12. En- glish has enough data to stand alone, so sharing capacity only dilutes it. The four languages whose optimum isk ∗ = 1(English, German, Dutch, Turkish–German) gain nothing from any partner. The other six instead take a few partners before crowding sets in: Danish and Slovenian peak at k = 2, Indonesian–English atk = 3, Croatian and Serbian atk = 4, and Turkish, the smallest dataset in the benchmark and the one needing the heaviest normalization, atk = 6, later than any other lan- guage among the ten, though its ERR reaches only 0.125 even at its best. Why Spanish and Italian fail, and what follows for the claim. These two behave unlike the rest, and the pattern is worth stating exactly, because it bears on how the aggregate should be read. Both score below zero at everyk: the model changes more tokens than it repairs. Both are at their worst trained alone (−0.847Spanish,−0.614Italian), both are better at twelve languages than at one (−0.451and−0.304), and both reach their best atk = 10andk = 8, later than any other lan- guage here, where the largest optimum among the remaining ten is k ∗ = 6. That direction rules out one explanation. Were the failure caused by pooling with languages that normalize more heavily, adding partners would deepen it; instead adding partners reduces it, and the single-language setting, in which no partner exists, is the worst case of all. What the direction is consistent with is a shortage of in-language signal. Spanish and Italian pair a low change rate with a small training set:7.7%and7.3%of tokens re- quire any change, against37.0%for Turkish, on only7,189and12,645training tokens (Table 1). That combination leaves them the fewest changed tokens of any dataset here, roughly550for Spanish and920for Italian, where no other language falls below1,500. Croatian has an even lower change rate (6.6%) but a much larger training set, so it sees about3,600changed tokens and does not fail. Langk ∗ ERR k=1 ERR k ∗ ERR k=12 Danish2.450.498.285 German1.108.108.010 English1.384.384.088 Spanish10−.847−.387−.451 Croatian4.219.246.196 Indo.–Eng.3.300.353.245 Italian8−.614−.190−.304 Dutch1.464.464.164 Slovenian2.459.507.346 Serbian4.259.329.252 Turkish6.102.125.102 Turkish–Ger.1.377.377.225 Table 3: Per-language results (growing-data setting). k ∗ is the optimal number of jointly trained languages; ERR is shown at one language, at the optimum, and at all twelve. Spanish and Italian have negative ERR throughout and, unlike every other language, improve as partners are added; they are excluded from the aggregate frontier and discussed in Section 4.1. Among the other ten, six peak at an interior k ∗ . What is scarce for Spanish and Italian is the abso- lute amount of in-language evidence about what to change, not the proportion. We did not anal- yse the model’s error types, so we give this as the reading the data supports and not as a measured mechanism. One consequence is definitional. Theirk ∗ values are not optima in the sense the other ten have; they mark the least-harm point on a curve that never becomes useful, which is why a best number of partners is not a meaningful quantity for them. Two consequences bear on the headline. First, these two serve as an internal control: the same sweep, applied where the binding constraint dif- fers, yields a rising curve instead of a falling one, so the decline in the other ten is not an artifact of how subsets were drawn or how ERR was com- puted. Second, the aggregate does not depend on excluding them. Averaging all twelve still gives a clear curse, peaking at0.177(k = 4) and falling to0.097atk = 12, a45%drop against the40% we report for the ten. We exclude them to avoid averaging over curves whose shape reflects a floor effect, not in order to obtain the result. What this does bound is scope. At the capacity we fixed the model is not merely crowded; it is below the level needed to serve the two datasets that carry the fewest changed tokens at all. The frontier we map is therefore the frontier for a model of that size, and a larger one might normalize both successfully and peak later. 123456789101112 number of jointly trained languages, k 0.10 0.15 0.20 0.25 0.30 0.35 0.40 mean ERR (e f ective langs.) growing data fixed data (ablation) Figure 1: Mean ERR against the number of jointly trained languagesk, averaged over the ten languages the model normalizes effectively. Both settings peak atk = 2and then decline steadily: the curse of mul- tilinguality. Holding total data fixed (dashed) makes the drop steeper, so the decline is not a matter of data volume. Shaded bands are 95% confidence intervals across languages. 4.2 The decline is not a data-volume effect A natural worry is that the drop simply reflects data volume: in our main setting the total train- ing pool grows withk, and one might suspect the frontier tracks that. The fixed-data-budget control (dashed line in Figure 1) rules this out. With the total number of training tokens capped, the peak and the decline remain; in fact the fall is steeper (about52%atk = 12versus40%) and the per- language optima move earlier (meank ∗ from2.5 to2.1). Whether we let the data grow or hold it fixed, adding languages past a small number hurts. The decline is therefore a cost of making one fixed-size model serve more languages at once, not a matter of how much data is available. Several mechanisms could contribute to that cost (fewer effective parameters per language, less training ex- posure per language under a fixed budget, and inter- ference between languages’ representations), and our design does not separate them; we refer to them together as capacity. What the ablation establishes is the negative result that matters in practice: the drop is not bought back by adding data. 4.3 Typology does not predict the sweet spot It is tempting to think a language that is typolog- ically far from the others should prefer fewer co- training partners. We find no dependable support, and if anything the trend runs the other way. Cor- relating each effective language’s isolation with its 0.0 0.2 0.4 da (k * = 2) 0.00 0.05 0.10 0.15 de (k * = 1) 0.0 0.1 0.2 0.3 0.4 en (k * = 1) −0.75 −0.50 −0.25 0.00 es (k * = 10) 0.0 0.1 0.2 hr (k * = 4) 0.0 0.1 0.2 0.3 0.4 iden (k * = 3) −0.8 −0.6 −0.4 −0.2 0.0 it (k * = 8) 0.0 0.2 0.4 nl (k * = 1) 510 0.0 0.2 0.4 sl (k * = 2) 510 0.0 0.1 0.2 0.3 sr (k * = 4) 510 0.00 0.05 0.10 0.15 tr (k * = 6) 510 0.0 0.1 0.2 0.3 0.4 trde (k * = 1) number of jointly trained languages, k ERR Figure 2: Per-language frontiers (growing-data setting). Dotted lines mark each language’s optimumk ∗ . Lan- guages well served alone (e.g. English, Dutch) gain nothing from partners; the rest benefit from a few before crowding hurts. optimumk ∗ (Figure 3) gives a positive coefficient (r = 0.69,p = 0.04), with isolated languages preferring more partners, not fewer. But the fig- ure shows why this should not be trusted: the re- lationship rests entirely on the two most isolated languages, Turkish and Indonesian–English, while the eight European languages form a tight cluster with no internal trend. And it is brittle: includ- ing the two languages the model fails on erases it (r = 0.25,p = 0.45over all twelve). We therefore draw no typological rule for how many languages to train on. What comes closer in our data is how well a language is served on its own, with those that already do well alone wanting fewer partners (r =−0.58), though across ten languages this too falls short of significance (p = 0.08) and we re- port it as the more promising direction rather than a rule. Training-set size by itself predicts nothing (r = 0.18, p = 0.62). 5 Discussion For anyone building a compact multilingual nor- malizer, the takeaway is concrete: more languages are not automatically better. For most languages here the best model used only a few, typically one to four, and some languages were best off training alone. Pooling all twelve was, on average, worse than using fewer. The pattern looks like a trade-off. A few extra languages can help: five of the ten do best with one to three partners, and a sixth, Turkish, with five; plausibly some noise patterns carry over between languages. Past that, adding languages hurts, and 0.200.250.300.350.40 typological isolation (mean URIEL distance) 1 2 3 4 5 6 optimal number of languages, k * 8 European languages Turkish Indo.–Eng. r= 0.69, p= 0.04 — driven by two points Figure 3: Typological isolation versus optimumk ∗ for the ten effective languages. The positive correlation (r = 0.69,p = 0.04) rests entirely on the two most isolated languages (Turkish, Indonesian–English); the European languages cluster with no internal trend, and the correlation vanishes over all twelve (r = 0.25). We draw no reliable typological rule. the fixed-data control shows this is not bought back by more data: one fixed-size model can only serve so many languages at once. The obvious lever is capacity. A larger model should tolerate more lan- guages before the curse sets in, and remedies from large multilingual models, such as per-language adapter modules that avoid sharing all parameters (Pfeiffer et al., 2020), may transfer to this setting. How the peak moves with model size, and whether lightweight per-language capacity flattens it, is a natural next step. Two limits should be read alongside the result rather than after it. We vary the number of lan- guages at one model size and one model fam- ily, so what we establish is that a curse exists in this regime and where its peak falls here, not that the peak sits at one to four languages generally. The Spanish and Italian failures make the point concrete: at this capacity the model is not only crowded but below the level needed to serve lan- guages with little changed-token evidence at all, and a model large enough to fix that might well peak later. We would expect the shape to persist and the peak to move right, but we have not mea- sured it, and a reader planning a system should treat “a few languages” as a finding about compact models, not a universal setting. Finally, typology gives no dependable handle. The intuitive story, that isolated languages want fewer partners, is not what we see; if anything the two most isolated languages want more, but that rests on two data points and disappears in the full set. What comes closest to tracking a language’s optimum is how well it is served on its own rather than its typology, but at ten languages even that is a direction and not a rule (r =−0.58, p = 0.08). 6 Conclusion Lexical normalization has a curse of multilingual- ity. Training a single fixed-size model on more languages helps only briefly: per-language accu- racy peaks at a handful of languages and then falls. A control that holds the data fixed shows the fall is not a matter of data volume but of many languages sharing one fixed model. A language’s typological distance from the rest does not reliably tell you where its sweet spot lies. For anyone building a compact multilingual normalizer, the takeaway is to use fewer languages rather than pool everything into one model. Limitations Our study fixes a single small model size; the peak we observe is expected to shift with capacity, and we do not map that dependence here. Relatedly, our design does not separate the mechanisms be- hind the decline. Fewer effective parameters per language, less training exposure per language un- der a fixed budget, and interference between lan- guages all move together as languages are added, so we treat them jointly as capacity rather than pin down which dominates. We use one model family, a character-level Transformer trained from scratch, and results may differ for large pretrained byte-level models. On two of the twelve languages (Spanish, Italian) our small model underperforms the leave-as-is baseline; we report them transpar- ently but exclude them from the aggregate, and a larger or pretrained model might well normal- ize them successfully. During the sweep we esti- mate test ERR on a fixed sample of tokens per lan- guage for efficiency, and we average over random language subsets rather than all possible subsets. That averaging is deliberate, since our question is how many partners a language wants, but it does mean we cannot say which partners help: the de- sign marginalizes over subset composition by con- struction, and we did not retain per-subset records that would let us recover it. Identifying good and bad language pairings is a natural follow-up and would need the sweep re-run with subset-level log- ging. Finally, the typology analysis covers only twelve languages, mostly European, uses syntactic URIEL vectors, and maps the two code-switched datasets onto a single matrix language; with so few and so clustered a sample, we treat the typology question as open rather than settled. References Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019. Massively multilingual neural machine translation in the wild: Findings and chal- lenges. arXiv preprint arXiv:1907.05019. Timothy Baldwin, Marie Catherine de Marneffe, Bo Han, Young-Bum Kim, Alan Ritter, and Wei Xu. 2015. Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normalization and named entity recognition. In Proceedings of the Workshop on Noisy User-generated Text (W-NUT), pages 126–135. Tyler A. Chang, Catherine Arnett, Zhuowen Tu, and Benjamin K. Bergen. 2024. When is multilingual- ity a curse? language modeling for 250 high- and low-resource languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 4074–4096. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics (ACL), pages 8440–8451. Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. In Advances in Neural Information Processing Systems (NeurIPS), pages 7059–7069. Bo Han and Timothy Baldwin. 2011. Lexical normalisa- tion of short text messages: Makn sens a #twitter. In Proceedings of the 49th Annual Meeting of the Asso- ciation for Computational Linguistics (ACL), pages 368–378. Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics (EACL), pages 8–14. Jonas Pfeiffer, Ivan Vuli ́ c, Iryna Gurevych, and Se- bastian Ruder. 2020. MAD-X: An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673. David Samuel and Milan Straka. 2021. ÚFAL at Multi- LexNorm 2021: Improving multilingual lexical nor- malization by fine-tuning ByT5. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 483–492. Rob van der Goot, Alan Ramponi, Arkaitz Zubiaga, Bar- bara Plank, Benjamin Muller, Iñaki San Vicente Ron- cal, Nikola Ljubeši ́ c, Özlem Çetino ̆ glu, Rahmad Ma- hendra, Talha Çolako ̆ glu, et al. 2021. MultiLexNorm: A shared task on multilingual lexical normalization. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 493–509. Rob van der Goot and Gertjan van Noord. 2017. MoNoise: Modeling noise using a modular normal- ization system. Computational Linguistics in the Netherlands Journal, 7:129–144. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems (NeurIPS), pages 5998–6008. Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. 2020. On negative interference in multilingual mod- els: Findings and a meta-learning treatment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4438–4450. Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transac- tions of the Association for Computational Linguis- tics (TACL), 10:291–306.