Paper deep dive
Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+
Sherzod Hakimov, Karl Osswald, Jelle Psurek, Eszter Bukovszky, A. Altar Lüser, David Schlangen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We evaluate large language models (LLMs) as language agents playing goal-directed dialogue games in self-play across 30 languages: the 24 official EU languages plus six others. Unlike static or preference-based evaluation, this paradigm is multi-turn, reference-free and programmatically scored, and because the game mechanics are language-agnostic it extends to a new language by localising a fixed set of prompt and word-list files. Evaluating nine open-weight and commercial LLMs, we find that no open-weight model covers the EU-24 well: in every official language both commercial systems outscore every open-weight model, and the two weakest average below 40 points across the EU-24. The commercial systems stay ahead even in languages with four orders of magnitude less public web text, showing that linguistic parity is achievable, but not from public crawls alone. A model's home region lifts it without closing the gap: Chinese is the strongest of all 30 languages for two Chinese-developed models, yet the best Chinese score of any model belongs to a US commercial system. Coverage is also not parity of service. Pooled over models and languages, the median non-English language costs 31% more to run than English, and scores 10% lower.
Tags
Links
- Source: https://arxiv.org/abs/2608.01395v1
- Canonical: https://arxiv.org/abs/2608.01395v1
Trouble viewing inline? Open PDF directly →
Full Text
88,236 characters extracted from source content.
Expand or collapse full text
Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+ Sherzod Hakimov, Karl Osswald, Jelle Psurek, Eszter Bukovszky A. Altar Lüser, David Schlangen 1 * Computational Linguistics, Department of Linguistics University of Potsdam, Germany 1 German Research Center for Artificial Intelligence (DFKI), Berlin, Germany firstname.lastname@uni-potsdam.de Abstract We evaluate large language models (LLMs) as language agents playing goal-directed dialogue games in self-play across 30 languages: the 24 official EU languages plus six others. Un- like static or preference-based evaluation, this paradigm is multi-turn, reference-free and pro- grammatically scored, and because the game mechanics are language-agnostic it extends to a new language by localising a fixed set of prompt and word-list files. Evaluating nine open-weight and commercial LLMs, we find that no open-weight model covers the EU-24 well: in every official language both commer- cial systems outscore every open-weight model, and the two weakest average below 40 points across the EU-24. The commercial systems stay ahead even in languages with four orders of magnitude less public web text, showing that linguistic parity is achievable, but not from public crawls alone. A model’s home region lifts it without closing the gap: Chinese is the strongest of all 30 languages for two Chinese- developed models, yet the best Chinese score of any model belongs to a US commercial system. Coverage is also not parity of service. Pooled over models and languages, the median non- English language costs 31% more to run than English, and scores 10% lower. 1 Introduction “Every person may write to the institutions of the Union in one of the languages of the Treaties and must have an answer in the same language.” (EU Charter of Fundamental Rights, Art. 41(4) (EU Charter 2012)) * Contributions: SH initiated and managed the project, wrote the publication, coded the game localization pipeline and ran all experiments. KO ran the open-weight model exper- iments, verified Chinese and Greek translations and applied fixes that improved all games. JP set up the manual translation and verification pipeline and verified the German language. EB set up the initial pipeline for localization and verified Hungarian language. AL verified the Turkish language. DS supervised the project, and edited the main part of the paper. This commitment to equal standing becomes hol- low when the AI systems that mediate such inter- actions perform unevenly across languages. Large language models are known to do exactly that (Lai et al., 2023; Üstün et al., 2024), which affects who benefits from them. Tests of LLM ability must therefore cover every target language, and for the EU that means all languages of the Treaties. What a benchmark measures matters as much as which languages it covers. Real language use is dialogic: it consists of sustained, goal-directed interaction in which each contribution builds on the last, not of one-shot question answering. Assessing multi- lingual ability therefore calls for evaluation that is interactive and spans multiple capabilities in both high- and low-resource languages. Existing multilingual evaluation falls short of this along four dimensions, on which Table 1 com- pares the main benchmarks. Most are single-turn: the dominant approach translates static English test sets into many languages (Singh et al., 2025; Xuan et al., 2025; Han et al., 2025; Thellmann et al., 2024), and native-source counterparts (Romanou et al., 2025; Zhang et al., 2023; Isbarov et al., 2025) avoid translation artefacts but are single-turn as well. Many score a selection among given options rather than open-ended generation. Few score ad- herence to formal constraints explicitly, and the multilingual instruction-following benchmarks that do (He et al., 2024; Li et al., 2025) script their turns in advance rather than making them contingent on the model’s own output. Finally, suites that do span multiple capabilities probe each one in a separate subtask (Huang et al., 2025; Ahuja et al., 2023), so no single episode shows how those capabilities interact. Section 2 details this landscape. We address these gaps with a multilingual, inter- active benchmark in which models are evaluated as agents playing goal-directed dialogue games in self-play (Chalamalasetti et al., 2023; Schlangen et al., 2025). The game mechanics are language- arXiv:2608.01395v1 [cs.CL] 2 Aug 2026 BenchmarkMulti-turn Open-ended generation Instruction following Multiple capabilities Global MMLU (Singh et al., 2025)× MMLU-ProX (Xuan et al., 2025)× M3Exam (Zhang et al., 2023)× INCLUDE (Romanou et al., 2025)× MultiLoKo (Hupkes and Bogoychev, 2025)×✓× MuBench (Han et al., 2025)×✓ Eurolingua (Thellmann et al., 2024)×✓ PolyMath (Wang et al., 2025)×✓× OneRuler (Kim et al., 2025)×✓× BenchMAX (Huang et al., 2025)×✓ Multi-IF (He et al., 2024)✓× XIFBench (Li et al., 2025)×✓× IrokoBench (Adelani et al., 2025)×✓×✓ TUMLU (Isbarov et al., 2025)× Karde ̧s-NLU (Senel et al., 2024)×✓ ProverbEval (Azime et al., 2025)×✓× Ours✓ Table 1: Comparison of multilingual LLM benchmarks along the four dimensions identified as missing in the preceding discussion: evaluation spans multiple dialogue turns (Multi-turn); responses are produced as free-form text rather than selected from options (Open-ended generation); adherence to formal constraints is explicitly scored (Instruction following); multiple distinct capabilities are covered (Multiple capabilities).✓ = yes,× = no. agnostic, so a new language requires only localised game files (prompts, parsing rules, word lists) and no reference answers. This turns full institutional coverage from a data-collection project into a local- isation task, which we use to evaluate interactive language use across all 24 official EU languages. It also lets us ask what static benchmarks cannot: not just how well a model performs in a language, but at what cost. We pair performance with the tokens and dollars it costs per language, and relate both to available web text and the economic weight of the speaker community. We make three contributions. First, we build a game-play benchmark that jointly measures in- struction following and functional capabilities in 30 languages: the 24 official EU languages, plus six more chosen for comparison and anchoring. Sec- ond, we evaluate nine open-weight and commercial LLMs and show that the gap between high- and low-resource languages is large under interactive use, and that it falls almost entirely on the open- weight models. Third, we relate performance to training-data availability, tokeniser fertility and the economic weight of a language 1 . 2 Related Work Single-turn knowledge benchmarks.The dom- inant paradigm machine-translates English test 1 Sourcecode:https://github.com/clembench/ multilingual Leaderboard of LLMs:https://clembench.github.io/ leaderboard.html sets, most prominently MMLU (Hendrycks et al., 2021), into other languages. Global MMLU (Singh et al., 2025), MMLU-ProX (Xuan et al., 2025) and MuBench (Han et al., 2025) follow this recipe, as do PolyMath (Wang et al., 2025) and OneRuler (Kim et al., 2025). Such scores conflate target- language ability with translation quality and stay Anglo-centric (Singh et al., 2025). Native-source benchmarks such as M3Exam (Zhang et al., 2023), INCLUDE (Romanou et al., 2025) and Multi- LoKo (Hupkes and Bogoychev, 2025) arose as a corrective, but remain static, single-turn and contamination-prone, measuring knowledge recall rather than language use. Multi-task suites and frameworks. A second line widens coverage by aggregating many such tasks (Ahuja et al., 2023, 2024; Lai et al., 2023; Üstün et al., 2024; Nielsen, 2023). BenchMAX (Huang et al., 2025) is closest in spirit, but probes each capability in a separate static subtask, so its scores cannot show how capabilities interact. Instruction following and multi-turn evaluation. Multi-IF (He et al., 2024) and XIFBench (Li et al., 2025) evaluate multilingual instruction following through verifiable constraints; Multi-IF adds turns, but they accumulate pre-scripted constraints rather than responding to the model’s output, isolating compliance from what instructions serve. Low-resource coverage and benchmark quality. Regional benchmarks document a persistent per- formance gap for low-resource languages in Africa Benchmark Construction & Validation Validation & Quality Check Automated Translation English Prompt Files Model Evaluation & Score Aggregation Per-Language Scores Multi-turn Dialogue Interactions Prompt in Each Language Benchmarked LLMs ... Cost Analysis Benchmark Analytics [DE] German [FR] French [ET] Estonian [SV] Swedish [ZH] Chinese Figure 1: Overview of the multilingual benchmark construction and score aggregation across evaluated LLMs. (Adelani et al., 2025; Ojo et al., 2025; Adebara et al., 2025), Southeast Asia (Lovenia et al., 2024), India (Kakwani et al., 2020) and the Turkic family (Senel et al., 2024; Isbarov et al., 2025), as well as for morphologically rich languages (Xia et al., 2025). European-language evaluation likewise re- lies on translated English test sets (Thellmann et al., 2024; Barth and Rehm, 2025). Yet such bench- marks correlate poorly with human judgments (Wu et al., 2025), and MCQ scores fluctuate with answer order and prompt (Azime et al., 2025). Game-based evaluation and our approach. A parallel line of work evaluates LLMs as agents in interactive game environments, among them TextWorld (Côté et al., 2019), TextArena (Guertler et al., 2025), TALES (Cui et al., 2025) and GameArena (Hu et al., 2025). These are devel- oped and run in English. The clembench frame- work we build on has been instantiated in English, German and Italian (Schlangen et al., 2025). This shows that the approach ports across languages, but it leaves open whether it scales to a full institu- tional language set, and at what cost. Our bench- mark goes beyond both lines (Table 1): evaluation proceeds through goal-directed dialogue contin- gent on the model’s own moves, with generated rather than frozen instances, scoring formal and functional competence in the same episode across 30 languages. 3 Methodology 3.1 Dialogue Game-Based Evaluation LLM evaluation is dominated by two paradigms (Schlangen et al., 2025). Reference-based eval- uation on static benchmarks offers control over what is tested, but it is single-turn, costly to extend, and prone to leakage and saturation. Preference- based evaluation, such as LM Arena, tests real interactive use but offers no control and is not repli- cable. Schlangen et al. (2025) argue for a third paradigm that combines the strengths of both. Dia- logue game-based evaluation is controlled, repeat- able, multi-turn and reference-free: success is deter- mined by reaching a goal state under formal rules, not by comparison to gold answers. We adopt this paradigm through clembench (Chalamalasetti et al., 2023) and the 14 games listed in Table 2. A game master orchestrates self-play episodes: it prompts one or more player LLMs, validates their responses against formal rules, and scores rule compliance and task suc- cess programmatically. The game mechanics them- selves are language-agnostic. A game scales to any language for which the language-dependent arte- facts exist, namely prompt text, response-parsing rules, feedback messages and, for some games, word lists. No reference answers need to be au- thored. We exploit this to extend clembench to 30 languages, acquiring the artefacts via machine translation with subsequent validation; Figure 1 shows the resulting pipeline, from localisation to per-language score aggregation. This differs from the translate-then-benchmark approach criticised in Section 2. There the translation is the test item and its reference answer; here it is only the interface. Success remains behavioural and is verified against the formal rules of the game, not against a gold answer that translation could corrupt. CapabilityGameDescription Lexical & world knowledgeTabooDescribe a target word without using the listed taboo words. CodenamesGive a one-word clue linking several words on the board. WordleGuess a hidden word from colour-coded letter feedback. Grounded referenceReference GameDescribe a target among distractors for a partner to pick out. Image GameInstruct a partner to recreate a pixel-art grid. Discourse & groundingGuessWhatIdentify a target object through yes/no questions. Match-ItDecide in dialogue whether two ASCII images are identical. Private & SharedTrack which information is private vs. mutually known. Strategic & social reasoning Deal or No DealNegotiate an item split meeting private utility targets. Hot Air BalloonAgree on items to discard under a survival constraint. Clean UpCollaboratively sort objects into correct locations. Spatial reasoning & planningTextMapWorldNavigate a text-based map to reach a goal location. TextMapWorld (Graph) Reason over the graph structure of a multi-room map. TextMapWorld (Room) Plan fine-grained movement within individual rooms. Table 2: The 14 games used in this benchmark, grouped by the capability cluster they primarily assess. 3.2 Game Localisation Pipeline Localising a game means translating three kinds of artefact: the prompt text, the response-parsing rules and the feedback messages. These must stay mutually consistent, since a loose prompt transla- tion can break the corresponding regex. Localisa- tion therefore proceeds in two stages, each carried out by a different model. First, GPT-5.2 translates each game’s file set in a single pass, preserving formatting markers, placeholders, and regex syn- tax. Second, Claude Sonnet 4.5 compares origi- nals against translations, checks that prompts and regexes still agree, and corrects errors. Splitting the stages across providers reduces the risk that system- atic errors of one model go undetected, and neither model is among those we evaluate (Section 4.3). Word lists. Four games additionally require language-specific word lists. For Taboo we ex- tracted target and related words from Concept- Net 5.7 (Speer et al., 2017). For Wordle we took 5-letter noun lemmas from Universal Dependen- cies v2.17 treebanks (Nivre et al., 2020), ranked by corpus frequency, and built the set of valid guesses from Wikipedia dumps. For Codenames and GuessWhat we translated the English lists us- ing the same pipeline. Details are in Appendix A. 3.3 Manual Verification We verified six languages manually, chosen for ty- pological and resource diversity: German, Russian, Chinese, Turkish, Greek and Hungarian. Native speakers inspected the translated files, checked that the parsing rules match the prompt wording, and confirmed the translations were fit to run. This Code Language Code Language Code Language arArabicfrFrenchptPortuguese bgBulgarian gaIrishroRomanian csCzechhrCroatianruRussian daDanishhuHungarian skSlovak deGermanitItalianslSlovenian elGreekltLithuanian srSerbian enEnglishlvLatviansvSwedish esSpanishmtMaltesetrTurkish etEstoniannlDutchukUkrainian fiFinnishplPolishzhChinese Table 3: The 30 languages included in the benchmark. was repeated while developing the pipeline, so each round of corrections propagated to all 30 lan- guages; the changes required were minimal, con- cerning keyword choices and punctuation. Each LLM is then run in self-play for every language independently, with prompts, responses, parsing and scoring all operating in the target language. 4 Experimental Setup 4.1 Languages The benchmark covers 30 languages (Table 3): the 24 official EU languages plus six others. Turkish, Arabic, Russian and Ukrainian have large speaker populations, and Chinese allows comparing mod- els developed in China. Serbian, written here in Latin script, is close enough to Croatian to test whether two closely related languages receive com- parable support. The selection spans five fami- lies, Indo-European (Germanic, Romance, Slavic, Baltic, Celtic and Hellenic branches), Uralic, Tur- kic, Afro-Asiatic and Sino-Tibetan, and five scripts: Latin, Cyrillic, Greek, Arabic and Han. 4.2 Games and Capabilities We include 14 dialogue games from the clembench suite, grouped in Table 2 into five capability clus- ters, each targeting a facet of language use that static benchmarks cannot probe. Wordle is ex- cluded for Chinese, because the game requires five- letter words whereas Chinese is character-based. 4.3 Models We evaluate nine LLMs, with details in Table 7. Two are commercial, Claude Opus 4.8 and GPT- 5.4. Seven are recent multilingual open-weight models: GLM-5.2, Mistral-Large-3, Apertus-1.5- 8B, Gemma-4-26B, Qwen3.6-35B, DeepSeek-V4- Pro and Nemotron-3-Ultra. The commercial mod- els were accessed through their provider APIs. GLM-5.2, DeepSeek-V4-Pro, Nemotron-3-Ultra and Mistral-Large-3 were accessed through Open- Router, and Qwen3.6, Gemma-4 and Apertus-1.5 were run locally on NVIDIA A100 GPUs. Hyperparameters. All models run in their de- fault configuration with temperature= 1and no other decoding parameters set, so reasoning be- haviour follows each provider’s default. The output limit is 500 tokens per turn for all games except Clean Up and Hot Air Balloon, which use 2,000 tokens as their turns are substantially longer. Each language is evaluated in a single run using the same game instances for every model: 400 episodes per language per model (380 for Chinese), distributed across the 14 games as listed in Table 5, for a total of roughly 108,000 episodes. 4.4 Metrics Each episode is scored along two dimensions. %Played is the share of episodes played to com- pletion without aborting, capturing instruction- following and format compliance. Quality is the mean task-specific main score over non-aborted episodes, reflecting how well the model solved the task when it engaged. Both are aggregated into the clemscore, the normalised product of %Played and Quality scaled to[0, 100], so a high clemscore requires a model to play reliably and perform well. 5 Results 5.1 Performance across Languages Overall Comparison.Table 4 reports clemscore for all models across the 30 languages (%Played and Quality separately in Tables 9 and 10). The LangGPT-5.4Opus-4.8GLM-5.2DS-V4NemotronGemma4Qwen3.6Mistral-L3ApertusAvg ar79.169.160.252.147.149.934.739.311.849.3 bg92.480.668.961.252.360.852.441.717.658.7 cs90.681.760.158.655.752.850.441.814.156.2 da89.983.068.565.058.559.247.444.811.358.6 de90.785.567.370.163.057.942.549.416.260.3 el91.475.569.963.151.347.654.934.719.156.4 en91.884.465.664.868.250.655.654.621.461.9 es85.270.062.057.856.252.944.445.117.054.5 et92.381.865.558.139.841.446.831.58.251.7 fi93.679.661.161.654.850.642.443.418.656.2 fr92.383.567.452.458.052.953.945.820.358.5 ga84.976.340.537.830.416.932.419.36.938.4 hr 93.083.764.256.453.255.246.042.118.756.9 hu81.178.363.452.646.249.640.040.117.652.1 it93.980.966.865.761.059.646.837.114.658.5 lt77.974.159.540.346.237.042.425.716.246.6 lv91.879.961.959.545.449.050.232.714.353.9 mt92.075.645.464.332.734.240.112.32.344.3 nl90.783.566.863.962.760.249.554.315.060.7 pl89.582.364.362.557.159.745.949.622.059.2 pt85.971.256.153.558.155.949.341.819.054.5 ro74.173.061.950.353.943.045.736.017.850.6 ru 89.380.162.362.559.058.053.846.823.059.4 sk90.382.159.459.345.553.648.638.512.954.5 sl86.178.759.855.153.436.648.843.918.853.5 sr90.684.660.758.748.660.242.737.019.655.8 sv87.880.665.265.561.256.845.951.320.559.4 tr91.075.063.363.551.442.041.433.29.452.3 uk93.079.667.055.750.456.750.050.417.757.8 zh91.373.971.362.868.449.061.750.718.060.8 Avg88.878.962.658.553.050.346.940.516.055.1 STD4.94.56.37.08.89.76.19.64.721.2 Table 4: Overall clemscore per language (rows) and model (columns): mean % Played across the 14 games times mean Quality Score across the 14 games, ordered by mean clemscore. Cells are shaded by octile of the clemscore distribution (darker = higher); the best model per language is in bold. dominant pattern is a wide gap between the two commercial models and everything else: GPT-5.4 and Claude Opus 4.8 lead the best open-weight model, GLM-5.2, by 26 and 16 points on average. In every one of the 24 official EU languages, in- cluding Irish, Maltese and Latvian, both commer- cial models score above every open-weight model, by margins running from 5.6 points in Greek to 35.8 in Irish. Across the EU-24 the commercial systems bottom out at 74.1 (GPT-5.4, Romanian) and 70.0 (Opus 4.8, Spanish), while the highest open-weight result anywhere in the benchmark is 71.3 (GLM-5.2, Chinese). Against their own En- glish results, the commercial systems return 80.7% and 83.0% in their weakest EU language, whereas the open-weight models return between 61.7% and 10.7%, so their deficit is a collapse concentrated in specific languages rather than uniform weak- ness. For all seven that weakest language is Irish or Maltese. The closely related Serbian and Croat- Lexical Gr.ref DiscourseStrategic Spatial 25 50 75 GPT-5.4 Opus-4.8 GLM-5.2 DS-V4 Nemotron Gemma4 Qwen3.6 Mistral-L3 Apertus Figure 2: Model performance averages across languages for each language capability (game results are averaged) ian (Section 4.1) do receive comparable support, at 55.8 and 56.9 averaged over models. The material itself is playable throughout: in each of the 30 lan- guages at least one model completes 86% or more of its episodes. Parameter count does not explain the ordering. The 675B Mistral-Large-3 averages 40.5, below the far smaller Gemma-4 and Qwen3.6, so coverage of the EU-24 is not simply a matter of scale. The strongest open-weight model, GLM- 5.2, spans 69.9 (Greek) to 40.5 (Irish) across the EU-24; none of the seven covers the full set. Provenance helps, but it does not confer advan- tage. Chinese is the single best language of all 30 for both Qwen3.6 (61.7) and GLM-5.2 (71.3), while the third Chinese-developed model, DeepSeek-V4- Pro, scores highest in German and places Chinese tenth. Yet the best Chinese result of any model in the benchmark belongs to GPT-5.4, at 91.3 — some 20 points above the strongest Chinese-developed system in its own language. Being trained where a language is spoken lifts a model within its own range without closing the gap to the commercial systems. Averaged over models, Chinese ranks second of the 30 behind English, consistent with its third place in available web text, though its 13- game average flatters it by two to six points relative to languages scored over all 14 (Appendix B). Averaging over languages, Figure 2 breaks per- formance down by capability cluster: lexical knowl- edge and strategic reasoning are hardest, at 39.1 and 48.0, while grounded reference and spatial rea- soning are the strongest, at 71.5 and 67.3. Spatial reasoning is also the cluster that separates the open- weight models most sharply, spanning 70 points between the best and the weakest of the seven (per- language detail in Tables 11–15). Correlation with other Benchmarks. To situ- ate clemscore among established evaluations, we compare the English clemscore ranking of five fron- tier models against their rankings on LMArena 2 , GPQA Diamond (Rein et al., 2023), HLE (Phan et al., 2025) and the agentic BrowseComp (Wei et al., 2025), using Kendall’sτ. The five mod- els are GPT-5.4, Claude Opus 4.8, Nemotron-3- Ultra, GLM-5.2 and DeepSeek-V4-Pro. We re- strict the comparison to English, as those bench- marks cover no other language for all five. Agree- ment is moderate and decreases as the target capa- bility moves away from interactive language use: τ = 0.60against LMArena human preferences, 0.40against GPQA Diamond,0.20against HLE, and0.00against BrowseComp. Much of the dis- agreement comes from a single model: Nemotron- 3-Ultra ranks third in English clemscore but 91st on LMArena, playing the games reliably without being a model users prefer in open-ended chat. These external scores are vendor-reported under differing protocols, so they indicate approximate standing only. Withn = 5these coefficients are descriptive rather than significant, but the ordering is what we would expect if dialogue-game evalua- tion measures something existing benchmarks do not capture, rather than reproducing them. Full scores and ranking charts are in Appendix D. 5.2 Economic Footprint, Data, and Tokenisation Linguistic Economic Footprint. To relate per- formance to the real-world economic value of a language, we introduce the Linguistic Economic Footprint (LEF). We take each country’s nominal GDP, weight it by the share of the population that speaks the language, and sum over the countries where the language holds (co-)official status: LEF(ℓ) = X c∈C ℓ GDP(c)× speakers(ℓ,c) population(c) Prior work weighted demand mainly by speaker population (Blasi et al., 2022); LEF makes the eco- nomic dimension explicit. Sources are given in Appendix G. Figure 3 plots LEF against clemscore. For the open-weight models, performance broadly follows a language’s economic footprint: the Spearman cor- relation is positive for all seven, fromρ = 0.22to 0.78, though only the two strongest are significant 2 https://lmarena.ai/leaderboard/text/overall English German Dutch Swedish Danish Spanish French Portuguese Italian Romanian Russian Polish Czech Slovak Ukrainian Bulgarian Serbian Croatian Slovenian Lithuanian Latvian Finnish Hungarian Estonian Irish Greek Arabic Maltese Turkish Chinese Language $100B $1.0T $10.0T Linguistic Economic Footprint in USD (log scale) Language Family Germanic Romance Slavic Baltic Uralic Celtic Hellenic Semitic Turkic Sino-Tibetan 0 20 40 60 80 100 clemscore Model (clemscore) GPT-5.4 Claude-4.8 GLM-5.2 DeepSeek-V4 Nemotron 3 Gemma 4 Qwen 3.6 Mistral Large 3 Apertus Figure 3: LEF (bars, left axis, log scale) against clemscore (lines, right axis) per language, grouped by family. Open-weight models drop for the smaller-footprint languages, while the commercial systems show no consistent trend, varying over 74.1–93.9 and 69.1–85.5. arbgcsdadeelenesetfifr ga hrhuitltlvmtnlplptroruskslsrsvtruk zh * GPT-5.4 Nemotron-550B Mistral-Large-3 Apertus-1.5-8B Gemma-4-26B Qwen3.6-35B GLM-5.2 DeepSeek-V4 Claude-Opus-4.8 1.872.212.352.001.822.501.321.462.552.821.541.902.152.791.742.632.672.541.612.351.502.082.172.512.242.511.932.342.620.85 1.752.182.242.151.822.461.391.512.763.011.532.262.172.651.702.772.873.191.782.251.582.082.332.512.362.412.082.352.620.94 1.752.182.242.151.822.461.391.512.763.011.532.262.172.651.702.772.873.191.782.251.582.082.332.512.362.412.082.352.620.94 1.752.182.242.151.822.461.391.512.763.011.532.262.172.651.702.772.873.191.782.251.582.082.332.512.362.412.082.352.620.94 2.032.182.292.141.872.601.371.482.753.081.612.142.162.761.702.702.873.001.792.231.582.082.192.522.372.512.072.292.450.75 1.852.262.442.151.922.571.401.542.863.081.632.242.372.761.672.762.963.041.842.281.582.142.242.612.492.672.122.462.740.68 2.402.302.652.242.032.771.351.613.073.301.722.282.632.961.863.203.363.001.952.441.732.242.252.912.742.912.152.712.770.80 2.192.512.492.272.033.001.321.643.063.261.712.302.543.041.813.043.293.051.892.391.712.322.412.812.632.672.173.323.300.66 3.882.453.023.043.594.432.152.474.134.832.573.093.343.622.834.154.074.232.993.482.552.932.983.573.362.792.954.462.941.10 Figure 4: Token fertility per model and language. Chinese is character-based and not directly comparable. atn = 30(Figure 7). The two commercial systems break this pattern, atρ =−0.13and−0.02. GPT- 5.4 scores as well on the three smallest footprints in our set (Maltese 92.0, Estonian 92.3, Latvian 91.8) as on English (91.8), and Claude Opus 4.8 drops by 8.8 points from English to Maltese (84.4 to 75.6), while remaining ahead of every open-weight model in each of those languages. The same pattern holds for the volume of crawled text available per language, which spans four orders of magnitude across our set. It cor- relates with open-weight performance (ρ = 0.72, p < 0.001) but not with the commercial systems (ρ = 0.03and0.06, both n.s.). Appendix H gives the per-language figures. The two explanations are not independent, since richer economies also pro- duce more web text, and neither fully determines performance: Serbian scores above Spanish (55.8 against 54.5) on a seventieth of the crawled text, and Croatian above Portuguese (56.9 against 54.5) on a tenth. LEF adds the demand side, spanning more than three orders of magnitude within our language set. If provider investment tracks market size, it alone will not close the gap for the smallest official languages, and the models that do serve them well are the closed ones. Equal linguistic standing therefore depends on effort that mar- ket incentives do not reward, which is where public funding and open, language-targeted re- sources matter most. Token Fertility. Figure 4 reports each model’s token fertility, the average number of tokens per word, which measures how well a tokeniser’s vo- cabulary covers a language. The≈1.4 observed for English with almost every model means words are mostly kept intact; the≈3.0 for Maltese means every word is split into three pieces. The same con- arbgcsdadeelenesetfifr ga hrhuitltlvmtnlplptroruskslsrsvtrukzh Gemma-4 DeepSeek-V4 GPT-5.4 Mistral Large 3 Claude GLM-5.2 Nemotron 3 Ultra Qwen 3.6 Apertus-v1.5-8B 1.080.890.790.971.100.681.001.820.671.080.930.170.880.791.000.650.630.521.091.061.930.661.230.900.571.090.900.631.091.06 0.680.770.690.700.790.471.001.010.550.650.620.310.620.530.870.460.560.530.710.690.990.480.640.630.670.650.770.590.524.40 0.670.770.780.850.830.711.000.690.800.830.840.420.800.670.890.390.770.750.860.770.650.590.760.760.760.710.850.700.780.94 0.810.860.700.720.740.701.001.100.740.590.960.360.730.700.790.540.620.240.680.611.240.520.690.560.890.650.731.130.680.63 0.660.810.770.830.740.541.000.660.730.700.810.340.810.680.800.320.720.620.820.740.670.690.790.750.750.840.820.640.751.03 0.510.710.610.630.750.571.001.130.570.580.760.180.610.470.650.330.480.310.810.570.820.590.700.630.550.550.720.400.733.18 0.720.750.640.630.770.401.001.110.480.590.700.220.610.500.790.410.430.190.780.611.170.600.550.430.620.550.720.520.502.57 0.790.700.810.590.590.601.000.930.650.620.950.220.930.480.660.320.740.280.550.581.060.530.800.540.680.690.560.400.560.92 0.550.140.080.370.140.241.000.920.090.090.420.020.380.680.120.430.030.080.060.601.180.540.070.650.300.630.180.050.250.21 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 clemscore per $ relative to English (1.0 = parity) Figure 5: Value—clemscore points per dollar—normalised against the same model’s English value: 1.0 matches what an English user of that model gets, lower values mean the same money buys less. tent then occupies roughly twice the context, and since APIs bill per token, this translates directly into higher cost (Ahia et al., 2023; Petrov et al., 2023). Within the EU-24 every provider shows the same gradient: English, Spanish, Portuguese, French, Italian, Dutch and German sit at 1.4–1.9 tokens per word, while Finnish, Maltese, Latvian, Lithua- nian, Estonian, Hungarian and Greek run at 2.6–3.1. Fertility is a downstream symptom rather than a cause: subword vocabularies are fitted to the text available, so languages with little of it get fewer dedicated units (Rust et al., 2021; Velayuthan and Sarveswaran, 2025). The profiles are near-identical across providers, which indicates that no provider currently invests in vocabulary allocation for all EU-24 languages; three of the nine are identical by construction, sharing Mistral’s Tekken tokeniser (Appendix F). Claude Opus 4.8 is the one model that departs from this profile: it averages roughly 50% more tokens per word than the median (Ap- pendix F). The tokeniser prices languages be- fore a single forward-pass is made: speakers of Finnish, Maltese or Estonian spend roughly twice as many tokens, and therefore roughly twice the money and context, as English speak- ers for the same task. 5.3 Cost vs. Performance Strong absolute performance does not imply a lan- guage is served efficiently, so we express results as value: clemscore points per dollar, normalised against the same model’s English value (Figure 5). Costs come from the token counts in the interaction transcripts and each model’s list price (Table 7); the per-language breakdown is in Table 8. The commercial systems, despite leading on raw scores, deliver markedly less value outside En- glish: all 29 non-English languages cost GPT-5.4 more than English, and 28 of 29 cost Claude Opus 4.8 more, Chinese being the exception. In high- resource languages the premium is moderate and performance close to parity, so value stays near the English level. In the lower-resourced languages users pay more and get less: the median model pays2.2×the English cost for Irish while return- ing58%of its English score. The mean,3.6×, is inflated by Apertus, whose Irish run costs15× its English one. Gemma-4 is the exception, with better-than-English value in twelve languages, led by Portuguese (1.93) and Spanish (1.82). Pooled over all models and non-English lan- guages, the median language costs31%more to run than English while scoring10%lower. Higher fer- tility inflates the billed token count, and lower per- formance reduces what those tokens buy. Speakers of the smaller EU languages are thus charged a double premium: equal access to the same model, at the same price, still yields unequal service. 6 Conclusion We presented an interactive benchmark evaluating LLMs as agents in goal-directed dialogue games across 30 languages: the 24 official EU languages plus six others. Built by localising game files rather than translating a static test set, it is cheap to extend and to re-run as models evolve. No open-weight model we evaluated covers the EU-24: in every offi- cial language both commercial systems outscore ev- ery open-weight model, and the open-weight mod- els pass 70 points in only two cells between them. That the commercial systems stay strong across the whole set cuts two ways. It shows linguistic parity is achievable: Irish and Maltese are not intrinsi- cally harder, only under-served. Yet parity cannot come from public crawls, which offer four orders of magnitude less text for these languages. The mod- els that serve Europe’s smaller languages well are closed and non-European: equality means either accepting that dependence or building resources beyond the open web, such as public broadcast archives. Limitations Our selection centres on the 24 official EU lan- guages plus six others, so most languages of Africa, South and Southeast Asia and East Asia are absent; extending to them requires only the localised game files, which makes broader coverage a question of funding rather than of method. The open-weight set spans 8B to 675B parameters, so the open-versus- closed contrast is partly confounded with scale, though not entirely, since the 675B Mistral-Large- 3 trails much smaller open-weight models. We evaluate one recent variant per family and omit the EU-funded models such as EuroLLM, Teuken and Salamandra, which would speak most directly to the policy framing of Section 5.2; budget and run time, at two to four days per model across 30 languages, did not allow more. We consider those models the most useful addition to a future version. Each language is run once, at temperature 1 and without a fixed seed, so we report no variance esti- mate and small differences between adjacent cells should not be read as meaningful. We also have no per-language human or random baseline, so scores are interpreted relative to other models rather than against an absolute difficulty scale. Games con- tribute unequally: Wordle depends on a five-letter constraint that is natural in some orthographies and marginal in others, and is dropped for Chinese, which raises Chinese scores by two to six points rel- ative to languages scored over all 14 games. Only six languages were verified by native speakers, and a small number of game–language pairs still show near-zero completion for reasons we attribute to parsing rather than to competence. Finally, reference-free scoring removes the gold answers that static benchmarks leak, but does not make the benchmark contamination-proof. Univer- sal Dependencies, Wikipedia and ConceptNet are public, as is clembench with its published English results, so a model may have seen individual target words or transcripts of earlier runs. What cannot leak is the ability to play: a memorised Wordle target does not help a model that fails to follow the feedback protocol over five turns. Ethics Statement The evaluated models can in principle produce in- accurate, biased, or unsafe text. In our setting this risk is limited. Outputs are generated only within tightly constrained dialogue games and serve only as input to the functions that score the interactions. They are not used for training, are not shown to end users, and are released only as game transcripts for reproducibility. The texts used for the token- fertility computation come from Wikipedia and are available under a free licence. No personal or sensi- tive data is collected, and no human subjects were involved beyond the verification of the translated game files. References Ife Adebara, Hawau Olamide Toyin, Nahom Tesfu Ghe- bremichael, AbdelRahim A. Elmadany, and Muham- mad Abdul-Mageed. 2025. Where are we? evalu- ating LLM performance on African languages. In Proceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 32704–32731, Vienna, Austria. Association for Computational Linguistics. David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba Oluwadara Al- abi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chia- maka Ijeoma Chukwuneke, Happy Buzaaba, Bless- ing K. Sibanda, Godson Koffi Kalipe, Jonathan Muki- ibi, Salomon Kabongo Kabenamualu, Foutse Yue- hgoh, Mmasibidi Setaka, Lolwethu Ndolela, and 8 others. 2025. Irokobench: A new benchmark for african languages in the age of large language mod- els. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 - Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, pages 2732–2757. Association for Computa- tional Linguistics. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language mod- els. In Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing, pages 9904–9923, Singapore. Association for Com- putational Linguistics. Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Ut- tama Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 4232– 4267. Association for Computational Linguistics. Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Mohamed Ahmed, Ka- lika Bali, and Sunayana Sitaram. 2024. MEGA- VERSE: benchmarking large language models across languages, modalities, models and tasks. In Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 2598–2637. Association for Computational Linguistics. Israel Abebe Azime,Atnafu Lambebo Tonja, Tadesse Destaw Belay, Yonas Chanie, Bontu Fufa Balcha, Negasi Haile Abadi, Henok Biadglign Ademtew, Mulubrhan Abebe Nerea, Debela De- salegn Yadeta, Derartu Dagne Geremew, Assefa Ats- biha tesfau, Philipp Slusallek, Thamar Solorio, and Dietrich Klakow. 2025. Proverbeval: Exploring LLM evaluation challenges for low-resource lan- guage understanding. In Findings of the Association for Computational Linguistics: NAACL 2025, Albu- querque, New Mexico, USA, April 29 - May 4, 2025, Findings of ACL, pages 6250–6266. Association for Computational Linguistics. Fabio Barth and Georg Rehm. 2025. Multilingual eu- ropean language models: Benchmarking approaches and challenges. CoRR, abs/2502.12895. Damian Blasi, Antonios Anastasopoulos, and Gra- ham Neubig. 2022. Systematic inequalities in lan- guage technology performance across the world‘s languages. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486–5505, Dublin, Ireland. Association for Computational Linguistics. Kranti Chalamalasetti, Jana Götze, Sherzod Haki- mov, Brielen Madureira, Philipp Sadler, and David Schlangen. 2023. clembench: Using game play to evaluate chat-optimized language models as conver- sational agents. In Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Pro- cessing, EMNLP 2023, Singapore, December 6-10, 2023, pages 11174–11219. Association for Computa- tional Linguistics. Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2019. Textworld: A learning environment for text-based games. In Computer Games, pages 41–75, Cham. Springer International Publishing. Christopher Zhang Cui, Xingdi Yuan, Ziang Xiao, Prithviraj Ammanabrolu, and Marc-Alexandre Côté. 2025. TALES: text adventure learning environment suite. CoRR, abs/2504.14128. EU Charter 2012. 2012. Charter of fundamental rights of the European Union. Official Journal of the Eu- ropean Union, OJ C 326, 26.10.2012, p. 391–407. Art. 22: “The Union shall respect cultural, religious and linguistic diversity.”. Leon Guertler, Bobby Cheng, Simon Yu, Bo Liu, Leshem Choshen, and Cheston Tan. 2025. Textarena. CoRR, abs/2504.11442. Wenhan Han, Yifan Zhang, Zhixun Chen, Binbin Li, Haobin Lin, Bingni Zhang, Taifeng Wang, Mykola Pechenizkiy, Meng Fang, and Yin Zheng. 2025. Mubench: Assessment of multilingual capabilities of large language models across 61 languages. CoRR, abs/2506.19468. Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, Shruti Bhosale, Chenguang Zhu, Karthik Abinav Sankararaman, Eryk Helenowski, Melanie Kambadur, Aditya Tayade, Hao Ma, Han Fang, and Sinong Wang. 2024. Multi-if: Benchmark- ing llms on multi-turn and multilingual instructions following. CoRR, abs/2410.15553. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. Lanxiang Hu, Qiyu Li, Anze Xie, Nan Jiang, Ion Sto- ica, Haojian Jin, and Hao Zhang. 2025. Gamearena: Evaluating LLM reasoning through live computer games. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Xu Huang, Wenhao Zhu, Hanxu Hu, Conghui He, Lei Li, Shujian Huang, and Fei Yuan. 2025. Benchmax: A comprehensive multilingual evaluation suite for large language models. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 16751– 16774. Association for Computational Linguistics. Dieuwke Hupkes and Nikolay Bogoychev. 2025. Multi- loko: a multilingual local knowledge benchmark for llms spanning 31 languages. CoRR, abs/2504.10356. Jafar Isbarov, Arofat Akhundjanova, Mammad Hajili, Kavsar Huseynova, Dmitry Gaynullin, Anar Rzayev, Osman Tursun, Aizirek Turdubaeva, Ilshat Saetov, Rinat Kharisov, Saule Belginova, Ariana Kenbayeva, Amina Alisheva, Abdullatif Köksal, Samir Rustamov, and Duygu Ataman. 2025. TUMLU: A unified and native language understanding benchmark for Turkic languages. In Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22816–22838, Vi- enna, Austria. Association for Computational Lin- guistics. Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 4948– 4961, Online. Association for Computational Lin- guistics. Yekyung Kim, Jenna Russell, Marzena Karpinska, and Mohit Iyyer. 2025. One ruler to measure them all: Benchmarking multilingual long-context language models. CoRR, abs/2503.01996. Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond en- glish: Towards a comprehensive evaluation of large language models in multilingual learning. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2023, Singapore, December 6-10, 2023, Findings of ACL, pages 13171–13189. Association for Computational Linguistics. Zhenyu Li, Kehai Chen, Yunfei Long, Xuefeng Bai, Yaoyin Zhang, Xuchen Wei, Juntao Li, and Min Zhang. 2025. Xifbench: Evaluating large language models on multilingual instruction following. CoRR, abs/2503.07539. Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James Validad Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno P. Kamp- man, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Jann Railey Montalan, Ryan Ignatius Hadiwijaya, Joanito Agili Lopo, William Nixon, Börje F. Karlsson, James Jaya, and 42 others. 2024. SEACrowd: A multilingual multimodal data hub and benchmark suite for South- east Asian languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 5155–5203, Miami, Florida, USA. Association for Computational Linguistics. Dan Nielsen. 2023. ScandEval: A benchmark for Scan- dinavian natural language processing. In Proceed- ings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 185–201, Tórshavn, Faroe Islands. University of Tartu Library. Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin- ter, Jan Haji ˇ c, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034–4043, Marseille, France. European Language Resources Association. Jessica Ojo, Odunayo Ogundepo, Akintunde Oladipo, Kelechi Ogueji, Jimmy Lin, Pontus Stenetorp, and David Ifeoluwa Adelani. 2025. Afrobench: How good are large language models on african languages? In Findings of the Association for Computational Lin- guistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, Findings of ACL, pages 19048–19095. As- sociation for Computational Linguistics. Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. Language model tok- enizers introduce unfairness between languages. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Pro- cessing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, and 3 others. 2025. Human- ity’s last exam. CoRR, abs/2501.14249. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R. Bowman. 2023. GPQA: A graduate-level google-proof q&a bench- mark. CoRR, abs/2311.12022. Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mo- hamed A. Haggag, Imanol Schlag, Marzieh Fadaee, Sara Hooker, Antoine Bosselut, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Viraat Aryabumi, Danylo Boiko, Michael Chang, and 40 others. 2025. INCLUDE: evaluating multilingual language under- standing with regional knowledge. In The Thirteenth International Conference on Learning Representa- tions, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Phillip Rust, Jonas Pfeiffer, Ivan Vuli ́ c, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tok- enizer? on the monolingual performance of multilin- gual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3118–3135, Online. Association for Computational Linguistics. David Schlangen, Sherzod Hakimov, Jonathan Jordan, and Philipp Sadler. 2025. A third paradigm for LLM evaluation: Dialogue game-based evaluation using clembench. CoRR, abs/2507.08491. Lütfi Kerem Senel, Benedikt Ebing, Konul Baghirova, Hinrich Schuetze, and Goran Glavaš. 2024. Karde ̧s- NLU: Transfer to low-resource languages with the help of a high-resource cousin – a benchmark and evaluation for Turkic languages. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1672–1688, St. Julian’s, Malta. Association for Computational Linguistics. Shivalika Singh, Angelika Romanou, Clémentine Four- rier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchi- sio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, André F. T. Martins, Leshem Choshen, Daphne Ippolito, and 4 others. 2025. Global MMLU: understanding and addressing cultural and linguistic biases in multilingual evalua- tion. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 18761–18799. Asso- ciation for Computational Linguistics. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI 2017, San Francisco, California, USA, February 4-9, 2017, pages 4444–4451. AAAI Press. Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores-Herr, Joachim Köhler, René Jäkel, and Mehdi Ali. 2024. Towards multilingual LLM evaluation for european languages. CoRR, abs/2410.08928. Ahmet Üstün, Viraat Aryabumi, Zheng Xin Yong, Wei- Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction finetuned open-access multilingual language model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 15894–15939. Association for Computational Linguistics. Menan Velayuthan and Kengatharaiyer Sarveswaran. 2025. Egalitarian language representation in lan- guage models: It all begins with tokenizers. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5987–5996, Abu Dhabi, UAE. Association for Computational Linguis- tics. Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, Qiqian Cang, Yichang Zhang, Fei Huang, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Polymath: Evaluating math- ematical reasoning in multilingual contexts. CoRR, abs/2504.18428. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McK- inney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. Browsecomp: A simple yet chal- lenging benchmark for browsing agents. CoRR, abs/2504.12516. Minghao Wu, Weixuan Wang, Sinuo Liu, Huifeng Yin, Xintong Wang, Yu Zhao, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2025. The bitter lesson learned from 2,000+ multilingual bench- marks. CoRR, abs/2504.15521. Chengxuan Xia, Qianye Wu, Hongbin Guan, Sixuan Tian, Yilun Hao, and Xiaoyu Wu. 2025. Evaluat- ing modern large language models on low-resource and morphologically rich languages:a cross-lingual benchmark across cantonese, japanese, and turkish. CoRR, abs/2511.10664. Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Jun- jue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, and 13 others. 2025. Mmlu- prox: A multilingual benchmark for advanced large language model evaluation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 1513–1532. Association for Computational Linguistics. Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Sys- tems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. A Word List Construction Four games require per-language word lists in ad- dition to translated prompts: Taboo, Wordle, Code- names and GuessWhat. Taboo. Target words and their taboo (forbid- den) words come from ConceptNet 5.7 (Speer et al., 2017), a multilingual commonsense knowl- edge graph. We iterate over all∼36M assertions and keep only/r/RelatedToedges with relation weight≥ 2.0, which filters for semantically close noun pairs with high crowdsourced confidence, and use/r/FormOfrelations to map surface forms to their base forms so that inflectional variants do not appear separately. For each candidate target we then take up to three related words by descend- ing weight, discarding targets with fewer than three qualifying neighbours, and retain up to 1,000 target– taboo pairs per language. A final pass removes potentially offensive or ambiguous entries using a manually curated blocklist. Wordle. Wordle needs two lists per language: the hidden target words and a larger set of allowed guesses. Targets come from Universal Dependen- cies v2.17 (Nivre et al., 2020), which provides gold part-of-speech tags and lemmas for over 120 languages. For each language we take all tokens taggedNOUN, keep the lemmas of exactly five char- acters, and rank them by corpus frequency; the 100 most frequent form the easy target list and the re- mainder the medium list. Allowed guesses come from Wikipedia dumps: after lowercasing and re- moving punctuation, all five-character tokens are retained, which covers inflected forms and proper nouns beyond the UD-derived targets. Codenames and GuessWhat. Both use closed lists whose playability depends on the words being common, unambiguous nouns, which automatic ex- traction cannot guarantee. We therefore translated the English lists with the pipeline of Section 3.2 (GPT-5.2 translating, Claude Sonnet 4.5 validating). Spot-checks on the manually verified languages (Section 3.3) confirmed that the translations keep the intended register and avoid culturally inappro- priate items. B Episodes per Game Each model plays 400 episodes per language (380 for Chinese), distributed across the 14 games as shown in Table 5. The counts follow the instance counts of the underlying clembench release rather than being balanced by us, so games with more pre-compiled instances contribute more episodes. The same instances are used for every model, so this affects the weight of a game in the aggregate but not comparisons across models. Comparisons across languages are affected in one case: drop- ping Wordle for Chinese removes a game on which every model scores poorly, which raises Chinese scores by two to six points relative to the other 29 languages. C Use of AI Assistants AI coding assistants, primarily Codex (GPT-5.2) and Claude Sonnet 4, were used to write some of the code that produces the figures and tables in this paper. All experimental design, data collection and interpretation of results are the authors’ own, GameEp.GameEp. TextMapWorld50TextMapWorld (Room)30 Clean Up45Private & Shared25 Image Game40Match-It20 Codenames35Taboo20 GuessWhat30Wordle20 Reference Game30Hot Air Balloon15 TextMapWorld (Graph)30Deal or No Deal10 Total: 400 (380 for Chinese, where Wordle is not played) Table 5: Episodes per game in a single model–language run. and every reported number was verified against the underlying result files. D Rank Correlation with External Benchmarks Figure 6 shows the ranking differences discussed in Section 5 as bump charts, with agreement quan- tified by Kendall’sτ:0.60against LMArena,0.40 against GPQA Diamond,0.20against HLE and 0.00against BrowseComp. Benchmark scores are taken from Table 6. Withn = 5these values are descriptive rather than significant. Table 6 reports benchmark scores for eight mod- els across four capability clusters: knowledge and reasoning, mathematics, coding, and agentic task completion. The remaining models in our study do not report results on these benchmarks and are omitted. Scores were collected from each model’s official release page or Hugging Face model card; cells marked “—” indicate that the model’s authors did not report a result for that benchmark. Note that MMLU-Pro (text knowledge) and MMMU-Pro (vi- sion understanding) are distinct benchmarks, as are MCPMark and MCP Atlas. Full model identifiers and source URLs for all nine models are listed in Table 7. E Additional Results This appendix reports the per-language detail be- hind Section 5: estimated API cost (Table 8), the two clemscore components %Played and Quality separately (Tables 9 and 10), and clemscore broken down by game and model within each capability cluster (Tables 11–15). Costs are computed from the token usage recorded in the interaction transcripts and each provider’s per-token pricing, over the full bench- mark run, which is the quantity a deployment ac- tually pays. Three of the nine models were run locally (Section 4.3), so their costs are OpenRouter clemscoreGPQA Diamond GPT-5.4 (91.8) (92.8) GPT-5.4Opus-4.8 (84.4) (93.6) Opus-4.8 Nemotron-Ultra-3 (68.2) (87.0) Nemotron-Ultra-3 GLM-5.2 (65.6) (91.2) GLM-5.2 DeepSeek-V4-Pro (64.8) (90.1) DeepSeek-V4-Pro (a) clemscore vs. GPQA Diamond (τ = 0.40) clemscoreHLE GPT-5.4 (91.8) (39.8) GPT-5.4 Opus-4.8 (84.4) (49.8) Opus-4.8 Nemotron-Ultra-3 (68.2) (26.7) Nemotron-Ultra-3 GLM-5.2 (65.6) (40.5) GLM-5.2 DeepSeek-V4-Pro (64.8) (37.7) DeepSeek-V4-Pro (b) clemscore vs. HLE (τ = 0.20) clemscoreBrowseComp GPT-5.4 (91.8) (82.7) GPT-5.4 Opus-4.8 (84.4) (83.7) Opus-4.8 Nemotron-Ultra-3 (68.2) (44.4) Nemotron-Ultra-3 GLM-5.2 (65.6)(79.3) GLM-5.2 DeepSeek-V4-Pro (64.8) (83.4) DeepSeek-V4-Pro (c) clemscore vs. BrowseComp (τ = 0.00) clemscoreLMArena GPT-5.4 (91.8)(rank 12) GPT-5.4 Opus-4.8 (84.4)(rank 13) Opus-4.8 Nemotron-Ultra-3 (68.2) (rank 91) Nemotron-Ultra-3 GLM-5.2 (65.6) (rank 26) GLM-5.2 DeepSeek-V4-Pro (64.8) (rank 38) DeepSeek-V4-Pro (d) clemscore vs. LMArena (τ = 0.60) Figure 6: Ranking differences between clemscore (English, left column of each panel) and external benchmarks (right column), with Kendall’s τ per comparison. list prices rather than amounts billed. The aggre- gate also does not separate models that complete episodes from models that abort early: a model that terminates a game prematurely consumes fewer to- kens and therefore appears cheaper, so cost should be read alongside %Played rather than on its own. F Token Fertility Calculation Token fertility is measured on natively written text rather than on game transcripts. For each language we collect Wikipedia articles about the countries where it is spoken (the same country lists used for LEF; for widely spoken languages, the ten largest by GDP), retrieved from that language’s own Wikipedia via the MediaWiki API and cropped to the first 12,000 characters per article so that an identical corpus fits into a single request for every model. Fertility is total tokens divided by total words, with words whitespace-delimited except in Chinese, where we count non-whitespace charac- ters. Token counts use each model’s own tokeniser: the Hugging FaceAutoTokenizerfor most open- weight models,tiktoken(o200k_base) for GPT- 5.4, and Mistral’smistral-common(Tekken) to- keniser for Mistral-Large-3, Nemotron-3-Ultra and Apertus, which all share it and therefore have iden- tical fertility values. Claude Opus 4.8 has no public tokeniser; its counts come from the Anthropic Mes- sages APIcount_tokensendpoint, after subtract- ing the constant per-message overhead measured with a one-character probe. Anthropic documents these as close estimates of billed usage and notes that Opus 4.7 and later use a tokeniser producing roughly 30% more tokens than earlier Claude mod- els for the same text, which is a different compar- ison from the gap to the median model reported in Section 5.2. That gap averages 50% across the EU-24 but varies widely by language, from+11% for Bulgarian to +92% for German. G Linguistic Economic Footprint: Data Figure 7 gives the per-model clemscore–LEF cor- relations discussed in Section 5.2. Table 16 gives the underlying data, one row per language. Coun- tries lists every country where the language is offi- cial or co-official, as CODE (population in millions, 2024 nominal GDP in bn USD from the World BankNY.GDP.MKTP.CDseries, share of the pop- ulation speaking it). Shares use Ethnologue to- BenchmarkQwen3.6NemotronDS-V4-ProGPT-5.4Opus-4.8GLM-5.2Gemma-4-26BMistral-L3 Knowledge & Reasoning GPQA ⋄ 86.087.090.192.893.691.282.343.9 HLE21.426.737.739.849.840.58.7— HLE + tools—37.448.252.157.954.717.2— MMLU-Pro85.2—87.5—82.6— MMMU-Pro75.386.8—81.2—73.8— MMLU—85.5 MMLU-Redux93.3— SuperGPQA64.7— Chinese-SimpleQA—84.4—75.0— SimpleQA—23.8 SimpleQA-Verified—57.9—38.1— BBH—87.5— Mathematics AIME 202692.7—95.799.288.3— HMMT Feb’2683.6—95.2—96.792.5— IMO-AB78.988.689.8—83.591.0— IMO-AB + tools—92.3— ARC-AGI-1—93.7— ARC-AGI-2—73.3— FrontierMath T1–3—47.643.8— Coding SWE-Verified73.471.980.6— SWE-Pro49.5—55.457.769.262.1— SWE-Multi67.267.776.2— LiveCode v680.489.0—88.8—77.1— LiveCodeBench—93.5—34.4 Codeforces—3206 ‡ 3168 ‡ — Agentic Terminal-B 2.051.5—67.9— Terminal-B 2.1—56.4—75.185.081.0— BrowseComp—44.483.482.783.779.3— Toolathlon26.9—51.854.659.948.2— MCP Atlas62.8—73.667.277.876.8— MCPMark37.0— OSWorld—75.0— GDPVal—46.7—83.0— Table 6: Benchmark scores for the eight models in this study with published results (scores in %; Code- forces reported as Elo rating). Qwen3.6 = Qwen3.6-35B-A3B; Nemotron = Nemotron-3-Ultra-550B-A55B; DS-V4-Pro = DeepSeek-V4-Pro; Opus-4.8 = Claude-Opus-4.8; Gemma-4-26B = Gemma-4-26b-a4b-it; Mistral- L3 = Mistral-Large-3-675B-Instruct-2512. GPQA ⋄ = GPQA Diamond; HLE = Humanity’s Last Exam; MMLU- Pro = Massive Multitask Language Understanding Professional (text); MMMU-Pro = Massive Multitask Multimodal Understanding Pro (vision); BBH = BIG-Bench Hard; IMO-AB = IMOAnswerBench; SWE-Multi = SWE-Bench Multilingual; LiveCode v6 = LiveCodeBench v6; LiveCodeBench = LiveCodeBench (no CoT). ‡ Elo rating rather than a percentage. Evaluation results for Apertus-1.5 are not available yet as of July 2026. All figures are vendor- self-reported under each vendor’s own evaluation protocol, so they indicate approximate standing rather than strictly comparable measurements. tal users (29th ed., 2024), which count first- and second-language speakers where the source reports both, capped at 1.0 where speaker and popula- tion estimates come from different sources. Spk sums speakers over the listed countries, and LEF is the speaker-weighted footprint P c GDP(c)× min(1, speakers/population) in bn USD. H Web Data Availability per Language Figure 8 shows the web-crawled text available for the 30 benchmark languages in two large multilin- gual pretraining corpora: HPLT v3 3 and FineWeb- 3 https://hplt-project.org/datasets/v3.0 2 4 (FineWeb for English, as FineWeb-2 covers only non-English languages). Counts are as reported by each dataset and are therefore approximate rather than strictly comparable across the two. The distri- bution is highly skewed: English alone accounts for roughly 35 trillion words, seven times more than Russian and nine times more than Chinese, while Serbian, Latvian, Estonian and Slovenian fall be- low 30 billion and Irish and Maltese, both official EU languages, below two billion each, four orders of magnitude less than English. Open-weight models are typically trained on 4 https://huggingface.co/datasets/HuggingFaceF W/fineweb-2 0.00.20.40.60.8 Spearman correlation: clemscore vs. LEF, across languages GPT-5.4 Claude-4.8 Qwen 3.6 DeepSeek-V4 Apertus Gemma 4 GLM-5.2 Mistral Large 3 Nemotron 3 -0.13 -0.02 0.22 0.28 0.30 0.32 0.36 0.61 0.78 Figure 7: Spearman correlation between each model’s clemscore and LEF across the 30 benchmark languages. Higher values indicate performance more skewed towards economically dominant languages; values near zero indicate performance largely independent of a language’s economic footprint. enruzhesdefritptplnltrcssvelrohuukfidaarbgskhrltsletlvsrgamt Language 10 1 10 2 10 3 10 4 10 5 Billions of words 34,807 4,991 3,663 920 872 806 475 428 343 248 192 162 148 138 138 133 107 94 91 82 65 55 42 38 29 27 23 13 1.5 1.3 HPLT v3 FineWeb-2 (FineWeb for en) Figure 8: Web data availability per language (billions of words, log scale) in HPLT v3 (blue) and FineWeb-2 (orange; FineWeb for English), sorted by combined total. Word counts are as reported by each dataset. such crawls, so their per-language competence largely inherits this distribution; matching English- level performance in the smallest languages evi- dently requires resources beyond these datasets. Correlating each model’s clemscore with crawled volume confirms this: the association is strong for the open-weight models (Pearsonr = 0.78with log 10 word count, Spearmanρ = 0.72,p < 0.001) and absent for the two commercial ones (ρ = 0.03 and0.06, both n.s.), which perform as well on the languages with the least crawled text as on those with the most. ModelSource$ in / out GPT-5.4 openai.com/index/gpt-5/2.50 / 15.00 Claude-Opus-4.8 anthropic.com/claude/opus5.00 / 25.00 GLM-5.2 hf.co/zai-org/GLM-5.21.00 / 4.00 Nemotron-3-Ultra hf.co/nvidia/NVIDIA-N emotron-3-Ultra-550B-A55 B-BF16 0.50 / 2.20 DeepSeek-V4-Pro hf.co/deepseek-ai/DeepS eek-V4-Pro 0.44 / 0.87 Mistral-Large-3 hf.co/mistralai/Mistral-L arge-3-675B-Instruct-2512 0.50 / 1.50 Qwen3.6-35B-A3B hf.co/Qwen/Qwen3.6-35B-A3B0.32 / 1.28 Gemma-4-26B hf.co/google/gemma-4-26B-A 4B-it 0.07 / 0.35 Apertus-1.5-8B hf.co/swiss-ai/Apertus-v1 .5-8B 0.07 / 0.15 Table 7: Model identifiers, sources and API pricing (USD per million input / output tokens);hf.coabbre- viateshuggingface.co. Benchmark scores were col- lected from each model’s official release page or Hug- ging Face model card, and pricing from OpenRouter in July 2026. Apertus-1.5 is not listed on OpenRouter, so we used the price of Qwen-3-8B, the closest model by size. LangGPT-5.4Opus-4.8GLM-5.2DS-V4NemotronGemma-4Qwen3.6Mistral-L3ApertusAvg ar31.738.19.343.163.190.471.531.770.419.97 bg32.336.37.763.263.370.702.591.762.4810.1 cs 31.038.77.903.494.260.692.152.183.2710.4 da28.336.58.613.804.530.622.792.270.599.78 de29.441.97.093.673.970.542.502.442.3110.4 el34.750.49.685.496.140.723.161.811.5512.6 en 24.730.65.222.673.310.521.932.000.417.93 es33.238.74.352.372.450.301.661.500.369.43 et31.140.59.114.324.040.632.511.551.6810.6 fi30.241.48.433.934.500.482.382.704.1010.9 fr29.737.37.093.504.040.581.961.740.939.66 ga53.882.517.45.106.801.005.161.956.1520.0 hr 31.437.58.393.784.230.641.712.100.9510.1 hu32.742.010.64.074.450.642.872.100.5011.1 it 28.436.78.123.113.730.612.451.722.279.67 lt53.685.314.23.595.440.584.521.750.7418.9 lv32.140.110.24.405.170.792.351.949.3811.8 mt33.144.211.85.038.320.674.981.850.5412.3 nl 28.436.86.523.713.890.573.092.914.7410.1 pl31.340.59.003.754.540.582.752.960.7110.7 pt35.738.65.412.222.420.301.611.230.319.76 ro34.138.48.394.304.360.673.012.520.6410.7 ru31.736.87.054.015.240.482.332.476.3110.7 sk31.839.87.523.855.100.613.092.510.3810.5 sl 30.538.08.683.384.210.662.501.811.2310.1 sr34.236.68.723.704.280.572.152.090.6010.3 sv27.735.97.193.494.120.652.852.562.169.62 tr35.042.412.64.464.810.683.601.083.5712.0 uk32.038.37.324.394.870.533.102.721.3710.5 zh26.126.21.780.591.290.482.332.921.677.04 Total980.31246.9255.4110.5131.118.081.662.962.32949.0 Avg32.741.68.513.684.370.602.722.102.0810.9 STD 6.1512.02.870.921.290.130.880.482.1415.1 Table 8: API cost in $USD to run the full benchmark per language (rows) and model (columns); columns ordered by mean clemscore. The Total row is the summed cost across all languages. Cell shading scales with log-cost; the cheapest model per language is in bold. LangGPT-5.4Opus-4.8GLM-5.2DS-V4NemotronGemma-4Qwen3.6Mistral-L3ApertusAvg ar89.083.589.678.772.871.970.967.840.473.8 bg99.293.289.981.979.085.187.569.648.781.6 cs 98.891.480.586.582.280.389.468.944.080.2 da97.493.390.690.988.288.885.473.741.583.3 de97.994.692.495.186.687.179.878.450.484.7 el98.785.589.385.077.977.583.164.650.179.1 en98.194.188.487.191.975.183.979.553.483.5 es99.885.886.485.778.080.577.274.448.879.6 et99.894.189.084.378.677.983.962.333.278.1 fi100.087.185.289.185.382.782.269.857.582.1 fr99.394.992.882.086.782.888.773.359.684.5 ga92.985.976.470.765.952.577.247.640.167.7 hr100.095.289.385.884.886.178.766.553.682.2 hu88.488.586.178.675.881.374.965.944.576.0 it99.892.289.990.984.988.682.662.941.481.5 lt86.585.384.769.578.172.186.853.052.874.3 lv99.390.787.588.180.180.280.660.245.379.1 mt99.887.482.387.072.371.083.336.628.972.1 nl98.192.190.887.389.786.382.982.147.984.1 pl98.191.887.687.387.490.284.276.657.384.5 pt 99.587.886.484.684.185.286.569.949.381.5 ro81.884.086.977.284.275.783.563.953.376.7 ru98.693.887.390.988.787.386.972.255.584.6 sk98.792.285.186.482.379.083.266.745.879.9 sl94.091.090.180.587.574.288.367.258.681.3 sr100.094.188.286.480.489.979.462.349.781.2 sv94.293.694.392.390.286.081.872.849.983.9 tr99.187.189.688.081.684.579.761.038.178.7 uk99.891.692.487.679.386.683.974.044.282.2 zh96.184.790.685.989.175.585.870.946.280.5 Avg96.790.288.085.082.480.782.767.147.780.1 STD 4.53.73.75.65.97.74.29.27.215.0 Table 9: Percentage of successfully played episodes (% Played) per language (rows) and model (columns), av- eraged over the 14 games; columns ordered by mean clemscore. Cells are shaded by octile of the distribution (darker = higher); the best model per language is in bold. LangGPT-5.4Opus-4.8GLM-5.2DS-V4NemotronGemma-4Qwen3.6Mistral-L3ApertusAvg ar88.982.867.266.264.769.448.958.029.163.9 bg93.286.576.774.766.271.460.059.936.169.4 cs 91.789.474.767.867.865.756.460.732.167.4 da92.489.075.671.566.366.655.560.827.267.2 de92.790.572.973.772.766.553.363.032.268.6 el92.788.378.374.265.861.566.153.738.168.7 en93.689.674.374.374.267.566.368.740.172.1 es85.481.671.767.572.165.757.560.734.966.3 et92.586.973.768.950.653.255.850.624.661.9 fi93.691.471.869.164.261.251.562.232.466.4 fr93.088.072.763.966.963.960.862.534.167.3 ga91.388.953.053.446.232.141.940.517.251.6 hr93.087.971.965.762.764.158.563.334.866.9 hu91.788.573.666.961.061.053.460.939.566.3 it 94.287.774.472.371.967.356.759.035.268.7 lt90.086.970.258.059.251.348.948.530.760.4 lv92.588.170.767.556.761.162.354.231.765.0 mt92.286.555.273.945.348.148.133.67.954.5 nl92.590.673.573.269.969.859.766.131.369.6 pl91.389.673.471.665.366.254.564.838.468.3 pt 86.381.065.063.369.165.557.059.938.565.1 ro90.687.071.265.164.156.854.756.333.464.4 ru90.685.471.468.866.466.461.964.841.468.6 sk91.589.069.868.755.267.858.357.828.265.1 sl91.686.466.468.561.049.455.365.332.164.0 sr90.689.968.867.960.467.053.859.339.466.3 sv93.386.169.171.067.966.056.270.541.069.0 tr91.886.170.772.263.049.751.954.424.862.7 uk93.286.972.563.663.665.559.668.140.268.1 zh95.087.378.773.076.764.871.971.539.173.1 Avg91.887.571.068.563.961.856.659.332.965.9 STD 2.02.45.54.87.48.45.98.07.217.5 Table 10: Quality Score of successfully played episodes per language (rows) and model (columns), averaged over the 14 games; columns ordered by mean clemscore. Cells are shaded by octile of the distribution (darker = higher); the best model per language is in bold. GameModel ar bg cs da de el en es et fi fr ga hr huitltlv mt nl pl pt ro ru sk sl sr svtr uk zh taboo GPT5898958279909592809898889595981009782909285989279909890889893 Opus 699295908895989276959285100951009894819898909088879095781009293 GLM7873907277859288607895308283928584587975887578606493789095100 DSV4638085607785727860859040857583859065727077507775809590909595 Nemo 60667670745298824267787626774703015636684636665686287738398 Gem46563277087658572306570565607340401074737055755459080808585 Qwen55687848606572674045682560620625028694253527263458554727888 Mist 707790647789806747796246737680807423828379607273658072909280 Aper353529323022424582024525454840291525425030433030484075265 codenames GPT628083808371867480868674777186838071807780807777717483748071 Opus50515848594755546357474055525238574757485453446050596438376 GLM 37373534312026373129290291414231411402631264620231131143746 DSV426313143402626372634291134373131433437292929403129293123269 Nemo2617172026614206171702061769020262311111492026171720 Gem4 3423402926040262331370292034232311433734263726233137342643 Qwen1423263431172029171717326232031266342617292623261111172620 Mist172929023923900002033015003134004326663233717 Aper 033003110063096090069361460090614 wordle GPT25948667366646474726664726274298174646273857617172626376- Opus03437615218564747336143532050744334860441059473743414232- GLM0522075151756192181901506562121003988- DSV4 01918334120393210272011106440282822281612142625293014- Nemo01041296191712558552005013618070927102- Gem4012016200013011700022025192625510100114260- Qwen000005120002000505020106050010- Mist0992621151410856114090828212251461493- Aper00000000000000000000000000000- Table 11: clemscore for the Lexical / world knowledge capability cluster: one row per game and model, one column per language. Shading uses this cluster’s own score distribution (equal-population octiles, darker = higher); “-” marks a missing run. Models:GPT= GPT-5.4;Opus= Claude Opus 4.8;GLM= GLM-5.2;DSV4= DeepSeek- V4-Pro;Nemo= Nemotron-3-Ultra;Gem4= Gemma-4-26B;Qwen= Qwen3.6-35B-A3B;Mist= Mistral Large 3; Aper = Apertus-v1.5-8B. GameModel arbgcsdadeelenesetfifr ga hrhuitltlvmtnlplptroruskslsrsvtrukzh reference GPT100100100100100100100100100100100310040100101001001001001001310010030100100100100100 Opus971009790971001001009797100979610010097100939710010097100100931008010097100 GLM839077676773938080808748805780808083867377537783608083939383 DSV483738373934780806053033703802057837383770576737063634363 Nemo739387738397905753576767731080806367837387737377807787737773 Gem4909387939797179393100100531009097737380979710009093010087909087 Qwen609387977080737083808360177983809393874710090907780100873787100 Mist738093806773909373778630837380206363808070777083907090878090 Aper41508285908993100857492509178958673407083961008771869297779397 imagegame GPT100991001001001001001001001001009810099100100100100100100100100100100100100100100100100 Opus7092658295624572989085871009857929992489892798070989510088812 GLM909487949693389491969546989577658652969287939891969597619318 DSV48382748092673871828281064746007164818178788361657467618733 Nemo 53734890947978867674928867521265842768086939336457591238533 Gem418894582846677566856212857861075296961816579546678690820 Qwen83939484959895849239775781728026888274619166869891839088925 Mist49539516847206349588803928004287758523645812363556512 Aper258815411913891451012107861662996613170144 Table 12: clemscore for the Grounded reference & spatial language capability cluster: one row per game and model, one column per language. Shading uses this cluster’s own score distribution (equal-population octiles, darker = higher); “-” marks a missing run. Models:GPT= GPT-5.4;Opus= Claude Opus 4.8;GLM= GLM- 5.2;DSV4= DeepSeek-V4-Pro;Nemo= Nemotron-3-Ultra;Gem4= Gemma-4-26B;Qwen= Qwen3.6-35B-A3B; Mist = Mistral Large 3; Aper = Apertus-v1.5-8B. GameModel arbgcsdadeelenesetfifrgahrhuitltlvmtnlplpt roruskslsrsvtrukzh matchit GPT100100100100100100100951009510010010010010095100100100959510010010010010010010010095 Opus1001001001001001001001009510010010010010010010010095100100951009510010010010010010055 GLM 809590100887390881008490679595758995508990908990799010086899575 DSV4859090100909090909085100858595100757595901009090859585100801005580 Nemo605570808585858050857070656075607585858595858055654595705095 Gem4 9090100100959595100808090659590908010090909095959595909510008580 Qwen809595909080659010090957095859080908595909570100100858590609580 Mist85797090803090908075757589849574900907590858580568810008080 Aper 60653525505565754560603555807060750507055707525607070608060 privshared GPT9397929597969796919492908991967474949787939476878575979588100 Opus959793979797979891669791909495777594988697948090849198968899 GLM758486718984868974327051738479555732936287827160516378745783 DSV4 522259587476746160746164506960334757824769725152625862606065 Nemo827866478443908533346449584575534531796673646533545862516478 Gem4555161354652484627104403442442428839306639383921393642382 Qwen 0613850407772673523523446649521725593157593038421446303183 Mist5147393144504133335436937442538170534250334027422526524244 Aper013061016272525287278114082181496262390161 guesswhat GPT 929491948881909289928784968991928992899197919897918796828990 Opus939392889697949792918289899494938793919386918293938791928992 GLM605244627771686446485651395983544444606662727353624747626371 DSV4 727856587784598066815667597969707763646970707864636450605960 Nemo675233637449727640596319495282334436796179677052624754515772 Gem4696057667264775836596414505661546713776674717252485861587660 Qwen646456645958397138505233665861573639436361594672575168376070 Mist 467743724256366233166020376172375210475662545032382863225937 Aper32270018214113046137196102300038491931036122000 Table 13: clemscore for the Discourse & grounding capability cluster: one row per game and model, one column per language. Shading uses this cluster’s own score distribution (equal-population octiles, darker = higher); “-” marks a missing run. Models:GPT= GPT-5.4;Opus= Claude Opus 4.8;GLM= GLM-5.2;DSV4= DeepSeek- V4-Pro;Nemo= Nemotron-3-Ultra;Gem4= Gemma-4-26B;Qwen= Qwen3.6-35B-A3B;Mist= Mistral Large 3; Aper = Apertus-v1.5-8B. GameModel arbgcsdadeelenesetfifrgahrhuitltlvmtnlplptroruskslsrsvtrukzh dond GPT 999798971001009910010010010010010098100889899100991001009098100891001009999 Opus595069709042804258685690506854496857977076607570407848497463 GLM676242857279606360865516848751765328526725475559667335776259 DSV461852968645884604723450565460485870475842597168516799785685 Nemo89757849302880833640490606977596820793075807729475359386276 Gem47975709559371009914546007680100565718708390559262477868598977 Qwen 10101926010481931243654919510393137404549287920642347 Mist382710374122472326343803531556581527141827281905737196640 Aper 001012004000602000700000007917000 balloon GPT100100100100100100100100100100100100100931001001001001001001007100100100100100100100100 Opus9199999999989991999998999998100999999999990369199989899989999 GLM277457554856486935564930475367705145446963373428703268603661 DSV41129293043344030413837381811303251463237187411235538571746 Nemo01846183517403726344314343439527442343385321239102232927 Gem4 125440170501729923004121610245122100022163616184040 Qwen 11582762122512546585062125383511241327166373310528404939 Mist03394947256951048600204618611113637484742029115902748 Aper154658213901629042831780371818024361814198227 cleanup GPT100100100100991001005100100991001001001001001001001001005100100100100100100100100100 Opus989899999999991999999989999999898989998199981009899100999994 GLM7081797679838417169774477677067756380672798082747681398266 DSV44041255648605915143406639435735506249400615152402854334545 Nemo203626363617003954532955525952482965570646054586064566073 Gem43141444538383404349451238434652232650530424139263643294427 Qwen 2245332639343514041443448394040433425503344046363838383456 Mist1511117193300096267113221618018179481851042 Aper51723131692301212153121116141061411016153181715121913 Table 14: clemscore for the Strategic / social reasoning capability cluster: one row per game and model, one column per language. Shading uses this cluster’s own score distribution (equal-population octiles, darker = higher); “-” marks a missing run. Models:GPT= GPT-5.4;Opus= Claude Opus 4.8;GLM= GLM-5.2;DSV4= DeepSeek- V4-Pro;Nemo= Nemotron-3-Ultra;Gem4= Gemma-4-26B;Qwen= Qwen3.6-35B-A3B;Mist= Mistral Large 3; Aper = Apertus-v1.5-8B. GameModel arbgcsdadeelenesetfifrgahrhuitltlvmtnlplptroruskslsrsvtrukzh tmw GPT919290919092918992919192909090909091899090929191908990939189 Opus 868689898486838486848862868587817876858886878485878690858384 GLM807670817979838479668357757884777868837882847855797779798083 DSV4717681817977797779788173778379748073808075828181797681788177 Nemo767575777667787651727954637479685640767877787878716778776378 Gem4 615660525461576343556144625562585855635259585862585661545658 Qwen 636164615863666161586562606063586056636362666463656064645664 Mist 76677769756875746478755357747575651846072765681746866747174 Aper1415111652014924851624187201711913719101516667 tmw-graph GPT 74807156648765868683768684382417885646287846976728420808255 Opus375064555227796353386916631749453121573056485155466549475672 GLM 70742364757555746964646178487267666367715875968717474717669 DSV4778083817780767876797772755773786678747873767481777079727777 Nemo716353637157726761677441644471595330666572666468696362735967 Gem4 585852515543575852555638564154545160555654575257576156545562 Qwen625150574345655459616662546256596056575861485748655767405565 Mist3307483504239565850290170005919220223750022454 Aper 000000000000000000000000000000 tmw-room GPT100100100100100100100100100100100100100100100100100100100100100100100100100100100100100100 Opus10010010010010010010010010010010010010010010010010010010097100100100100100100100100100100 GLM9710087100971009710097100100871001001001009783100100971001009010093100100100100 DSV4 93939790939093909310097808790939397100931009010097971009010010097100 Nemo1008797971001001001006793977073100100807753100971001009797938310010077100 Gem4778087879063878777838770936077777763739093708383779090778773 Qwen 777390807387837780678777774787737370778083778090838083838383 Mist938610094939396968396967393839710010067100909393100100921001009310090 Aper3320437332743407333333337407100433733172733274320304027 Table 15: clemscore for the Spatial reasoning & planning capability cluster: one row per game and model, one column per language. Shading uses this cluster’s own score distribution (equal-population octiles, darker = higher); “-” marks a missing run. Models:GPT= GPT-5.4;Opus= Claude Opus 4.8;GLM= GLM-5.2;DSV4= DeepSeek- V4-Pro;Nemo= Nemotron-3-Ultra;Gem4= Gemma-4-26B;Qwen= Qwen3.6-35B-A3B;Mist= Mistral Large 3; Aper = Apertus-v1.5-8B. Code LanguageCountries (pop. M, GDP bn USD, share %)Spk (M) LEF (bn USD) Germanic daDanishDK (6, 424, 94.0%)5.64399 deGermanDE (83.5, 4,686, 96.7%),AT (9.2, 535, 96.7%),CH (9, 937, 61.1%),LU (0.68, 93.3, 66.8%),BE (11.9, 671, 0.7%), LI (0.04, 7, 95.0%) 95.75,693 enEnglishUS (340, 28,751, 87.9%) ,GB (69.2, 3,686, 92.3%),CA (41.3, 2,244, 76.5%),AU (27.2, 1,757, 89.7%), IE (5.4, 609, 87.5%),NZ (5.3, 260, 93.3%),IN (1,451, 3,910, 19.2%),ZA (64, 401, 31.2%),NG (233, 252, 25.8%), SG (6, 547, 51.2%),PH (116, 462, 46.3%),KE (56.4, 120, 78.1%),GH (34.4, 82.3, 41.3%),TZ (68.6, 78.8, 9.8%), UG (50, 53.9, 58.0%),ZM (21.3, 25.3, 16.0%),ZW (16.6, 41.5, 31.1%),BW (2.5, 19.4, 42.8%), NA (3, 13.4, 14.8%), RW (14.3, 14.3, 13.9%), MT (0.57, 25, 72.1%), PK (251, 372, 6.8%) 96834,425 nlDutchNL (18, 1,215, 91.1%), BE (11.9, 671, 66.4%), SR (0.63, 4.4, 21.0%)24.41,554 svSwedishSE (10.6, 604, 93.2%), FI (5.6, 299, 5.2%)10.2578 Romance esSpanishES (48.8, 1,726, 96.2%),MX (131, 1,856, 97.2%),CO (52.9, 419, 97.9%),AR (45.7, 638, 100.0%), CL (19.8, 330, 100.0%),PE (34.2, 289, 97.2%),VE (28.4, 120, 100.0%),EC (18.1, 125, 97.4%), GT (18.4, 113, 89.8%),CU (11, 107, 100.0%),BO (12.4, 54.9, 82.1%),DO (11.4, 124, 85.8%), HN (10.8, 37.1, 95.3%),PY (6.9, 44.5, 94.9%),SV (6.3, 35.4, 100.0%),NI (6.9, 19.7, 97.2%), CR (5.1, 95.4, 100.0%), PA (4.5, 86.5, 97.4%), UY (3.4, 81, 100.0%), GQ (1.9, 12.8, 64.5%) 4656,126 frFrenchFR (68.6, 3,160, 96.3%),BE (11.9, 671, 82.1%),CH (9, 937, 66.2%),CA (41.3, 2,244, 27.3%), LU (0.68, 93.3, 87.6%),CD (109, 71, 48.3%),CI (31.9, 87.1, 35.9%),CM (29.1, 53.3, 40.9%), SN (18.5, 32.8, 27.4%),ML (24.5, 26.8, 19.4%),BF (23.5, 23.1, 21.5%),NE (27, 19.9, 14.0%), TD (20.3, 19.5, 11.8%),GN (14.8, 25, 27.0%),RW (14.3, 14.3, 5.2%),BJ (14.5, 21.5, 32.6%),BI (14, 3.1, 8.5%), TG (9.5, 10.7, 40.2%),CF (5.3, 2.8, 27.3%),GA (2.5, 20.9, 65.4%),CG (6.3, 15.7, 60.4%),MG (32, 17.4, 25.9%), KM (0.87, 1.4, 34.6%),DJ (1.2, 4.2, 50.1%),HT (11.8, 25.2, 4.2%),MU (1.2, 14.9, 76.7%),SC (0.12, 2.2, 48.5%), VU (0.33, 1.1, 25.1%) 2235,086 itItalianIT (59, 2,381, 94.1%), CH (9, 937, 7.5%), SM (0.034, 2.1, 100.0%), VA (0.001, 0.5, 100.0%)56.22,314 ptPortuguese BR (212, 2,186, 100.0%),PT (10.7, 313, 93.6%),AO (37.9, 101, 58.2%),MZ (34.6, 22.7, 42.4%), GW (2.2, 2.2, 18.0%),CV (0.52, 2.7, 70.7%),ST (0.24, 0.8, 95.0%),TL (1.4, 1.9, 35.7%),GQ (1.9, 12.8, 0.3%), MO (0.69, 49.5, 2.8%) 2642,553 roRomanianRO (19.1, 383, 79.6%), MD (2.4, 18.2, 100.0%)17.7323 Slavic bgBulgarianBG (6.4, 113, 89.4%)5.723101 csCzechCZ (10.9, 347, 86.6%)9.443301 hrCroatianHR (3.9, 93, 94.4%), BA (3.2, 29.6, 14.6%)4.14992.1 plPolishPL (36.6, 918, 100.0%)39.5918 ruRussianRU (144, 2,174, 93.4%), BY (9.1, 76, 77.2%), KZ (20.6, 292, 87.1%), KG (7.2, 17.5, 50.9%)1632,351 skSlovakSK (5.4, 141, 90.3%)4.877127 slSlovenianSI (2.1, 73, 98.1%)2.060671.6 srSerbianRS (6.6, 90.1, 91.7%), BA (3.2, 29.6, 33.6%), ME (0.62, 8.3, 51.5%), XK (1.6, 11.2, 6.2%)7.54597.5 ukUkrainianUA (37.9, 191, 58.6%)22.2112 Baltic ltLithuanian LT (2.9, 84.9, 92.9%)2.69578.9 lvLatvianLV (1.9, 43.7, 96.8%)1.8442.3 Uralic etEstonianEE (1.4, 43.1, 63.9%)0.89527.6 fiFinnishFI (5.6, 299, 94.6%)5.3283 huHungarianHU (9.6, 223, 99.3%)9.5376221 Celtic gaIrishIE (5.4, 609, 21.7%)1.171132 Hellenic elGreekGR (10.4, 256, 98.0%), CY (1.4, 37.6, 84.9%)11.4283 Semitic arArabicSA (35.3, 1,240, 81.9%) ,AE (11, 552, 31.7%),EG (116, 389, 66.3%),IQ (46, 280, 76.3%),DZ (46.8, 269, 75.6%), QA (2.9, 219, 12.1%),KW (4.9, 160, 76.1%),MA (38.1, 161, 71.7%),OM (5.3, 107, 42.5%), JO (11.6, 53.4, 74.0%),TN (12.3, 51.3, 79.6%),BH (1.6, 47.1, 51.1%),LY (7.4, 48.5, 84.5%), YE (40.6, 22, 52.5%),SY (24.7, 60, 81.8%),LB (5.8, 21, 82.4%),SD (50.4, 49.7, 53.8%),MR (5.2, 10.9, 48.3%), SO (19, 12, 2.6%), DJ (1.2, 4.2, 25.8%), KM (0.87, 1.4, 34.5%), PS (5.3, 13.7, 95.7%) 3212,444 mtMalteseMT (0.57, 25, 88.4%)0.50422.1 Turkic trTurkishTR (85.5, 1,359, 99.1%), CY (1.4, 37.6, 21.4%)851,355 Sino-Tibetan zhChineseCN (1,409, 18,744, 80.8%) ,TW (23.4, 775, 89.9%),SG (6, 547, 47.3%),MO (0.69, 49.5, 83.0%), HK (7.5, 407, 89.7%) 1,16916,501 Table 16: Countries, speaker estimates, and computed LEF per language. Country codes are ISO 3166-1 alpha-2 (XK = Kosovo (EU practical code; non-ISO), PS = Palestine). Speaker counts are total users from Ethnologue (29th ed., 2024), including L1+L2 where available.