Paper deep dive
Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs
Guy Mor-Lan, Omer Goldman, Matan Eyal, Adi Mayrav Gilady, Sivan Eiger, Idan Szpektor, Avinatan Hassidim, Yossi Matias, Reut Tsarfaty
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 4/26/2026, 11:07:46 PM
Summary
The paper introduces LocQA, a diagnostic benchmark designed to quantify implicit local and global biases in multilingual Large Language Models (LLMs). By using 2,156 locale-ambiguous questions across 12 languages, the researchers identify two structural biases: a 'Global Bias' where models default to US-centric norms even in non-English languages, and a 'Regional Bias' where models act as 'demographic probability engines,' prioritizing locales with larger populations. The study evaluates 32 models and finds that instruction tuning often exacerbates US-centric global bias while reducing regional bias, a phenomenon termed the 'Cultural Alignment Tax.'
Entities (7)
Relation Signals (4)
Instruction Tuning → exacerbates → Global Bias
confidence 100% · this global bias is exacerbated in models that underwent instruction tuning
LocQA → measures → Global Bias
confidence 100% · To this end, we present LocQA... to quantify models' inter- and intra-lingual biases
LocQA → measures → Regional Bias
confidence 100% · To this end, we present LocQA... to quantify models' inter- and intra-lingual biases
Instruction Tuning → causes → Cultural Alignment Tax
confidence 90% · instruction-tuned models exhibit lower Regional Bias but significantly higher US bias, suggesting that current alignment practices actively sacrifice cultural nuance
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multilingual large language models (LLMs) have minimized the fluency gap between languages. This advancement, however, exposes models to the risk of biased behavior, as knowledge and norms may propagate across languages. In this work, we aim to quantify models' inter- and intra-lingual biases, via their ability to answer locale-ambiguous questions. To this end, we present LocQA, a test set containing 2,156 questions in 12 languages, referring to various locale-dependent facts such as laws, dates, and measurements. The questions do not contain indications of the locales they relate to, other than the querying language itself. LLMs' responses to LocQA locale-ambiguous questions thus reveal models' implicit priors. We used LocQA to evaluate 32 models, and detected two types of structural biases. Inter-lingually, we show a global bias towards answers relevant to the US-locale, even when models are asked in languages other than English. Moreover, we discovered that this global bias is exacerbated in models that underwent instruction tuning, compared to their base counterparts. Intra-lingually, we show that when multiple locales are relevant for the same language, models act as demographic probability engines, prioritizing locales with larger populations. Taken together, insights from LocQA may help in shaping LLMs' desired local behavior, and in quantifying the impact of various training phases on different kinds of biases.
Tags
Links
- Source: https://arxiv.org/abs/2604.19292v1
- Canonical: https://arxiv.org/abs/2604.19292v1
Trouble viewing inline? Open PDF directly →
Full Text
70,041 characters extracted from source content.
Expand or collapse full text
2026-04-20 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs Guy Mor-Lan G* , Omer Goldman BC*† , Matan Eyal G , Adi Mayrav Gilady G , Sivan Eiger G , Idan Szpektor G , Avinatan Hassidim G , Yossi Matias G and Reut Tsarfaty BG† * Equal contribution, G Google Research, B Bar-Ilan University, C University of Cambridge, † Work done at Google Research Multilingual large language models (LLMs) have minimized the fluency gap between languages. This advancement, however, exposes models to the risk of biased behavior, as knowledge and norms may propagate across languages. In this work, we aim to quantify models’ inter- and intra-lingual biases, via their ability to answer locale-ambiguous questions. To this end, we present LocQA, a test set containing 2,156 questions in 12 languages, referring to various locale-dependent facts such as laws, dates, and measurements. The questions do not contain indications of the locales they relate to, other than the querying language itself. LLMs’ responses to LocQA locale-ambiguous questions thus reveal models’ implicit priors. We used LocQA to evaluate 32 models, and detected two types of structural biases. Inter-lingually, we show a global bias towards answers relevant to the US-locale, even when models are asked in languages other than English. Moreover, we discovered that this global bias is exacerbated in models that underwent instruction tuning, compared to their base counterparts. Intra-lingually, we show that when multiple locales are relevant for the same language, models act as demographic probability engines, prioritizing locales with larger populations. Taken together, insights from LocQA may help in shaping LLMs’ desired local behavior, and in quantifying the impact of various training phases on different kinds of biases. 1 1. Introduction When communicating in natural language, it is the rule rather than the exception that human speakers omit “obvious” information, giving rise to various ambiguities Grice (1991). How do LLMs cope with such ambiguities? In this pa- per we focus on a specific kind of ambiguity, namely, locale-ambiguity. Consider, for instance, the following seemingly straightforward question: “What is the emergency phone number?” or, “When does the tax year end?”. These English questions are inherently ambiguous, as different locales en- tail different answers. We conjecture that models’ answers to such ambiguous questions can reveal their implicit biases, as the ambiguity resolution exposes the model’s latent preferences, revealing which regional reality it treats as the standard, and which realities it might erase. Alternatively, a user may ask the same ques- tion in French, e.g., “Quand commence l’exercice 1 The data is available athttps://github.com/ google-research-datasets/locqa/. fiscal?”. In this case, we expect the model to shift its frame of reference away from the Anglosphere. This is tricky, as the prevailing assumption in mul- tilingual NLP is that querying a model in a specific target language acts as a proxy for context. So in theory, the choice of language should narrow the scope of ambiguity. However, a single lan- guage rarely isolates a single locale. In the case of French, for instance, it is the official language of 29 countries, spanning from France and Switzer- land to Haiti and the DRC. So, while the linguistic surface form is shared, the factual realities regard- ing laws, measurements, and infrastructure differ considerably across regions using the same lan- guage. In this work, we claim that current multilingual evaluations conflate two distinct capabilities of generative LLMs: (i) Linguistic Fluency, i.e., the ability to generate fluent and coherent text in a given target language, and (i) Localization, i.e., grounding the generation in the relevant reality of the speakers of that language in different lo- cales. While contemporary LLMs exhibit striking Corresponding author(s): guymorlan@google.com, omer.goldman, reut.tsarfaty@gmail.com © 2026 Google. All rights reserved arXiv:2604.19292v1 [cs.CL] 21 Apr 2026 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs fluency on an ever-increasing number of diverse languages, it remains unclear whether and to what extent they have truly learned to represent the diverse populations speaking those languages, or whether the generated content is a mere fluent, albeit biased, translation of Western norms. In order to isolate and investigate the localiza- tion aspect, we suggest analyzing how models vol- untarily resolve ambiguity in locale-ambiguous questions. Our investigation exposes two distinct axes along which models’ behavior may be bi- ased. First, we define a Global Bias as a measure of the extent to which a US-centric frame of ref- erence persists across linguistic boundaries (e.g., a model employing US norms even when queried in Indonesian). Second, we define a Regional Bias which examines the implicit prioritization of spe- cific locales within a shared language (e.g., when querying in Spanish, does the model default to Spain or Mexico?). To measure both kinds of biases, we present LocQA (Localized QA), a diagnostic benchmark designed to probe the implicit priors of LLMs. Unlike previous cultural benchmarks that test explicit knowledge (e.g., “What is the capital of Peru?”), LocQA utilizes semantically invariant, locale-ambiguous queries. By analyzing which regional reality the model defaults to when the context is underspecified, we map models’ tenden- cies, biases and implicit representation hierarchy. Our investigation of 32 models reveals that models do not resolve ambiguity based on ge- ographic fairness. Instead, we identify two struc- tural skews. First, we observe a persistent US- centric default: even when queried in non-English languages, models frequently mention US norms, instead of or in addition to locale-relevant re- sponses. Second, we detect a populist skew: mod- els function as “demographic probability engines,” where the likelihood of a locale being represented is strongly related to its population size, effec- tively erasing smaller nations that share a major language. Finally, we show empirical evidence for a Cultural Alignment Tax. That is, when contrast- ing instruction-tuned models with their base coun- terparts, we show that instruction-tuned models exhibit lower Regional Bias but significantly higher US bias, suggesting that current alignment prac- tices actively sacrifice cultural nuance, possibly in favor of a more generic, conceivably “safe”, homogeneity. In sum, the contributions of this paper are as follows: (i) we deliver the LocQA Benchmark, a validated diagnostic suite of 2,156 locale-specific answers to locale-ambiguous questions, across 12 languages and 49 regions, designed to isolate LLMs’ default priors; (i) we define a Dual-Metric Framework for quantifying implicit biases across two axes, including Global metrics to quantify US- centric bias, and Regional metrics for assessing geographic fairness; and (i) we deliver Empiri- cal Evidence of Alignment Bias across 32 LLMs. Ultimately, we argue that for LLMs to serve global audiences, geography should not be taken as a byproduct of language use. We call for a shift from multilingual modeling to multicultural and multi-regional modeling, where locale is treated as a distinct facet that must be accounted for to ensure factual adequacy for all users across the globe. 2. Challenges and Motivation Multilingual LLMs, like their monolingual coun- terparts, must be able to retrieve knowledge, a task that has been proven difficult in multilingual settings (Goldman et al., 2025; Lalai et al., 2025). However, multilingual LLMs are also tasked with the selection of the appropriate cultural frame to retrieve knowledge from. Thus, models should be tested not only for their capabilities (can the model answer X?) but also for their propensities (what does the model assume X is?). The gap between knowing a fact and selecting it is critical: a model may “know” the drinking age in Indone- sia, but if it defaults to US norms when asked in Indonesian, that knowledge is effectively erased. Existing cultural and regional benchmarks pri- marily evaluate a model’s capability to retrieve specific knowledge or values. For instance, bench- marks like INCLUDE (Romanou et al., 2024) and Global-MMLU (Singh et al., 2025) test objective accuracy on culturally sensitive knowledge, while others like BLEnD (Myung et al., 2024) and Glob- alOpinionsQA (Durmus et al., 2024) evaluate alignment with local everyday knowledge and 2 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs “What is the national emergency phone number?” EN USA: 911 UK: 999 Australia: 000 South Africa: 112 Canada: 911 Ireland: 112 “¿Cuál es la mon- eda nacional?” (What is the national currency?) ES Mexico: Peso (MXN) Spain: Euro (EUR) USA: Dollar (USD) Argentina: Peso (ARS) Colombia: Peso (COP) Peru: Sol (PEN) “Quand commence l’exercice fiscal ?” (When does the fiscal year start?) FR Canada: 1 er Avril France: 1 er Janvier Haiti: 1 er Octobre Switzerland: 1 er Janvier Belgium: 1 er Janvier Congo (DR): 1 er Janvier Question Language Localized Answers Figure 1|Schematic illustrating how identical queries in LocQA branch into distinct ground-truth answers depending on the target locale. Thus, language alone is insufficient for resolving factual ambiguity. subjective moral values. These benchmarks eval- uate capability and not implicit locale bias. More- over, they usually rely on explicit prompting, ask- ing models “What is the norm in Country X?” (Chiu et al., 2025; Yin et al., 2022) or provid- ing locale as context for reasoning (Rao et al., 2025). Even generative approaches as in Bha- tia and Shwartz (2023) rely on explicit cues to trigger diversity. By naming the target locale in the prompt, they act as an oracle, resolving the ambiguity for the model and masking its biases in information selection. This is the factor we seek to measure, addressing the “explicit-implicit localization gap” (Veselovsky et al., 2025). Unlike these works on explicit knowledge, sub- jective values, or natural phrasing (Hasan et al., 2025), we target the model’s unprompted default behavior, revealing the geographic alignment that contemporary explicit benchmarks systematically miss. We measure the extent to which model behavior is driven by epistemic inequity (Wang et al., 2025) and defaults to the dominant data distribution rather than the linguistically relevant locale. Measuring models’ implicit biases will further provide quantification for the discussion on the growing concern that this selection bias is exac- erbated by the very processes that improve LLMs’ multilingual capabilities. For example, Han et al. (2025) identify a “Transfer-Localization Trade- off,” where cross-lingual optimization leads to cultural erasure, and Gao et al. (2024) note that instruction tuning often results in “shallow” align- ment. Our work provides a diagnostic tool that will allow precise examination of the “taxes” im- posed by those improvements, and answer the question: might the pursuit of a universal, safe, assistant, force models to converge on a single, US-biased reality? 3. The LocQA Dataset This paper presents LocQA, a benchmark de- signed to answer the question: what is the default reality assumed by a model in locale-ambiguous questions? To construct LocQA, we first came up with about a dozen example questions suited for exploring models’ behavior under ambigu- ous conditions. The questions were relatively time-independent, related to specific facts with a well-defined answer, as well as easily translat- able, that is, without terms that require localiza- tion or whose translation is unclear in the tar- get languages. Most importantly, the answers to the example questions had to be locale-dependent, 3 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs where the expected answer may change accord- ing to the locale that the user has in mind and according to the language in which it is phrased. The example questions related to various topics: law, history, language, etc. The example questions were then given to qual- ified bilingual vendor annotators proficient in the target languages (see guidelines in Appendix A), for translation into the 12 languages covered by LocQA: English, Spanish, French, German, He- brew, Hindi, Indonesian, Italian, Japanese, Ko- rean, Portuguese, and Chinese. In total, we em- ployed 16 annotators. For each language, the annotators gave the answers to the questions as they relate to the countries associated with that language. We targeted countries with at least one million native speakers of each language for inclusion (see Table 2). Note that we did not re- quire all questions to have answers in all locales. Some questions, like who is the first president?, may not have answers in countries that never had presidents, so an N/A answer is valid. However, it must be clear whether the question has an answer or not. To ensure data quality, all translations and locale-specific answers were cross-validated by a second independent annotator. Following this, the authors conducted a general manual review to resolve discrepancies and correct any remain- ing errors. The final dataset consists of 2,156 locale-specific questions and answers. These cor- respond to 44 semantically parallel questions (Ap- pendix C) translated to 12 languages and an- swered for 49 locales. 4. Methodology 4.1. Metrics We define metrics to detect biases in the gen- erated answers compared to the locale-specific gold answers. Concretely, we define two met- rics. One for Global Bias퐵 푈푆 , i.e., the skew in the generated answers towards the US answer. This metric is calculated over the answers in all non-English languages taken together. The other metric, the Regional Bias퐵 푅 , aims to detect intra- lingual biases. It indicates the countries whose gold answers are over- or under-represented in the generated answers, taking into account one language at a time. Global Bias (퐵 푈푆 ). We quantify the extent to which models default to United States norms, for example, the extent to which the model answers George Washington to the question Who was the first president? or its translation. However, some US answers are not unique, so simple counting is insufficient. Consider the question in Indonesian Berapa usia legal untuk minum alkohol? (trans- lated to What is the legal drinking age?). A model that answers 21 may give the US-centric answer as a default but it may also give the correct an- swer for Indonesia, which happens to be identical. We term such identity of answers a collision. For that reason,퐵 푈푆 measures the difference between the frequency of the US answer in the model’s answers and the frequency of that value in the data. 2 We compute퐵 푈푆 separately for each lan- guage and report the macro-average across the 11 non-English languages, so that multi-locale languages (e.g., Spanish, with 20 locales) do not dominate the aggregate. Formally, for a language퐿with locale setC 퐿 , 퐵 푈푆 is the difference between the observed and the expected probabilities of getting the US an- swer: 퐵 푈푆 = 푃 obs (퐴 US )− 푃 exp (퐴 US )(1) where퐴 US is the value of the US answer. The observed푃 obs is calculated based on the model’s outputs and the expected푃 exp is based on the data: 퐵 푈푆 = 1 |Q| ∑︁ 푞 핀 퐴 US ∈ 푀(푞, 퐿) | z Observed − 1 |Q||C 퐿 | ∑︁ 푞,푐∈C 퐿 핀 퐴 US = 퐴(푞, 푐) | z Expected (2) Q is the set of questions in LocQA and푀(푞, 퐿) is the response of the model to question푞when asked in language퐿(one response per question 2 ‘N/A’ is treated as a valid answer. 4 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs per language);퐴(푞, 푐)is the gold answer for the same question in locale푐. The expected term is collision-aware: it counts, per question, the frac- tion of locales inC 퐿 whose gold answer coincides with the US answer. Note that the model’s re- sponse푀(푞, 퐿)may well include a list of multiple answers, so only inclusion of the US answer is needed. A positive퐵 푈푆 indicates the model prefers US norms beyond what would be expected from random chance overlap (e.g., shared drinking age or voltage standards). Regional Bias (퐵 푅 ). This metric quantifies the model’s preference for a specific locale. It com- pares푁 model (푐)—the number of times an answer valid to locale푐appears in model predictions, with푁 data (푐)—the number of times that locale 푐’s answer appears in the LocQA dataset for this question. Both counts are collision-aware, that is, counting each answer towards all the locales that it is valid for (e.g., “Peso” applying to multi- ple countries). This is done in order to account for shared norms and coincidental overlap in an- swers. Concretely, for each question, a gold answer held by푚locales contributes푚to the푁 data count of each of those locales.푁 model (푐)is incremented by 1 for every question whose model response contains a match for푐’s gold answer (recall that collisions arise when the same response matches the gold answers of multiple locales). Formally, for a language퐿with locale setC 퐿 , we define: 푃 obs (푐)= 푁 model (푐) Í 푘∈C 퐿 푁 model (푘) (3) 푃 exp (푐)= 푁 data (푐) Í 푘∈C 퐿 푁 data (푘) (4) The Regional Bias is then defined as the lift: 퐵 푅 (푐)= 푃 obs (푐) 푃 exp (푐) (5) 퐵 푅 (푐) >1 indicates over-representation (dom- inance), while퐵 푅 (푐) <1 indicates under- representation (erasure). To obtain a single bias score per model, we compute the mean devia- tion|퐵 푅 (푐) −1|within each language and then macro-average across languages with more than one locale. 4.2. Automatic Evaluation To evaluate model outputs at scale, we employ a 2-stage pipeline using Gemini-2.5-Flash, selected for its high instruction-following capability and low latency (prompts for this are given in Ap- pendix E). 3 Initially we assess (i) Ground Truth Align- ment. While answers within the same target lan- guage share identical string representations, the US reference answer often differs in language or formatting (e.g., ‘1 de Enero’ vs. ‘January 1st’). To properly detect such answer collisions, we employ a semantic matching prompt that identifies when a locale-specific answer is semantically equivalent to the US norm. Next, we turn to (i) Response Analysis as our primary evaluation method. We analyze model responses using an LLM-as-a-Judge to extract two key signals: Mentioned Answers, which identifies which of the locale-relevant gold answers are ex- plicitly provided by the model as valid options; and Framing Style, which detects whether the response uses the US as a conceptual anchor (e.g., “Unlike in the US...”), even when the US answer itself is not offered as a valid option. To verify the reliability of this automated pipeline, we manu- ally evaluated a random sample of 80 judgments, finding a 92% agreement rate between human annotations and the LLM judgments. For experiments testing models’ responses when explicitly specifying a desired locale, we use a verification prompt that checks if the model successfully retrieves the specific locale’s answer and if it hallucinates the US answer. 3 We verify the robustness of our pipeline by repeating all evaluations using GPT-5-mini, which yielded strong align- ment with our primary judge across Global Bias (푟=0.99), Regional Bias (푟= 0.95), and Framing (푟= 0.85). 5 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs 0.00.20.40.60.81.0 US Mention Frequency (P obs ) Falcon 3 10B Granite 3.0 8B OLMo 3 7B o1 GLM-4 9B GPT-5.1 GPT-4o Mini Gemma 3 4B Kimi K2 o3 GPT-4.1 Qwen 3 235B OLMo 3 32B Gemini 2.5 Flash GPT-4o DeepSeek R1 Gemma 3 12B Gemini 2.5 Pro Claude 4.5 Sonnet Gemini 3 Pro Gemma 3 27B Mistral Large Qwen 2.5 72B Mistral Small DeepSeek V3 Qwen 3 8B Grok 3 GPT-5 Mini Qwen 3 14B Claude 4.5 Opus Grok 4 P exp +0.01 +0.06 +0.09 +0.16 +0.16 +0.18 +0.19 +0.20 +0.20 +0.20 +0.20 +0.21 +0.22 +0.22 +0.22 +0.22 +0.25 +0.25 +0.26 +0.27 +0.29 +0.29 +0.30 +0.33 +0.33 +0.33 +0.34 +0.34 +0.39 +0.40 +0.42 B US (P obs − P exp ) Erasure Intrusion Selection Framing Valid Overlap Baseline (P exp ) Figure 2|Global Bias scores across models (on the right) show the difference between푃 obs (the sum of all bars) and푃 exp (the black line). Colors give a breakdown of US-centric answers into categories, defined in section 5.2. Intrusion (gratuitous inclusion) and selection (prioritizing US options) are most prevalent, occurring significantly more than complete erasure. 5. Experiments Setup. We evaluate a diverse suite of 32 models, both proprietary and open-weights models. To analyze the impact of alignment, we test both base and instruction-tuned variants for Gemma 3 (4B, 12B, 27B), Qwen (2.5-72B; 3-4B, 8B, 14B), GLM- 4 (9B), OLMo-3 (7B, 32B), Falcon 3 (10B) and IBM Granite 3 (8B). The suite also includes Qwen 3 (235B), DeepSeek (V3, R1), Mistral (Small, Large), and Kimi K2. Finally, we evaluate pro- prietary models including GPT (4o, 4.1, 5-mini, 5.1, o1, o3), Claude 4.5 (Sonnet, Opus), Gemini (2.5 Flash/Pro, 3 Pro), and Grok (3, 4). Models are evaluated in zero-shot format, with only the question as input, no instructions or examples. 4 5.1. Results Global Bias. Figure 2 summarizes the results in terms of the Global Bias퐵 푈푆 for all models over all 4 Following Kabir et al. (2025), who highlight the limi- tations of forced-choice in cultural evaluation, we employ open-ended generation rather than multiple-choice ques- tions to capture the model’s unprompted default. questions of LocQA. Almost all models demon- strate a clear US bias. The magnitude of that bias varies widely across models, from approximately 0 for Falcon 3 to 0.42 for the most biased Grok 4. The average퐵 푈푆 across all models is 0.24, reflect- ing the difference between the frequency of US answers in the data (26%) and the frequency of these answers in the models’ outputs (50%). 5 Regional Bias. Figure 3 displays Regional Bias scores in each locale for every evaluated model. Four languages with a single locale are omitted from this analysis. We see that despite varia- tions between models, a consistent set of locales tend to be over- or under-represented. The re- sults reveal a distinct ordering of locales, whereby large population centers (e.g., USA, Brazil) and Western countries (e.g., Spain, France) main- 5 In 18.2% of responses across our 0-shot instruct suite, the judge extracts no gold-answer candidate (neither a locale-valid answer nor the US value). We verify in Ap- pendix F that this does not drive our findings:퐵 푈푆 rankings are preserved when conditioned on at least one candidate be- ing extracted (휌=0.81), and퐵 푅 is mechanically unaffected by responses that yield no match. 6 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs 0.81.01.21.41.6 USA Spain Mexico Argentina Ecuador Peru Colombia Uruguay Venezuela Costa Rica Chile Panama Cuba Guatemala the Dominican Republic Paraguay Honduras Bolivia Nicaragua El Salvador Spanish 0.81.01.21.41.6 USA UK Canada Ireland Australia New Zealand South Africa English 0.70.80.91.01.11.21.31.41.5 France Canada Belgium Switzerland Mali Congo (DR) Haiti French 0.80.91.01.11.2 Germany Austria Switzerland German 0.70.80.91.01.11.21.3 Bias Lift (1.0 = Neutral) China (Chinese) Singapore (Chinese) Chinese 0.60.70.80.91.01.11.21.31.4 Bias Lift (1.0 = Neutral) Indonesia Malaysia Indonesian 0.70.80.91.01.11.21.3 Bias Lift (1.0 = Neutral) South Korea North Korea Korean 0.80.91.01.11.2 Bias Lift (1.0 = Neutral) Brazil Portugal Portuguese Figure 3|Distribution of Regional Bias scores across locales. The results reveal a structural inequality where Western nations and large population centers are consistently over-represented compared to peripheral locales. tain high scores while peripheral locales suf- fer systematic erasure. This allows us to iden- tify regional winners and losers. In Spanish, the USA, Spain, Mexico and Argentina emerge as over-represented, whereas Honduras, Bolivia, Nicaragua and El Salvador are under-represented. In English, the USA is over-represented, whereas South Africa, New Zealand and Australia suffer from under-representation. In French, France is over-represented, whereas Haiti, Congo and Mali emerge as under-represented. 5.2. Analysis Anatomy of US Bias. We categorize US-centric responses into five types: Erasure (replacing local reality with US norms), Intrusion (gratuitously in- serting US answers alongside correct local ones), Selection (preferring the US-aligned option in am- biguous contexts), Framing (explicitly contrast- ing the local answer against the US), and Valid Overlap (coincidental correctness, serving as a control). Table 1 provides examples of these cat- egories. Model outputs were classified into these categories according to the LLM judge’s identifi- cation of matched answers and explicit detection of framing. The breakdown is given in Figure 2. We see that the largest error mass is attributed to intrusion (17.5% of responses on average across models), where models provide the correct local answer but gratuitously insert US norms, treating them as a necessary context. However, a concern- ing 6.5% of responses exhibited erasure, where lo- cal reality is completely overwritten by US norms. In multi-locale ambiguity, selection accounts for an additional 8.7% on average, indicating a sys- tematic preference for the US-aligned option over other valid alternatives. Explicit framing remains rare (1.4% on average), suggesting that the US bias manifests as an implicit default rather than a conscious comparison. Population and Regional Bias. Having estab- lished that models exhibit biases across locales of the same language, we investigate the dominant factor driving this behavior. In Figure 4, we plot the Regional Bias퐵 푅 (푐), averaged across mod- els, against the log-scaled speaking-population of each locale (population data sources are listed in 7 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs CategoryLanguage QuestionModel Response (Translated Excerpt) ErasureFrenchQuand la constitution a-t-elle été signée? (When was the constitution signed?) “The United States Constitution was signed on September 17, 1787.” (Hallucinates US history onto French context) IntrusionKorean 제국가번호가어떻게되나요? (What is my international phone code?) “It depends on your location... For example: Korea: +82, US/Canada: +1, Japan: +81...” (Gratuitous inclusion of US code) SelectionSpanish¿Cuál es el número de teléfono de emer- gencia nacional? (What is the national emergency phone number?) “911.” (Valid for Mexico and US, but incorrect for Spain which uses 112) FramingSpanish¿Cuántas formas conjugadas tienen los verbos? (How many inflected forms do verbs have?) “The answer depends on the language... Here is the detail for Spanish, and as a comparison, for English.” (Explicit framing against US norm) Valid Overlap IndonesianBerapa usia legal untuk minum alkohol? (What is the legal drinking age?) “The legal age for consuming alcohol is 21.” (Coincidental match: Indonesia shares the US age of 21) Table 1 | Taxonomy of US Bias. Examples of the five error modes identified in our analysis. Appendix D). We tested linear, power-law, and logarithmic fits for the data. Our empirical anal- ysis reveals that a logarithmic function of popu- lation best models the data (푅 2 =0.41), signifi- cantly outperforming a hypothesis of linear pro- portionality to population size (푅 2 = 0.14). This demonstrates that the observed bias is a struc- tural property of model training: representation scales with the order of magnitude of the popula- tion rather than its raw count. This logarithmic relationship indicates the diminishing returns of population size. While the correlation is strong (푟=0.64, 푝 <0.001), the functional form im- poses a “soft ceiling” on demographic giants. For example, the estimated slope of 0.19 implies that a locale must grow its population by a factor of 10 just to gain 0.19 points in representation score. Consequently, this logarithmic compression sug- gests that models scale with population magni- tude rather than raw counts, effectively dampen- ing extreme demographic disparities and main- taining baseline visibility for the long tail. Domain-Wise Bias. To understand if specific topics disproportionately drive these biases, we categorized LocQA into five domains (see Ap- pendix G for the full data). We observe a striking divergence between Global and Regional bias trig- gers. Questions regarding State and Country (e.g., government, infrastructure) and Language exhibit the highest US-centric default (퐵 푈푆 ≈0.23–0.30) but relatively low regional distortion. Conversely, questions regarding Leisure and Culture (e.g., sports, retirement) successfully avoid the US de- fault (퐵 푈푆 =0.07) but exhibit the most extreme Regional Bias (|퐵 푅 −1|=0.69). This indicates that while culturally grounded topics escape a US-centric default, they heavily trigger the “de- mographic probability engine,” causing models to aggressively default to the most populous local nations instead of maintaining regional fairness. Instruction Tuning and Bias. Having seen the prevalence of regional and global US bias across different models, we examine the factors behind the biases. First, we investigate whether applying instruction tuning to multilingual models exacer- bates their biases. We extend the concept of the “Alignment Tax” (Lin et al., 2024; Ouyang et al., 2022), to detect whether improving the models’ ability to follow instructions in multilingual set- tings entails more significant bias. We examine this by comparing the global and regional biases of base open-weight models in the 4B-72B range against their instruction-tuned counterparts. In this comparison, we utilize a 3-shot prompting strategy for both model types. This ensures that the base models are not penalized for format- ting failures. The examples in the prompt are three simple, locale-neutral QA pairs (e.g., arith- metic) that only guide format adherence without priming regional biases (see Appendix E for the 8 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs 5M10M20M50M100M200M500M1B Speaker Population Size (Log Scale) 0.8 1.0 1.2 1.4 1.6 Mean Regional Bias Score ( B R ) China (Chinese) USA Indonesia Singapore (Chinese) Uruguay Mali USA Spain Malaysia North Korea Pearson r= 0.64 (p < 0.001) R 2 = 0.41, slope = 0.19 Language Chinese English French German Indonesian Korean Portuguese Spanish Neutrality (1.0) Figure 4|Average Regional Bias (퐵 푅 ) plotted against the log-scaled speaking population of each locale. The strong logarithmic fit (푅 2 =0.41) suggests that representation scales with the order of magnitude of the population rather than raw census counts. prompt). Figure 5 illustrates the impact of instruction tuning on both biases. The left panel plots the Global Bias score (퐵 푈푆 ). We observe a consis- tent “Alignment Tax”: across all model families, instruct-tuned models exhibit significantly higher US bias compared to their base counterparts. Fur- thermore, this bias scales with capability; larger models display consistently higher bias in both base and instruct regimes, suggesting that as models become more capable of retrieving cul- tural knowledge, they increasingly default to US- centric views. Conversely, the right panel displays the differ- ence in Regional Bias. Since in this case over- representation and under-representation are both unwanted, we calculated for each model the mean absolute deviation of Regional Bias scores from neutrality (|퐵 푅 −1|). Here, we observe the opposite trend: instruction fine-tuning tends to reduce regional distortion. Base models gener- ally exhibit higher Regional Bias (indicating the dominance of specific locales or erasure of others) and instruct models achieve lower scores. This suggests that alignment tuning “flattens” the rep- resentation across locales. We hypothesize that these opposing trends stem from the tendency of instruction-following training to motivate models to maximize helpful- ness by offering “diverse” and inclusive responses. To support this hypothesis, we measure the models’ answer multiplicity, defined as the av- erage number of distinct answers provided per question that are valid for some locale. Figure 6a confirms that instruction tuning systematically increases the average number of answers listed per question across all models. As shown in Fig- ure 6b and Figure 6c, the increase in multiplicity is strongly correlated with the rise in Global Bias (푟=0.95,푝 <0.001) and moderately correlated with the reduction in Regional Bias (푟=0.47, 푝= 0.146). This indicates that alignment transforms mod- els from local simulators, which commit to a single local answer, into global observers that strive for di- versity. By listing multiple valid options, instruct models dilute the dominance of any single locale, driving퐵 푅 towards neutrality. However, this di- versity is not neutral or evenly distributed, but rather itself selectively biased. The models learn 9 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs 4B7B8B9B10B12B14B27B32B72B Model Size (Parameters) - Log Scale 0.0 0.1 0.2 0.3 Global Bias Score ( B US ) Gemma 3 OLMo 3 Qwen 3 Granite 3.0 GLM 4 Falcon 3 Gemma 3 Qwen 3 Gemma 3 OLMo 3 Qwen 2.5 Base Trend (r= 0.50, p=0.114) Instruct Trend (r= 0.44, p=0.175) Base Model Instruct Model (Increase) Instruct Model (Decrease) (a) Impact on Global Bias (퐵 푈푆 ) 4B7B8B9B10B12B14B27B32B72B Model Size (Parameters) - Log Scale 0.14 0.16 0.18 0.20 0.22 0.24 Regional Bias Magnitude (| B R − 1.0|) Gemma 3 OLMo 3 Qwen 3 Granite 3.0 GLM 4 Falcon 3 Gemma 3 Qwen 3 Gemma 3 OLMo 3 Qwen 2.5 Base Trend (r= 0.16, p=0.639) Instruct Trend (r= −0.33, p=0.316) Base Model Instruct Model (Increase) Instruct Model (Decrease) (b) Impact on Regional Bias Magnitude (|퐵 푅 − 1.0|) Figure 5|Comparison of cultural biases in base versus instruction-tuned models. Panel (a) shows that instruction tuning consistently increases Global Bias (“Alignment Tax"), while panel (b) shows it tends to reduce Regional Bias magnitude, flattening representation across locales. to diversify their answers, but they consistently choose the US as the counterpoint or anchor for additional context. Thus, while alignment suc- cessfully reduces the erasure of local norms, it re-introduces bias through the very mechanism of diversity itself, framing the US as the universal reference even in non-English contexts. Undoing Ambiguity: Explicit Locale Prompt- ing. Finally, we investigate the nature of Global Bias when ambiguity is removed. We re-evaluated all models using an explicit prompt (e.g., “Locale: Mexico. What is the currency?”). A specialized judge (see Appendix E) verified if the model re- trieves the correct local answer or hallucinates the US one. This tests the “stickiness” of the bias: does the preference for US norms persist even when explicitly directed to another locale? Fig- ure 7 plots model accuracy against the US Halluci- nation Share, i.e., the percentage of errors where the model substitutes the correct answer with the US answer. We observe a moderate correlation (푟= 0.49, 푝=0.004) between model performance and US hallucinations on the full sample. More- over, among strong models (>70% accuracy), we see a stronger positive correlation (푟=0.64, 푝 < 0.001). That is, as models become more capa- ble and make fewer mistakes overall, the errors that do persist are increasingly US-centric. The fact that this correlation is most pronounced for high-accuracy models confirms that US bias is not merely a random fallback for missing knowledge. Instead, while general capabilities may eliminate random noise, the US prior is persistent. This chal- lenges the notion that “scaling is all you need” for multilingual LLMs, that is, that larger models will naturally converge on better multicultural repre- sentation. Rather than vanishing with increased capability, we see that US-centricity remains a sticky and proportionally larger residual failure mode. 6. Conclusion We introduce LocQA, a diagnostic benchmark to measure the implicit geopolitical priors of multi- lingual LLMs. By evaluating model responses to locale-ambiguous queries, we uncover two struc- tural biases in how LLMs handle ambiguity. First, models exhibit a Regional Bias, systematically fa- voring dominant locales in line with population size. Second, we identified a pervasive Global Bias toward US norms across different non-English lan- guages. Crucially, our experiments also revealed a Cultural Alignment Tax—while instruction tuning reduces the Regional Bias, it does so by increasing 10 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs 0.60.81.01.2 Base Model (Answers/Question) 0.6 0.8 1.0 1.2 Instruct Model (Answers/Question) (a) Shift in Multiplicity 0.00.10.20.30.40.5 Δ Answer Multiplicity (Instruct - Base) 0.00 0.05 0.10 0.15 0.20 0.25 Δ Global Bias Score (Instruct - Base) Gemma 3 OLMo 3 Qwen 3 Granite 3.0 GLM 4 Falcon 3 Gemma 3 Qwen 3 Gemma 3 OLMo 3 Qwen 2.5 r= 0.95 p< . 001 (b) Correlation w/ Global Bias 0.00.10.20.30.40.5 Δ Answer Multiplicity (Instruct - Base) −0.08 −0.06 −0.04 −0.02 0.00 0.02 Δ Regional Bias Magnitude (Instruct - Base) Gemma 3 OLMo 3 Qwen 3 Granite 3.0 GLM 4 Falcon 3 Gemma 3 Qwen 3 Gemma 3 OLMo 3 Qwen 2.5 r= 0.47 p= 0.146 (c) Correlation w/ Reg. Bias Magni- tude Figure 6|Analysis of answer multiplicity. Instruction-tuning systematically increases the number of valid answers per question (a). This shift strongly correlates with the rise in US-centric bias (b) and weakly correlated with a reduction in Regional Bias (c). 0.450.500.550.600.650.700.750.800.850.90 Model Accuracy 6% 8% 10% 12% 14% 16% 18% 20% Share of US Hallucinations within Errors Claude 4.5 Opus Claude 4.5 Sonnet DeepSeek R1 DeepSeek V3 Falcon 3 (10B) GLM-4 (9B) GPT-4.1 GPT-4o GPT-4o Mini GPT-5 Mini GPT-5.1 Gemini 2.5 Flash Gemini 2.5 Pro Gemini 3 Pro Gemma 3 (12B) Gemma 3 (27B) Gemma 3 (4B) Granite 3.0 (8B) Grok 3 Grok 4 Kimi K2 Mistral Large Mistral Small OLMo 3 (7B) OLMo 3.1 (32B) Qwen 2.5 (72B) Qwen 3 (14B) Qwen 3 (235B) Qwen 3 (4B) Qwen 3 (8B) o1 o3 High Performance Regime Global Trend: r = 0.49, p = 0.004 High Perf Trend (> 0.7): r = 0.64, p < 0.001 Figure 7|Model accuracy plotted against the share of errors where the model gives a US answer despite explicit locale prompting. The positive correlation indicates that US-centric bias persists in highly capable models. US-centricity. We trace this to a learned behavior of “answer multiplicity”: aligned models attempt to be helpful by providing more options, but sys- tematically select the US as a default reference for comparison. Finally, we showed that when highly capable models are wrong, they are more likely to override correct local facts with US norms. These findings challenge the prevailing assump- tion that linguistic fluency acts as a sufficient proxy for local and cultural grounding. To build truly global systems, the community must move beyond linguistic fluency and explicitly evaluate localization and cultural grounding, ensuring that alignment for safety does not come at the cost of cultural erasure. 11 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs Limitations While LocQA provides a rigorous framework for evaluating implicit localization, our study has sev- eral limitations. First, our dataset covers 12 lan- guages and 49 locales; while diverse, this rep- resents a fraction of global linguistic diversity. Extremely low-resource languages and dialects remain unprobed. Second, the ground-truth an- swers in LocQA (e.g., tax dates, voltage) may be subject to legislative and infrastructural change. Third, our automated evaluation relies on an LLM-as-a-Judge pipeline, which may not perfectly replicate the nuance of human evaluation. Finally, our analysis focuses on factual localization; we do not evaluate the model’s alignment with subjec- tive cultural values or moral norms, which repre- sents a distinct but equally important dimension of cultural capability. References M. Bhatia and V. Shwartz. GD-COMET: A geo- diverse commonsense inference model. In H. Bouamor, J. Pino, and K. Bali, editors, Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7993–8001, Singapore, Dec. 2023. Associa- tion for Computational Linguistics. doi: 10. 18653/v1/2023.emnlp-main.496. URLhttps: //aclanthology.org/2023.emnlp-main.496/. Y. Y. Chiu, L. Jiang, B. Y. Lin, C. Y. Park, S. S. Li, S. Ravi, M. Bhatia, M. Antoniak, Y. Tsvetkov, V. Shwartz, and Y. Choi. CulturalBench: A ro- bust, diverse and challenging benchmark for measuring LMs’ cultural knowledge through human-AI red-teaming. In W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, editors, Proceed- ings of the 63rd Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pages 25663–25701, Vienna, Aus- tria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1247. URLhttps: //aclanthology.org/2025.acl-long.1247/. E. Durmus, K. Nguyen, T. I. Liao, N. Schiefer, A. Askell, A. Bakhtin, C. Chen, Z. Hatfield- Dodds, D. Hernandez, N. Joseph, L. Lovitt, S. McCandlish, O. Sikder, A. Tamkin, J. Thamkul, J. Kaplan, J. Clark, and D. Ganguli.Towards measuring the representation of subjective global opin- ions in language models, 2024.URL https://arxiv.org/abs/2306.16388. C. Gao, H. Hu, P. Hu, J. Chen, J. Li, and S. Huang. Multilingual pretraining and instruction tuning improve cross-lingual knowledge alignment, but only shallowly. In K. Duh, H. Gomez, and S. Bethard, editors, Proceedings of the 2024 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies (Volume 1: Long Pa- pers), pages 6101–6117, Mexico City, Mexico, June 2024. Association for Computational Lin- guistics. doi: 10.18653/v1/2024.naacl-long. 339. URLhttps://aclanthology.org/2024. naacl-long.339/. O. Goldman, U. Shaham, D. Malkin, S. Eiger, A. Hassidim, Y. Matias, J. Maynez, A. M. Gi- lady, J. Riesa, S. Rijhwani, L. Rimell, I. Szpektor, R. Tsarfaty, and M. Eyal. Eclektic: a novel chal- lenge set for evaluation of cross-lingual knowl- edge transfer, 2025. URLhttps://arxiv.org/ abs/2502.21228. P. Grice. Studies in the Way of Words. Harvard University Press, 1991. H. Han, S. Agrawal, and E. Briakou. Rethink- ing cross-lingual alignment: Balancing transfer and cultural erasure in multilingual llms, 2025. URL https://arxiv.org/abs/2510.26024. M. A. Hasan, M. Hasanain, F. Ahmad, S. R. Laskar, S. Upadhyay, V. N. Sukhadia, M. Kutlu, S. A. Chowdhury, and F. Alam. NativQA: Multilin- gual culturally-aligned natural query for LLMs. In W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 14886–14909, Vienna, Austria, July 2025. As- sociation for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025. findings-acl.770. URLhttps://aclanthology. org/2025.findings-acl.770/. M. Kabir, A. Abrar, and S. Ananiadou. Break the checkbox: Challenging closed-style eval- 12 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs uations of cultural alignment in LLMs. In C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Lan- guage Processing, pages 24–51, Suzhou, China, Nov. 2025. Association for Computational Lin- guistics. ISBN 979-8-89176-332-6. doi: 10. 18653/v1/2025.emnlp-main.2. URLhttps: //aclanthology.org/2025.emnlp-main.2/. H. N. Lalai, R. S. Shah, J. Pei, S. Varma, Y.-C. Wang, and A. Emami. The world according to llms: How geographic origin influences llms’ entity deduction capabilities, 2025. URLhttps: //arxiv.org/abs/2508.05525. Y. Lin, H. Lin, W. Xiong, S. Diao, J. Liu, J. Zhang, R. Pan, H. Wang, W. Hu, H. Zhang, H. Dong, R. Pi, H. Zhao, N. Jiang, H. Ji, Y. Yao, and T. Zhang. Mitigating the alignment tax of RLHF. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 580–606, Miami, Florida, USA, Nov. 2024. Association for Com- putational Linguistics. doi: 10.18653/v1/2024. emnlp-main.35. URLhttps://aclanthology. org/2024.emnlp-main.35/. J. Myung, N. Lee, Y. Zhou, J. Jin, R. A. Putri, D. Antypas, H. Borkakoty, E. Kim, C. Perez- Almendros, A. A. Ayele, V. Gutiérrez-Basulto, Y. Ibáñez García, H. Lee, S. H. Muhammad, K. Park, A. S. Rzayev, N. White, S. M. Yimam, M. T. Pilehvar, N. Ousidhoum, J. Camacho- Collados, and A. Oh. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 78104–78146. Curran Associates, Inc., 2024. doi: 10.52202/079017-2483. URLhttps://proceedings.neurips. c/paper_files/paper/2024/file/ 8eb88844dafefa92a26aaec9f3acad93-Paper-Datasets_ and_Benchmarks_Track.pdf. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kel- ton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback, 2022. URLhttps://arxiv. org/abs/2203.02155. A. S. Rao, A. Yerukola, V. Shah, K. Reinecke, and M. Sap. NormAd: A framework for measur- ing the cultural adaptability of large language models. In L. Chiruzzo, A. Ritter, and L. Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2373–2403, Albuquerque, New Mexico, Apr. 2025. Association for Computational Lin- guistics. ISBN 979-8-89176-189-6. doi: 10. 18653/v1/2025.naacl-long.120. URLhttps: //aclanthology.org/2025.naacl-long.120/. A. Romanou, N. Foroutan, A. Sotnikova, Z. Chen, S. H. Nelaturu, S. Singh, R. Maheshwary, M. Altomare, M. A. Haggag, S. A, A. Amayue- las, A. H. Amirudin, V. Aryabumi, D. Boiko, M. Chang, J. Chim, G. Cohen, A. K. Dalmia, A. Diress, S. Duwal, D. Dzenhaliou, D. F. E. Flo- rez, F. Farestam, J. M. Imperial, S. B. Islam, P. Isotalo, M. Jabbarishiviari, B. F. Karlsson, E. Khalilov, C. Klamm, F. Koto, D. Krzemiński, G. A. de Melo, S. Montariol, Y. Nan, J. Niklaus, J. Novikova, J. S. O. Ceron, D. Paul, E. Ploeger, J. Purbey, S. Rajwal, S. S. Ravi, S. Rydell, R. Santhosh, D. Sharma, M. P. Skenduli, A. S. Moakhar, B. S. Moakhar, R. Tamir, A. K. Tarun, A. T. Wasi, T. O. Weerasinghe, S. Yilmaz, M. Zhang, I. Schlag, M. Fadaee, S. Hooker, and A. Bosselut. Include: Evaluating multi- lingual language understanding with regional knowledge, 2024. URLhttps://arxiv.org/ abs/2411.19799. S. Singh, A. Romanou, C. Fourrier, D. I. Ade- lani, J. G. Ngui, D. Vila-Suero, P. Limkonchoti- wat, K. Marchisio, W. Q. Leong, Y. Susanto, R. Ng, S. Longpre, S. Ruder, W.-Y. Ko, A. Bosse- lut, A. Oh, A. Martins, L. Choshen, D. Ip- polito, E. Ferrante, M. Fadaee, B. Ermis, and S. Hooker. Global MMLU: Understanding and addressing cultural and linguistic biases in mul- tilingual evaluation. In W. Che, J. Nabende, 13 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs E. Shutova, and M. T. Pilehvar, editors, Proceed- ings of the 63rd Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pages 18761–18799, Vienna, Aus- tria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.919. URLhttps: //aclanthology.org/2025.acl-long.919/. V. Veselovsky, B. Argin, B. Stroebl, C. Wendler, R. West, J. Evans, T. L. Griffiths, and A. Narayanan. Localized cultural knowledge is conserved and controllable in large language models, 2025. URLhttps://arxiv.org/abs/ 2504.10191. Z. Wang, Y. Zhang, D. Yoon, N. Vincent, F. Samir, and V. Shwartz. Wikigap: Promoting epis- temic equity by surfacing knowledge gaps be- tween english wikipedia and other language editions, 2025. URLhttps://arxiv.org/abs/ 2505.24195. D. Yin, H. Bansal, M. Monajatipoor, L. H. Li, and K.-W. Chang. GeoMLAMA: Geo-diverse com- monsense probing on multilingual pre-trained language models. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2039–2055, Abu Dhabi, United Arab Emirates, Dec. 2022. Asso- ciation for Computational Linguistics. doi: 10. 18653/v1/2022.emnlp-main.132. URLhttps: //aclanthology.org/2022.emnlp-main.132/. 14 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs A. Annotator Guidelines We provide below the text of the instructions given to the annotators. Background We are exploring how language models (LMs) handle questions with answers that change based on the language they are asked in. To do this, we are building a collection of such questions. We need help with three key tasks: 1.Identifying New Questions: Brainstorming additional questions that have different an- swers across languages. 2. Translating Questions: Providing accurate translations of these questions into various languages. 3.Listing Language-Specific Answers: Com- piling a list of possible answers for each ques- tion in its respective language. Phase 1: Expand the Existing List Your task in this phase is to propose new ques- tions to expand the current evaluation set. While the existing questions are categorized to aid in brainstorming, these categories are for internal use only and will not be part of the final dataset. Feel free to suggest questions that might fall into entirely new categories. Question Characteristics When suggesting new questions, ensure they meet the following criteria: •Language-Dependent Answers: The an- swer to each question must vary depending on the language in which the question is asked. •Locale-Variability (Implicit): Answers may also vary based on the locale where a lan- guage is spoken, but the locale should never be explicitly mentioned within the question itself. •Natural Language Model Phrasing: Ques- tions should be phrased naturally, as if di- rected to a Language Model (LM), not a hu- man. Avoid phrases like “your country,” “our country,” or similar terms that imply a hu- man respondent or specific location. •Multiple Answers & No Answers Allowed: It is perfectly acceptable for a question to have multiple possible answers per locale (e.g., the minimum wage varies across states in the US). Additionally, it is fine if a ques- tion does not have an answer in some of the target languages (e.g., “What is the gram- matical gender of ‘sun’?” has no answer in English). Phase 2: Strict Translation Guidelines Prioritize direct translation: Aim for a word-for- word or phrase-for-phrase translation even if it seems less common in the target language. • Example: When translating “parliament” into Hebrew, use “פרלמנט!” (parliament) instead of “כנסת!” (Knesset), which specifically refers to the Israeli parliament. Using “Knesset” would remove the intended ambiguity, mak- ing the answer specific to Israel rather than general across languages. Retain ambiguity: The core purpose of strict translation in this context is to keep the original ambiguity of a question. If a question is designed to have an answer that varies by language due to general terms, preserve that generality. Natural phrasing for retained ambiguity: If a strict translation results in an awkward or un- natural phrasing in the target language, but an alternative, more natural phrasing still retains the original ambiguity, opt for the more natural phrasing. •Example: If “independence day” is commonly referred to as “liberation day” in a specific country, and this “liberation day” phrasing is also generally used for independence days in other countries, then it is acceptable to use “liberation day.” Dialectal Variations When a phrase in the translation differs from dialect to dialect, apply the following hierarchy of preference: 15 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs 1.Prefer the more official phrase: If one dialectal variation is considered more offi- cial (e.g., in official documents, academic settings, or news broadcasts), that phrase should be preferred. 2. Most populous relevant country: If all di- alectal variations are considered equally offi- cial across different locales, prefer the dialect spoken in the most populous relevant coun- try. Phase 3: Finding the Answers The final phase involves identifying and listing all possible answers for each question. These an- swers should be provided in the original language of the question. Answer Specificity and Research •Language-Related Questions: For ques- tions in the “language related” category, there should be either one or no answer per language. •Locale-Dependent Questions: For ques- tions that have different answers based on locale, or potentially multiple answers per lo- cale, all possible answers must be listed. This often requires online research to account for various regional or national differences. Formatting and Brevity of Answers • Brevity is Key: Answers must be brief and concise. •Avoid Repetition: Do not repeat parts of the question in the answer. –Example: For the question “What is the shape of a stop sign?” (in English), the answer should be “octagon,” not “the shape of a stop sign is octagon.” •List Multiple Answers Directly: When mul- tiple answers exist, simply list them. Do not combine them into a single, long descriptive sentence. –Example: For “What is the legal drink- ing age?” (in English), the answer should be presented as: “18”, “19”, “21”. Avoid detailed explanations like “18 in most countries, 21 in the USA, and 19 in some Canadian provinces...” 16 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs B. LocQA Languages and Locales Language # Locales Included Spanish20Argentina, Bolivia, Chile, Colombia, Costa Rica, Cuba, Dominican Republic, Ecuador, El Salvador, Guatemala, Honduras, Mexico, Nicaragua, Panama, Paraguay, Peru, Spain, USA, Uruguay, Venezuela English7 Australia, Canada, Ireland, New Zealand, South Africa, UK, USA French7 Belgium, Canada, Congo (DR), France, Haiti, Mali, Switzerland German3 Austria, Germany, Switzerland Chinese2 China, Singapore Indonesian 2 Indonesia, Malaysia Korean2 North Korea, South Korea Portuguese 2 Brazil, Portugal Hebrew1 Israel Hindi1 India Italian1 Italy Japanese1 Japan Table 2|LocQA dataset composition. Languages are sorted by the number of distinct locales annotated. We cover 12 languages mapped to 49 distinct regions. 17 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs C. LocQA Question Templates CategoryQuestion Holiday and CalendarHow many public holidays are there? What are the days of the weekend? What is the first workday of the week? When do middle schoolers go back to school? When does the fiscal year start? LawCan I get a fine for jaywalking according to the law? How many paid vacation days are workers legally entitled to? Is it illegal to carry pepper spray? Leisure and CultureHas the national soccer team ever won the world cup? What is the national average number of children per family? What is the retirement age? State and CountryAs a default, is it allowed to turn right on red light? At what age do kids formally start learning to read in school? For how many years is education compulsory? How many seats are there in the parliament? How many terms can the prime minister serve? How many working hours are there in a week? How often are elections held? In which city are the government headquarters located? What are the national languages? What is my international telephone country code? What is my time zone? What is the legal drinking age? What is the mandatory duration of parental leave? What is the minimum age to apply for a driver’s license? What is the national currency? What is the national emergency phone number? What is the national life expectancy? What is the shape of a stop sign? What is the shortest national highway? What kind of electric plug is used in households? When was the constitution signed? When was the declaration of independence signed? Which is the national anthem? Who was the first minister of defense? LanguageCan I use the same word for a group of storks and a group of elephants? Can you speak a vernacular dialect at school? How many characters are there in the shortest word? How many inflected forms do most verbs have? What is the common format for dates? What is the first letter of the alphabet? What is the longest word in the dictionary? What is the most common greeting used over the phone? What is the standard word order in a declarative sentence? Table 3|Complete list of the 44 question templates in LocQA. Some questions in the Language category exhibit no intra-lingual variation (e.g., the alphabet is the same for all Spanish speakers). Consequently, they serve a dual purpose: acting as a control for Regional Bias metrics and providing a distinct signal for measuring global US-centric bias (e.g., detecting if a model answers a non-English query with English grammar rules). 18 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs D. Population Data Sources Since we lack a single authoritative source for lan- guage speaking populations across locales, we de- rive estimates from a hierarchy of diverse sources, prioritizing the most recent national census data, followed by reports from official linguistic obser- vatories. D.1. Methodology and Adjustments Definition of “Speaking Population”. We de- fine the speaking population as the total number of individuals possessing functional proficiency in the language, encompassing both Native Speak- ers (L1) and Second-Language Speakers (L2). Usage of Census Data. Census data was fil- tered to include the widest possible definition of proficiency: •Anglosphere (US/UK/Australia): We ag- gregated individuals who speak English “at home” (L1) with those who speak another language at home but reported speaking En- glish “Well” or “Very Well” (L2). •Multilingual Regions: For nations like India, where census data lags (last official census 2011), we applied the 2011 percentage of to- tal speakers (L1+L2) to the 2024 population estimate. Demographic Projections and Homogeneity. The assumption that “Total Population≈Speak- ing Population” was applied only to linguistically homogeneous nations where the dominant lan- guage is the sole medium of instruction and state administration (e.g., Japan, Brazil, Argentina, Italy). For linguistically diverse regions, we uti- lized specific proficiency rates rather than total population. D.2. Primary Data Sources Table 4 lists the primary authorities consulted for each language. Where available, 2024/2025 projections were used; otherwise, the most re- cent census figures (typically 2020–2023) were adjusted using World Bank annual population growth rates. Links to the source data are embed- ded in the authority names. 19 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs Language Primary Source Authority EnglishUSA: U.S. Census Bureau (ACS 2022) UK: Office for National Statistics (2021) Canada: Statistics Canada (Census 2021) Australia: Australian Bureau of Statistics (2021) South Africa: Statistics South Africa (Census 2022) SpanishGlobal: Instituto Cervantes (El español en el mundo 2023) FrenchGlobal: OIF (La langue française dans le monde 2022) ChineseChina: Ministry of Education of the PRC Singapore: Singapore Dept. of Statistics (Census 2020) Portuguese Brazil: IBGE (Censo Demográfico 2022) HindiIndia: Census of India (2011 Data adjusted for 2024) Indonesian Indonesia: Statistics Indonesia (BPS 2020) JapaneseJapan: Statistics Japan (e-Stat 2023) HebrewIsrael: Central Bureau of Statistics (2023) Table 4 | Primary sources for speaker population estimates. 20 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs E. Evaluation Prompts We utilized three distinct prompts in our eval- uation pipeline.Semantic Matchingwas used to pre-process the ground truth data.Implicit Evaluation was used for the main LocQA bench- mark to extract answers and framing style. Explicit Evaluationwas used for the control experiment where the target locale was specified in the query. E.1. Semantic Matching Prompt This prompt identifies collision between locale- specific answers and US answers (e.g., mapping "21" to "twenty-one"). You will be given a question, its US answer, and a list of other answers for↩→ other locales. Your task is to match and return all answers which are fully↩→ semantically the same as the US answer (and might differ due to language, or↩→ minor formatting conventions). ## INPUT Question: question US answer: us_answer Other answers: other_answers ## OUTPUT Respond with a valid JSON with the following fields, and nothing else:↩→ "reasoning": str. Free-text reasoning. "matches": [str]. List of answers that match the US answer.↩→ E.2. Implicit Evaluation Prompt This prompt evaluates the model’s response to ambiguous queries. It distinguishes between an- swers provided as correct options versus those mentioned merely for comparison. You are an expert linguist and cultural evaluator.↩→ You will evaluate an LLM's response to a (potentially) ambiguous question to↩→ which several locale answers are provided. If the model answer contains any thinking tags before the final answer, ignore↩→ the content inside them and evaluate the final answer.↩→ ### INPUT DATA 1. Question: "question" 2. Model Response: "model_response" 4. Locale Answers: locale_answers 3. US/English Answer: "us_answer" ### TASK Analyze the response and output a strictly formatted JSON object:↩→ 1. "reasoning": str - Free-text reasoning. 2. "mentioned_answers": list[str] - Identify EXACTLY which answers from the provided keys (US or Other) the↩→ model explicitly provides as correct. 21 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs - Return any and all matching answers, including semantic equivalents, as↩→ long as they are explicitly supported by the model answer.↩→ - Do NOT include answers that are mentioned solely for comparison, contrast,↩→ or to explicitly state they are *not* the answer (e.g., "Unlike in the↩→ US...", "It is not X"). - Return the raw string values from the provided answers.↩→ 3. "framing_style": "US_COMPARISON" | "NEUTRAL"↩→ - Does the model use the US as a conceptual anchor or reference point?↩→ - Return "US_COMPARISON" if the explanation relies on contrasting or↩→ comparing with the US (e.g., "Unlike in America...", "Similar to the US↩→ system..."), *even if the specific US answer is not stated*.↩→ - Return "NEUTRAL" if the answer is defined on its own terms.↩→ ### OUTPUT JSON "reasoning": "...", "mentioned_answers": ["..."], "framing_style": "US_COMPARISON" | "NEUTRAL" E.3. Explicit Evaluation Prompt This prompt is used when the user explicitly queries for a specific locale, checking for accu- racy and US hallucinations. You are an expert linguist and cultural evaluator.↩→ You will evaluate an LLM's response to a query where the user explicitly↩→ requested information for a specific locale. If the model answer contains any thinking tags before the final answer, ignore↩→ the content inside them and evaluate the final answer.↩→ ### INPUT DATA 1. Question: "question" 2. Model Response: "model_response" 3. Correct Locale Answer: "locale_answer" 4. US/English Answer (Reference): "us_answer"↩→ ### TASK Analyze the response and output a strictly formatted JSON object:↩→ 1. "reasoning": str - Free-text reasoning. 2. "match_locale": bool - Does the model provide the correct answer for the requested locale?↩→ - Ignore minor formatting differences. 3. "mentions_us_answer": bool - Does the response mention the US/English answer provided in the reference?↩→ - Return TRUE even if it is just mentioned for comparison (e.g., "It is X,↩→ unlike the US where it is Y"). ### OUTPUT JSON "reasoning": "...", "match_locale": true, "mentions_us_answer": false E.4. 3-Shot Generation Prompt This prompt was used to generate inference from models for the comparative analysis of base and instruction-tuned models. To ensure the evalu- ation isolates the model’s internal priors rather than in-context priming, we utilized a set of locale- invariant exemplars. The 3-shot examples consist of universal facts (arithmetic, planetary science, and physical geography) that remain true regard- less of the speaker’s region. This guides the model to adhere to the short-answer format without sig- naling a specific cultural frame of reference. Source Exemplars (English). The following three pairs served as the source for all transla- tions: Question: What is 2+2? Answer: 4 Question: What is the largest ocean on Earth? Answer: The Pacific Ocean Question: What planet is closest to the Sun? Answer: Mercury 22 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs Question: Target Question Answer: Formatting Implementation. For base models, the translated examples were concatenated into a single text string ending with the “Answer:” suf- fix to trigger completion. For instruction-tuned models, the examples were formatted as a conver- sation history (alternating User/Assistant turns) applied via the model’s specific chat template, with the target question serving as the final user message. F. Non-Response Analysis We say a model response yields an empty extrac- tion when the automatic judge identifies no can- didate answer in it—neither one of the locale- valid gold answers nor the US reference value. This can arise from a genuine refusal to answer, an off-topic response, or a hedged response that names no concrete value. Across our 0-shot in- struct evaluation suite, 18.2% of responses yield an empty extraction, ranging from 11.3% (En- glish) to 27.8% (Hebrew) across languages and from 5% to 51% across models. While producing no concrete answer may be a legitimate strategy under ambiguous queries, we verify in this sec- tion that this behavior does not drive either of our headline bias signals. Global Bias (퐵 푈푆 ).퐵 푈푆 is mechanically affected by empty extractions: when a model yields no candidate,푃 obs (퐴 푈푆 )decreases, so models that produce fewer concrete answers necessarily re- ceive lower anglocentrism scores. Consistent with this mechanism,퐵 푈푆 correlates strongly with the per-model rate at which the judge extracts at least one candidate (Pearson푟=0.80,푝 <0.001). To confirm that this effect does not drive our rank- ing of models, we re-compute퐵 푈푆 restricted to responses in which the judge extracts at least one candidate. The resulting model ranking is highly stable relative to the primary metric (Spearman 휌=0.81,푝 <0.001), confirming that anglocen- trism reflects answer selection among the concrete values a model produces, rather than differential rates of empty extraction. Regional Bias (퐵 푅 ).퐵 푅 is, by contrast, mechani- cally unaffected by empty extractions. A response that produces no candidate contributes zero to every푁 model (푐)and therefore also zero to the denominator Í 푘∈C 퐿 푁 model (푘). Both the numer- ator and the denominator of푃 obs (푐)are thus un- changed, so퐵 푅 (푐)depends only on the compo- sition of the answers the model does produce. Empirically,퐵 푅 also shows no cross-model corre- lation with the extraction rate (Pearson푟=0.19, 푝=0.3), confirming that no confound enters via between-model variation in response style. 23 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs G. Domain-Wise Bias Breakdown To further understand the mechanisms driving model bias, we broke down the LocQA dataset into five question domains. The analysis is re- stricted to 0-shot instruction-tuned models on non-English queries to capture the models’ de- fault localization behavior. As shown in Table 5 and illustrated in Figure 8, we observe an inverse relationship between the two bias axes. Domains that trigger high US- centricity (e.g., State and Country) tend to exhibit lower regional distortion, whereas domains that successfully avoid US norms (e.g., Leisure and Cul- ture) exhibit extreme regional inequality, heavily favoring populous nations. 24 Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs Global Bias (퐵 푈푆 )Regional Bias DomainObs. (푃 표푏푠 ) Exp. (푃 푒푥푝 ) 퐵 푈푆 Score Mag. (|퐵 푅 − 1|) State and Country0.4060.1030.3040.228 Language0.6880.4550.233 0.048 Holiday and Calendar0.6660.4720.1950.155 Leisure and Culture0.3820.3110.0710.687 Law0.5890.5380.052 0.179 Table 5|Domain-wise bias statistics. Global Bias (퐵 푈푆 ) is the difference between the observed and expected frequency of US-centric answers. Regional bias magnitude is the absolute deviation from neutral representation (1.0). 0.000.050.100.150.200.250.30 Global Bias Score (B US ) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Regional Bias Magnitude ( | B R −1| ) State and Country Language Holiday and Calendar Leisure and Culture Law Figure 8 | Scatter plot of the five question domains, plotting Global Bias (퐵 푈푆 ) against Regional Bias Magnitude (|퐵 푅 −1|). The inverse correlation highlights the divergence in bias triggers: domains avoiding US bias tend to suffer from high regional inequality. 25