Paper deep dive
P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs
Rafael Ferreira, Inês Vieira, Inês Calvo, James Furtado, Iago Paulo, Diogo Tavares, Diogo Glória-Silva, David Semedo, João Magalhães
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 6/20/2026, 8:50:51 AM
Summary
The paper introduces P3B3, an expert-curated multi-turn conversational benchmark designed to measure linguistic variety bias and controllability in Large Language Models (LLMs) regarding European Portuguese (pt-PT) and Brazilian Portuguese (pt-BR). The study reveals that most LLMs exhibit a strong inherent bias toward pt-BR, often due to training data imbalances. While larger and more recent models show better controllability when explicitly prompted, many models struggle to maintain pt-PT consistency across multiple dialogue turns, often drifting back to pt-BR patterns. The evaluation utilizes both BERT-based classifiers and an 'LLM-as-Judge' framework (using Gemini-3-Flash) to assess alignment.
Entities (7)
Relation Signals (4)
PeroVaz → evaluates → Portuguese Varieties
confidence 100% · We use PeroVaz (PV) and PtBrVId (VId), both fine-tuned BERT-based models
P3B3 → measuresbiasin → LLMs
confidence 100% · P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs
AMALIA → specializedfor → European Portuguese
confidence 100% · AMALIA, which is specifically pre-trained and fine-tuned for pt-PT
Gemini 3 Flash → usedasjudgefor → P3B3
confidence 100% · We use Gemini-3-Flash to assign a score from 0 (pt-BR) to 10 (pt-PT)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As Large Language Models (LLMs) become embedded in everyday communication, capturing regional linguistic variation is essential for reliable and equitable language use. In Portuguese, European (pt-PT) and Brazilian (pt-BR) varieties remain unevenly represented, with pt-BR dominating in data quantity, while LLM preference for Portuguese variants remains underexplored. To address this gap, we introduce P3B3, an expert-curated language variety agnostic benchmark of conversational prompts, along with an evaluation framework for measuring variety bias and controllability. Experiments on several models show that most LLMs exhibit a strong bias toward pt-BR, with variation in controllability across models. These results highlight the need for more balanced multilingual representation across language varieties.
Tags
Links
- Source: https://arxiv.org/abs/2606.16753v1
- Canonical: https://arxiv.org/abs/2606.16753v1
Trouble viewing inline? Open PDF directly →
Full Text
46,704 characters extracted from source content.
Expand or collapse full text
P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs Rafael Ferreira1,2, Inês Vieira1, Inês Calvo1, James Furtado1,2, Iago Paulo1,2, Diogo Tavares1,2, Diogo Glória-Silva1,2, David Semedo1,2, João Magalhães1,2 1NOVA University of Lisbon, Portugal, 2NOVA LINCS rah.ferreira, im.paulo, jh.furtado, dc.tavares, dmgc.silva@campus.fct.unl.pt im.vieira, i.calvo, df.semedo, jmag@fct.unl.pt Abstract As Large Language Models (LLMs) become embedded in everyday communication, capturing regional linguistic variation is essential for reliable and equitable language use. In Portuguese, European (pt-PT) and Brazilian (pt-BR) varieties remain unevenly represented, with pt-BR dominating in data quantity, while LLM preference for Portuguese variants remains underexplored. To address this gap, we introduce P3B3, an expert-curated language variety agnostic benchmark of conversational prompts, along with an evaluation framework for measuring variety bias and controllability. Experiments on several models show that most LLMs exhibit a strong bias toward pt-BR, with variation in controllability across models. These results highlight the need for more balanced multilingual representation across language varieties.111https://github.com/Amalia-LLM/p3b3-benchmark P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs Rafael Ferreira1,2, Inês Vieira1, Inês Calvo1, James Furtado1,2, Iago Paulo1,2, Diogo Tavares1,2, Diogo Glória-Silva1,2, David Semedo1,2, João Magalhães1,2 1NOVA University of Lisbon, Portugal, 2NOVA LINCS rah.ferreira, im.paulo, jh.furtado, dc.tavares, dmgc.silva@campus.fct.unl.pt im.vieira, i.calvo, df.semedo, jmag@fct.unl.pt 1 Introduction LLMs are increasingly used in everyday communication, supporting applications such as customer service, creative writing, and educational tools. As these systems become more widely deployed, their ability to capture regional linguistic variation is critical for reliable and equitable language use, as well as for linguistic and cultural diversity preservation Aepli et al. (2023); Chiu et al. (2025). Portuguese is a pluricentric language with over 250 million speakers worldwide. While it shares a common core, the divergence between European Portuguese (pt-PT) and Brazilian Portuguese (pt-BR) is substantial, spanning lexical choices (e.g., autocarro vs. ônibus), forms of address (e.g., tu vs. você), and syntax (e.g., clitic placement and gerund usage) Rio-Torto et al. (2022); Preda et al. (2024). Despite these differences, most training and evaluation frameworks treat Portuguese as a single, uniform language Team (2024, 2025c); AI (2026). Given the strong prevalence of Brazilian data in large-scale datasets Penedo et al. (2024); Common Crawl (2026), this often leads to a silent bias, where models default to pt-BR even when the context suggests pt-PT Simplício et al. (2026); Riley et al. (2023); González et al. (2026). This behavior can degrade user experience and contribute to linguistic uneven leveling, where regional variation is underrepresented in LLMs. Figure 1: P3B3 example and evaluation methodology. The prompts elicit different Portuguese variants, illustrated by the lexical variations of “table football” in the outputs. This imbalance raises two key questions: (i) to what extent do current LLMs exhibit a preference for one Portuguese variant, and (i) can models be steered toward a target variant. Facing these questions, our contributions are threefold. First, we introduce the pt-PT/pt-BR Bias Benchmark (P3B3), a benchmark of 74 expert-curated, variety-agnostic multi-turn dialogues designed to elicit Portuguese variety-specific linguistic markers in model responses. Second, we propose a human validated evaluation framework combining classifiers Sousa et al. (2025) and LLM-as-Judge methods Gu et al. (2024) to assess variant alignment and controllability. Finally, we conduct an extensive evaluation of diverse LLMs, revealing a bias toward pt-BR and variable success in steering generation toward pt-PT. Together, the contributions summarized in Figure 1 enable the systematic analysis of pt-PT and pt-BR language varieties in LLMs and support the development of more equitable multilingual systems. 2 Related Work LLM Performance across Linguistic Varieties. A growing body of work shows that LLMs exhibit systematic disparities across linguistic varieties. Pan et al. (2025) and Lee et al. (2025) show that model performance degrades on non-standard linguistic varieties across a wide range of English dialects and evaluation settings. Extending this line of work, Hassan et al. (2025) further find a dialectal preference bias, where models assign more negative sentiment to African American English than to equivalent Standard American English inputs. Portuguese Variant Classification. Given the differences across Portuguese varieties Rio-Torto et al. (2022), prior work has studied variant identification using classification-based methods Sousa et al. (2025); Preda et al. (2024). Furthermore, Freitag and Gois (2024) show that LLMs yield inconsistent and unreliable predictions in Brazilian Portuguese dialect profiling. Multilingual and Portuguese-Specific Models. Most multilingual LLMs treat Portuguese as a monolithic language, without explicit modeling of variations Team (2025c); AI (2026); Team (2024). In response, recent work introduces variant-specific LLMs, including Sabiá Abonizio et al. (2024); Laitz et al. (2026) and Bode Garcia et al. (2024) for pt-BR, and GlórIA Lopes et al. (2024) and AMALIA Simplício et al. (2026) for pt-PT, aiming to improve cultural and variety alignment. While prior work examines variety bias in English or frames Portuguese variation as a classification task, we study generation-time bias across Portuguese variants via the expert-designed P3B3, assessing inherent model preferences and their controllability. Figure 2: Examples of multi-turn variety-agnostic prompts from P3B3. Translation in Figure 4. 3 The P3B3 Benchmark The novelty of P3B3 lies in systematically evaluating whether LLMs exhibit preferences toward Portuguese variants and the controllability of such behavior. Specifically, P3B3 is designed around three guidelines: (i) Variety-agnostic, with no explicit references to variants in the input to prevent priming effects; (i) Variety Sensitivity, where prompts elicit natural lexical and grammatical differences between Portuguese varieties; and (i) Conversational, with multi-turn examples that accumulate dialogue history. These guidelines ensure that variation emerges from generation behavior rather than explicit cues or isolated inputs. 3.1 Expert-Based Dialogue Creation Following MT-Bench Zheng et al. (2023), P3B3 consists of 74 multi-turn dialogues (203 turns in total), each with 2 to 6 turns. As in MT-Bench, later user inputs build on prior dialogue context without requiring specific responses in earlier turns, enabling isolation of generative behavior while maintaining conversational coherence. This design enables cumulative context effects that may amplify biases not observable in single-turn prompts. To ensure quality and validity, P3B3 was manually constructed by two language experts holding Master’s degrees in linguistics-related fields and with expertise in Portuguese variants. Having this expert knowledge was essential to pinpoint subtle but systematic variation cues, particularly in vocabulary, orthography, and forms of address Rio-Torto et al. (2022), while remaining variant-agnostic. The benchmark focuses on diverse everyday domains (e.g., transportation, shopping, household products), which exhibit high variability across Portuguese varieties. These characteristics make P3B3 suitable for probing generation-time preferences rather than simple surface-level markers. Additional benchmark statistics are provided in Appendix A. Figure 2 shows representative examples from the dataset, which are variety-agnostic and elicit both lexical and syntactic variation in pt-PT and pt-BR. For instance, in Dialogue-1, Turn-1, the prompt elicits lexical variation such as ônibus (pt-BR) vs. autocarro (pt-PT), as well as grammatical differences in the response. 4 P3B3 Experimental Setup 4.1 Priming for Variant Bias and Controllability We evaluate inherent bias and controllability using P3B3 under three settings: (i) No-Prompt, where no explicit variety specification is provided, measuring implicit bias; (i) pt-BR Prompt; and (i) pt-PT Prompt, where in the first dialogue turn the model is explicitly instructed to respond in the corresponding variant. Following MT-Bench Zheng et al. (2023), models generate a response at each turn, with the full dialogue history provided as context. 4.2 Automatic Evaluation Methods We employ two evaluation methods. Encoder Classifiers. Model outputs are scored using pt-PT/pt-BR binary classifiers. We use PeroVaz (PV) Bastos (2024) and PtBrVId (VId) Sousa et al. (2025), both fine-tuned BERT-based models Devlin et al. (2019). The encoder outputs a probability score with lower values indicating pt-BR and higher values indicating pt-PT. Due to input length constraints, we apply a sliding window over each answer and average the scores. While efficient, these models may miss subtle or context-dependent variation, motivating the use of LLM-based evaluation. LLM-as-Judge. We use Gemini-3-Flash Team (2025b) to assign a score from 0 (pt-BR) to 10 (pt-PT), along with a chain-of-thought trace Wei et al. (2022). We use Gemini-3 due to its reliable instruction-following ability and strong understanding of Portuguese Chang and Arnett (2025)222In Appendix E, we show that Gemma-4 Gemma Team (2026) can be used as a strong open-source alternative.. To improve reliability, we vary three factors: prompt language (EN or PT), evaluation scope (Single-Turn, ST, or full dialogue history, D), and prompt detail (Simple, S, or Category-based, C). Category prompts include 11 expert-defined linguistic categories added to the prompt as examples, such as pronoun placement, preposition contractions, vocabulary, and spelling. Method Weighted κ (Quadratic) Pearson MAE ↓ Humans 0.69 0.75 2.07 PeroVaz (PV) 0.42 0.50 2.43 PtVId (VId) 0.57 0.60 2.66 Gemini-3 Flash - EN-S-ST 0.79 0.81 1.79 Gemini-3 Flash - PT-S-ST 0.79 0.82 1.72 Gemini-3 Flash - PT-S-H 0.80 0.83 1.72 Gemini-3 Flash - PT-C-ST 0.81 0.83 1.58 Gemini-3 Flash - PT-C-D 0.81 0.82 1.60 Table 1: Agreement between human annotations and automatic metrics. EN/PT = language; S = Simple; C = Category; ST = Single-Turn; D = Dialogue history. 4.2.1 Validating Automatic Metrics We validate the metrics using 200 responses annotated by two linguistic experts on a 0 (pt-BR) to 10 (pt-PT) scale, with -1 for invalid outputs. Samples are randomly drawn equally from all settings (No-Prompt, pt-BR, pt-PT) across 12 models. Considering valid responses (88.5%), Table 1 reports agreement with human judgments. Human annotators show strong agreement, indicating task reliability. Classifiers achieve moderate alignment with humans, with VId performing best. The LLM-as-a-Judge exhibits strong alignment, performing best under the PT-C-ST setup, which is used in subsequent experiments. Overall, classifiers provide a low-cost complementary signal, while LLM-based evaluation yields more aligned and interpretable results. 4.3 Models and Implementation Details We evaluate a range of multilingual instruction-tuned LLMs of varying sizes (list in Appendix B). Models are selected based on multilingual and Portuguese performance, public availability, and architectural diversity. All models are evaluated in BF16 using greedy decoding. Additionally, we include closed-source models: Sabiá-4 Laitz et al. (2026), specialized for Brazilian Portuguese, and Gemini-3-Flash Team (2025b) as a frontier model. 5 Results and Discussion No-Prompt pt-BR (↓ ) pt-PT (↑ ) Model PV VId LLM PV VId LLM PV VId LLM Fully Open Salamandra-7B [rgb]0.52605, 0.76975, 0.758447.9 [rgb]0.5336, 0.732, 0.818832.8 [rgb]0.53275, 0.73625, 0.81234.5 [rgb]0.92195, 0.9, 0.5544.9 [rgb]0.79655, 0.9, 0.5522.1 [rgb]0.78885, 0.9, 0.5520.7 [rgb]0.939, 0.9, 0.5552.0 [rgb]0.95, 0.8622, 0.5543.7 [rgb]0.9346, 0.9, 0.5552.8 OLMo-3-7B [rgb]0.5307, 0.7465, 0.795638.6 [rgb]0.5315, 0.7425, 0.80237.0 [rgb]0.54115, 0.69425, 0.879217.7 [rgb]0.88125, 0.9, 0.5537.5 [rgb]0.8367, 0.9, 0.5529.4 [rgb]0.7619, 0.9, 0.5515.8 [rgb]0.95, 0.8748, 0.5545.8 [rgb]0.95, 0.846, 0.5541.0 [rgb]0.95, 0.7398, 0.5523.3 OLMo-3.1-32B [rgb]0.53055, 0.74725, 0.794438.9 [rgb]0.5389, 0.7055, 0.861222.2 [rgb]0.5417, 0.6915, 0.883616.6 [rgb]0.85595, 0.9, 0.5532.9 [rgb]0.719, 0.9, 0.558.0 [rgb]0.73275, 0.9, 0.5510.5 [rgb]0.90105, 0.9, 0.5558.9 [rgb]0.93955, 0.9, 0.5551.9 [rgb]0.95, 0.8568, 0.5542.8 EuroLLM-22B [rgb]0.53095, 0.74525, 0.797638.1 [rgb]0.53735, 0.71325, 0.848825.3 [rgb]0.53875, 0.70625, 0.8622.5 [rgb]0.8708, 0.9, 0.5535.6 [rgb]0.73385, 0.9, 0.5510.7 [rgb]0.7278, 0.9, 0.559.6 [rgb]0.83835, 0.9, 0.5570.3 [rgb]0.74265, 0.9, 0.5587.7 [rgb]0.73825, 0.9, 0.5588.5 Apertus-8B [rgb]0.5284, 0.758, 0.777243.2 [rgb]0.5366, 0.717, 0.842826.8 [rgb]0.53985, 0.70075, 0.868820.3 [rgb]0.90545, 0.9, 0.5541.9 [rgb]0.77895, 0.9, 0.5518.9 [rgb]0.75585, 0.9, 0.5514.7 [rgb]0.8972, 0.9, 0.5559.6 [rgb]0.87245, 0.9, 0.5564.1 [rgb]0.8895, 0.9, 0.5561.0 Apertus-70B [rgb]0.5295, 0.7525, 0.78641.0 [rgb]0.53715, 0.71425, 0.847225.7 [rgb]0.54195, 0.69025, 0.885616.1 [rgb]0.87575, 0.9, 0.5536.5 [rgb]0.7498, 0.9, 0.5513.6 [rgb]0.7223, 0.9, 0.558.6 [rgb]0.8466, 0.9, 0.5568.8 [rgb]0.75365, 0.9, 0.5585.7 [rgb]0.75035, 0.9, 0.5586.3 AMALIA-9B [rgb]0.51655, 0.81725, 0.682466.9 [rgb]0.5072, 0.864, 0.607685.6 [rgb]0.5044, 0.878, 0.585291.2 [rgb]0.92745, 0.9, 0.5545.9 [rgb]0.85925, 0.9, 0.5533.5 [rgb]0.8664, 0.9, 0.5534.8 [rgb]0.83285, 0.9, 0.5571.3 [rgb]0.7201, 0.9, 0.5591.8 [rgb]0.69755, 0.9, 0.5595.9 Open Weight Llama-3.1-8B [rgb]0.5295, 0.7525, 0.78641.0 [rgb]0.53795, 0.71025, 0.853624.1 [rgb]0.5437, 0.6815, 0.899612.6 [rgb]0.8873, 0.9, 0.5538.6 [rgb]0.7685, 0.9, 0.5517.0 [rgb]0.7333, 0.9, 0.5510.6 [rgb]0.91095, 0.9, 0.5557.1 [rgb]0.84385, 0.9, 0.5569.3 [rgb]0.95, 0.8862, 0.5547.7 Llama-3.3-70B [rgb]0.52955, 0.75225, 0.786440.9 [rgb]0.5372, 0.714, 0.847625.6 [rgb]0.54235, 0.68825, 0.888815.3 [rgb]0.8653, 0.9, 0.5534.6 [rgb]0.7487, 0.9, 0.5513.4 [rgb]0.71185, 0.9, 0.556.7 [rgb]0.8862, 0.9, 0.5561.6 [rgb]0.77675, 0.9, 0.5581.5 [rgb]0.8521, 0.9, 0.5567.8 Gervásio-8B-ptpt [rgb]0.53015, 0.74925, 0.791239.7 [rgb]0.53715, 0.71425, 0.847225.7 [rgb]0.54365, 0.68175, 0.899212.7 [rgb]0.8873, 0.9, 0.5538.6 [rgb]0.7729, 0.9, 0.5517.8 [rgb]0.73605, 0.9, 0.5511.1 [rgb]0.92085, 0.9, 0.5555.3 [rgb]0.86475, 0.9, 0.5565.5 [rgb]0.95, 0.8796, 0.5546.6 Gervásio-70B-ptpt [rgb]0.5263, 0.7685, 0.760447.4 [rgb]0.5338, 0.731, 0.820432.4 [rgb]0.5386, 0.707, 0.858822.8 [rgb]0.8774, 0.9, 0.5536.8 [rgb]0.76685, 0.9, 0.5516.7 [rgb]0.7344, 0.9, 0.5510.8 [rgb]0.8697, 0.9, 0.5564.6 [rgb]0.7586, 0.9, 0.5584.8 [rgb]0.8114, 0.9, 0.5575.2 Ministral-3-14B [rgb]0.5282, 0.759, 0.775643.6 [rgb]0.53835, 0.70825, 0.856823.3 [rgb]0.5411, 0.6945, 0.878817.8 [rgb]0.85375, 0.9, 0.5532.5 [rgb]0.71295, 0.9, 0.556.9 [rgb]0.6981, 0.9, 0.554.2 [rgb]0.8136, 0.9, 0.5574.8 [rgb]0.7014, 0.9, 0.5595.2 [rgb]0.72615, 0.9, 0.5590.7 Qwen3-8B [rgb]0.529, 0.755, 0.78242.0 [rgb]0.54215, 0.68925, 0.887215.7 [rgb]0.5448, 0.676, 0.908410.4 [rgb]0.8752, 0.9, 0.5536.4 [rgb]0.72505, 0.9, 0.559.1 [rgb]0.71845, 0.9, 0.557.9 [rgb]0.95, 0.8886, 0.5548.1 [rgb]0.95, 0.846, 0.5541.0 [rgb]0.95, 0.7962, 0.5532.7 Qwen3.5-9B [rgb]0.52855, 0.75725, 0.778442.9 [rgb]0.529, 0.755, 0.78242.0 [rgb]0.53245, 0.73775, 0.809635.1 [rgb]0.83285, 0.9, 0.5528.7 [rgb]0.7454, 0.9, 0.5512.8 [rgb]0.7421, 0.9, 0.5512.2 [rgb]0.807, 0.9, 0.5576.0 [rgb]0.7102, 0.9, 0.5593.6 [rgb]0.75145, 0.9, 0.5586.1 Qwen3.5-27B [rgb]0.52625, 0.76875, 0.7647.5 [rgb]0.5263, 0.7685, 0.760447.4 [rgb]0.52775, 0.76125, 0.77244.5 [rgb]0.83175, 0.9, 0.5528.5 [rgb]0.7388, 0.9, 0.5511.6 [rgb]0.74045, 0.9, 0.5511.9 [rgb]0.7938, 0.9, 0.5578.4 [rgb]0.68545, 0.9, 0.5598.1 [rgb]0.71185, 0.9, 0.5593.3 Gemma-3-12b [rgb]0.53055, 0.74725, 0.794438.9 [rgb]0.53675, 0.71625, 0.84426.5 [rgb]0.5383, 0.7085, 0.856423.4 [rgb]0.84165, 0.9, 0.5530.3 [rgb]0.71185, 0.9, 0.556.7 [rgb]0.70195, 0.9, 0.554.9 [rgb]0.8279, 0.9, 0.5572.2 [rgb]0.7025, 0.9, 0.5595.0 [rgb]0.71405, 0.9, 0.5592.9 Gemma-4-E4B [rgb]0.5298, 0.751, 0.788440.4 [rgb]0.53535, 0.72325, 0.832829.3 [rgb]0.53905, 0.70475, 0.862421.9 [rgb]0.84055, 0.9, 0.5530.1 [rgb]0.72615, 0.9, 0.559.3 [rgb]0.6981, 0.9, 0.554.2 [rgb]0.83945, 0.9, 0.5570.1 [rgb]0.6948, 0.9, 0.5596.4 [rgb]0.7058, 0.9, 0.5594.4 Gemma-4-31B [rgb]0.5254, 0.773, 0.753249.2 [rgb]0.52385, 0.78075, 0.740852.3 [rgb]0.5254, 0.773, 0.753249.2 [rgb]0.8312, 0.9, 0.5528.4 [rgb]0.72065, 0.9, 0.558.3 [rgb]0.71515, 0.9, 0.557.3 [rgb]0.84165, 0.9, 0.5569.7 [rgb]0.6992, 0.9, 0.5595.6 [rgb]0.686, 0.9, 0.5598.0 Close Source Sabiá-4 [rgb]0.53275, 0.73625, 0.81234.5 [rgb]0.5465, 0.6675, 0.9227.0 [rgb]0.54595, 0.67025, 0.91768.1 [rgb]0.85375, 0.9, 0.5532.5 [rgb]0.69975, 0.9, 0.554.5 [rgb]0.7003, 0.9, 0.554.6 [rgb]0.8565, 0.9, 0.5567.0 [rgb]0.7278, 0.9, 0.5590.4 [rgb]0.7454, 0.9, 0.5587.2 Gemini-3-Flash [rgb]0.52225, 0.78875, 0.72855.5 [rgb]0.51845, 0.80775, 0.697663.1 [rgb]0.51815, 0.80925, 0.695263.7 [rgb]0.8048, 0.9, 0.5523.6 [rgb]0.7069, 0.9, 0.555.8 [rgb]0.69755, 0.9, 0.554.1 [rgb]0.78665, 0.9, 0.5579.7 [rgb]0.67885, 0.9, 0.5599.3 [rgb]0.6761, 0.9, 0.5599.8 Table 2: Results across models and prompting settings in P3B3. All scores are scaled to 0 (pt-BR) – 100 (pt-PT) measuring variant alignment. 5.1 Bias and Controllability Analysis Table 2 reports model results across three settings. We note that we measure variant preference in generation, rather than response quality; therefore, lower or higher scores indicate a tendency toward pt-BR or pt-PT, respectively, rather than differences in output quality. No-Prompt. This setting measures inherent model bias, showing that most models exhibit a consistent preference for pt-BR, likely reflecting training data imbalance. AMALIA, which is specifically pre-trained and fine-tuned for pt-PT, is the only model consistently biased toward pt-PT (91.2). In contrast, Gervásio, a LLaMA model fine-tuned for pt-PT, still exhibits a pt-BR bias, suggesting that pre-training data and scale play a critical role in capturing variety-specific behavior. Interestingly, larger and more recent models such as Qwen-3.5 27B, Gemma-4-31B, and Gemini-3-Flash show more balanced behavior across varieties (44.5 to 63.7). pt-BR. When prompted for pt-BR, most models maintain or increase their use of the pt-BR variety, indicating that models are able to follow explicit variety instructions, particularly at larger model sizes. In contrast, AMALIA is an outlier showing a higher tendency toward pt-PT even in this setting, consistent with its pt-PT specialized training setup. pt-PT. Results for pt-PT are more variable. Strong alignment is observed from EuroLLM, Apertus-70B, and AMALIA (≥ 86), while LLaMA-based models struggle to produce consistent pt-PT outputs. All Gemma models perform well in this setting (≥ 92). Notably, Qwen-3.5 shows a substantial improvement over Qwen-3 (32.7 to 86.1), suggesting that changes in training significantly improved pt-PT handling. Despite being pt-BR-oriented, Sabiá-4 generalizes well to pt-PT (87.2). Classifiers vs. LLM Scores. Classifier-based and LLM-based evaluations yield broadly consistent rankings. Overall, classifiers provide a low-cost complementary signal, while LLM-based evaluation serves as the primary analysis due to higher accuracy and interpretability. Figure 3: Alignment across turns for pt-PT prompt. 5.2 Turn-level Analysis Figure 3 shows LLM scores across the first three dialogue turns (later turns are excluded due to sample size). We focus on pt-PT as it has the most variance. Except for Llama, we observe that models with strong initial pt-PT alignment generally maintain consistent behavior across turns. Models with lower initial tendency to pt-PT tend to drift toward pt-BR over time. This suggests that a single initial instruction is often insufficient to sustain language variety adherence across multiple turns, indicating that controllability degrades in longer contexts. For the No-Prompt and pt-BR settings, behavior remains stable across turns (Appendix C). 5.3 Human Analysis Linguistic experts evaluated a random subset of various model outputs in the more challenging pt-PT setting. As observed in the automatic metrics, weaker models tend to drift toward pt-BR, exhibiting proclisis, gerund constructions, pronominal usage (notably você), and pt-BR lexical choices (e.g. geladeira and ônibus). They also show inconsistencies in orthography and accentuation, often producing hybrid outputs that mix pt-BR grammar with partial pt-PT vocabulary Rio-Torto et al. (2022). In contrast, stronger models largely follow pt-PT conventions, indicating better syntactic and morphological control. We provide examples of model and judge outputs in Appendix F. 6 Conclusion We introduced P3B3, a multi-turn benchmark for evaluating bias and controllability in LLM generation across Portuguese variants. Most models exhibit a strong bias toward pt-BR, and while LLMs can be steered toward different varieties, larger and newer models show greater controllability. However, pt-PT performance remains less consistent across models, highlighting the need for better representation of linguistic variation in training and evaluation. Limitations P3B3 provides a systematic evaluation of LLM adherence to Portuguese varieties in conversational domains. The current setup does not include technical or specialized domains, focusing on everyday topics. Moreover, while P3B3 supports analysis of pt-BR and pt-PT, extending it to other Portuguese varieties spoken in countries such as Angola, Mozambique, and Cape Verde is a promising direction for future work. Acknowledgments This work was supported by the AMALIA project under Measure RE-C05-i08 of the Portuguese national Programa de Recuperação e Resiliência. We also acknowledge the support of Fundação para a Ciência e Tecnologia (FCT) and the NOVA LINCS project (UID/04516/2025). Finally, we thank the Barcelona Supercomputing Center (BSC) for providing the computational resources that made this work possible. References H. Q. Abonizio, T. S. Almeida, T. S. Laitz, R. M. Junior, G. K. Bonás, R. Nogueira, and R. Pires (2024) Sabiá-3 technical report. CoRR abs/2410.12049. External Links: Link, Document, 2410.12049 Cited by: §2. N. Aepli, Ç. Çöltekin, R. Van Der Goot, T. Jauhiainen, M. Kazzaz, N. Ljubešić, K. North, B. Plank, Y. Scherrer, and M. Zampieri (2023) Findings of the VarDial evaluation campaign 2023. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), p. 251–261. External Links: Link, Document Cited by: §1. M. AI (2026) Ministral 3. CoRR abs/2601.08584. External Links: Link, Document, 2601.08584 Cited by: Table 3, §1, §2. P. Apertus (2025) Apertus: democratizing open and compliant llms for global language environments. CoRR abs/2509.14233. External Links: Link, Document, 2509.14233 Cited by: Table 3, Table 3. F. Bastos (2024) PeroVaz pt-br classifier. External Links: Link Cited by: §4.2. T. A. Chang and C. Arnett (2025) Global PIQA: evaluating physical commonsense reasoning across 100+ languages and cultures. CoRR abs/2510.24081. External Links: Link, Document, 2510.24081 Cited by: §4.2. Y. Y. Chiu, L. Jiang, B. Y. Lin, C. Y. Park, S. S. Li, S. Ravi, M. Bhatia, M. Antoniak, Y. Tsvetkov, V. Shwartz, and Y. Choi (2025) CulturalBench: a robust, diverse and challenging benchmark for measuring LMs’ cultural knowledge through human-AI red-teaming. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 25663–25701. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1. Common Crawl (2026) Common crawl dataset. Note: https://commoncrawl.org/ Cited by: §1. J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), p. 4171–4186. External Links: Link, Document Cited by: §4.2. A. Ettinger, A. Bertsch, B. Kuehl, D. Graham, D. Heineman, D. Groeneveld, F. Brahman, F. Timbers, H. Ivison, J. Morrison, J. Poznanski, K. Lo, L. Soldaini, M. Jordan, M. F. Chen, M. Noukhovitch, N. Lambert, P. Walsh, P. Dasigi, R. Berry, S. Malik, S. Shah, S. Geng, S. Arora, S. Gupta, T. Anderson, T. Xiao, T. Murray, T. Romero, V. Graf, A. Asai, A. Bhagia, A. Wettig, A. Liu, A. Rangapur, C. Anastasiades, C. Huang, D. Schwenk, H. Trivedi, I. Magnusson, J. Lochner, J. Liu, L. J. V. Miranda, M. Sap, M. Morgan, M. Schmitz, M. Guerquin, M. Wilson, R. Huff, R. L. Bras, R. Xin, R. Shao, S. Skjonsberg, S. Z. Shen, S. S. Li, T. Wilde, V. Pyatkin, W. Merrill, Y. Chang, Y. Gu, Z. Zeng, A. Sabharwal, L. Zettlemoyer, P. W. Koh, A. Farhadi, N. A. Smith, and H. Hajishirzi (2025) Olmo 3. CoRR abs/2512.13961. External Links: Link, Document, 2512.13961 Cited by: Table 3, Table 3. R. M. K. Freitag and T. S. d. Gois (2024) Performance in a dialectal profiling task of llms for varieties of brazilian portuguese. In Anais do XV Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL 2024), STIL 2024, p. 317–326. External Links: Link, Document Cited by: §2. G. Garcia, R. Rodrigues, D. Silva, et al. (2024) Bode: a LLaMA-based large language model for Portuguese. arXiv preprint arXiv:2401.00000. Cited by: §2. Gemma Team (2026) Gemma 4. External Links: Link Cited by: Table 3, Table 3, Appendix E, footnote 2. J. Á. González, I. Borrego-Obrador, Á. R. Herrero, A. M. Sarvazyan, M. Chinea-Rios, A. Basile, and M. Franco-Salvador (2026) IberBench: LLM evaluation on iberian languages. Comput. Speech Lang. 96, p. 101899. External Links: Link, Document Cited by: §1. A. Gonzalez-Agirre, M. Pàmies, J. Llop, I. Baucells, S. D. Dalt, D. Tamayo, J. J. Saiz, F. Espuña, J. Prats, J. Aula-Blasco, M. Mina, I. Pikabea, A. Rubio, A. Shvets, A. Salles, I. Lacunza, J. Palomar, J. Falcão, L. Tormo-Bañuelos, L. Vasquez-Reina, M. Marimon, O. Pareras, V. Ruíz-Fernández, and M. Villegas (2025) Salamandra technical report. CoRR abs/2502.08489. External Links: Link, Document, 2502.08489 Cited by: Table 3. J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu, Y. Wang, and J. Guo (2024) A survey on llm-as-a-judge. CoRR abs/2411.15594. External Links: Link, Document, 2411.15594 Cited by: §1. M. Hassan, F. Khattak, and L. Seyyed-Kalantari (2025) Dialectic preference bias in large language models. Proceedings of the AAAI Symposium Series 5, p. 365–369. External Links: Document Cited by: §2. T. Laitz, T. S. Almeida, H. Abonizio, R. M. Junior, G. K. Bonás, M. Piau, C. Larcher, R. Pires, and R. Nogueira (2026) Sabiá-4 technical report. External Links: 2603.10213, Link Cited by: Table 3, §2, §4.3. J. Lee, S. Kim, J. Han, J. Lee, K. Kim, A. Oh, and E. Choi (2025) Trans-env: a framework for evaluating the linguistic robustness of llms against english varieties. External Links: 2505.20875, Link Cited by: §2. R. Lopes, J. Magalhães, and D. Semedo (2024) Glória: A generative and open large language model for portuguese. In Proceedings of the 16th International Conference on Computational Processing of Portuguese, PROPOR 2024, Santiago de Compostela, Galicia/Spain, March 12-15, 2024, Volume 1, P. Gamallo, D. B. Claro, A. J. S. Teixeira, L. Real, M. García, H. G. Oliveira, and R. Amaro (Eds.), p. 441–453. External Links: Link Cited by: §2. P. H. Martins, P. Fernandes, J. Alves, N. M. Guerreiro, R. Rei, D. M. Alves, J. Pombal, M. A. Farajian, M. Faysse, M. Klimaszewski, P. Colombo, B. Haddow, J. G. C. de Souza, A. Birch, and A. F. T. Martins (2024) EuroLLM: multilingual language models for europe. CoRR abs/2409.16235. External Links: Link, Document, 2409.16235 Cited by: Table 3. E. Pan, A. S. G. Choi, M. ter Hoeve, S. Seto, and A. Koenecke (2025) Analyzing dialectical biases in llms for knowledge and reasoning benchmarks. External Links: 2510.00962, Link Cited by: §2. G. Penedo, H. Kydlíček, L. B. Allal, A. Lozhkov, M. Mitchell, C. Raffel, L. Von Werra, and T. Wolf (2024) The fineweb datasets: decanting the web for the finest text data at scale. NIPS ’24. External Links: ISBN 9798331314385 Cited by: §1. D. Preda, T. F. Osório, and H. L. Cardoso (2024) Across the atlantic: distinguishing between european and brazilian portuguese dialects. In Proceedings of the 16th International Conference on Computational Processing of Portuguese, PROPOR 2024, Santiago de Compostela, Galicia/Spain, March 12-15, 2024, Volume 1, p. 353–363. External Links: Link Cited by: §1, §2. P. Riley, T. Dozat, J. A. Botha, X. Garcia, D. Garrette, J. Riesa, O. Firat, and N. Constant (2023) FRMT: A benchmark for few-shot region-aware machine translation. Trans. Assoc. Comput. Linguistics 11, p. 671–685. External Links: Link, Document Cited by: §1. G. Rio-Torto, T. Ferreira, Z. Greksakova, A. I. Guerra, and Y. Zhang (2022) Português brasileiro e português europeu: um diálogo de séculos. Universidade Politécnica de Macau. Cited by: §1, §2, §3.1, §5.3. R. Santos, J. Silva, L. Gomes, J. Rodrigues, and A. Branco (2024) Advancing generative AI for portuguese with open decoder gervásio PT. CoRR abs/2402.18766. External Links: Link, Document, 2402.18766 Cited by: Table 3, Table 3. A. Simplício, G. Vinagre, M. M. Ramos, D. Tavares, R. Ferreira, G. Attanasio, D. M. Alves, I. Calvo, I. Vieira, R. Guerra, J. Furtado, B. Canaverde, I. Paulo, V. Ramos, D. Glória-Silva, M. Faria, M. Treviso, D. Gomes, P. Gomes, D. Semedo, A. Martins, and J. Magalhães (2026) AMALIA: a fully open large language model for European Portuguese. In Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1, Salvador, Brazil, p. 380–391. External Links: Link, ISBN 979-8-89176-387-6 Cited by: Table 3, §1, §2. H. O. Sousa, R. Almeida, P. Silvano, I. Cantante, R. Campos, and A. M. Jorge (2025) Enhancing portuguese variety identification with cross-domain approaches. In Thirty-Ninth AAAI Conference on Artificial Intelligence, Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence, Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI, T. Walsh, J. Shah, and Z. Kolter (Eds.), p. 25192–25200. External Links: Link, Document Cited by: §1, §2, §4.2. G. Team (2025a) Gemma 3 technical report. CoRR abs/2503.19786. External Links: Link, Document, 2503.19786 Cited by: Table 3. G. Team (2025b) Gemini 3 flash model card: frontier-class performance at scale. Technical report Google DeepMind. Note: Accessed: 2026-05-01 External Links: Link Cited by: Table 3, §4.2, §4.3. L. Team (2024) The llama 3 herd of models. CoRR abs/2407.21783. External Links: Link, Document, 2407.21783 Cited by: Table 3, Table 3, §1, §2. Q. Team (2025c) Qwen3 technical report. External Links: 2505.09388, Link Cited by: Table 3, §1, §2. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou (2022) Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS 2022, External Links: Link Cited by: §4.2. L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, External Links: Link Cited by: §3.1, §4.1. Appendix A P3B3 Benchmark Statistics P3B3 was constructed through a manual process conducted by two language experts with expertise in Portuguese and comprises of 74 conversations and 203 turns. Structural Distribution. The dataset targets short-form multi-turn interactions with dialogue turns distributed as follows: 31 dialogues have 2 turns, 35 have 3, 5 have 4, and 3 have 5–6 turns. Lexical and Sequence Diversity. The dataset shows strong lexical diversity. Per-conversation TTR is 0.73, with Distinct-2 and Distinct-3 scores of 0.71 and 0.88, indicating high phrasal variation. Lower Distinct-1 (0.26) and global TTR (0.25) reflect expected function word frequency. Entropy increases from 7.7 (1-gram) to 10.9 (3-gram), confirming linguistic variability via different conversational domains. Length Statistics. Turns average 14.1 words (SD=6.0), with word counts averaging 38.7 per dialogue and ranging from 14 to 74 words, consistent with the benchmark’s focused design. Appendix B Detailed Model List Table 3 provides a comprehensive list of the models evaluated in this work, along with their respective model identifiers. Model Provider & Name Model Identifier BSC-LT Salamandra 7B Gonzalez-Agirre et al. (2025) salamandra-7b-instruct allenai OLMo 3 7B Ettinger et al. (2025) Olmo-3-7B-Instruct OLMo 3.1 32B Ettinger et al. (2025) Olmo-3.1-32B-Instruct utter-project EuroLLM 22B Martins et al. (2024) EuroLLM-22B-Instruct-2512 swiss-ai Apertus 8B Apertus (2025) Apertus-8B-Instruct-2509 Apertus 70B Apertus (2025) Apertus-70B-Instruct-2509 AMALIA-LLM AMALIA 9B Simplício et al. (2026) AMALIA-9B-50-1225-DPO meta-llama Llama 3.1 8B Instruct Team (2024) Llama-3.1-8B-Instruct Llama 3.3 70B Team (2024) Llama-3.3-70B-Instruct PORTULAN Gervásio 8B-pt-pt Santos et al. (2024) gervasio-8b-portuguese-ptpt-decoder Gervásio 70B-pt-pt Santos et al. (2024) gervasio-70b-portuguese-ptpt-decoder mistralai Ministral 3 14B AI (2026) Ministral-3-14B-Instruct-2512 Qwen Qwen 3 8B Team (2025c) Qwen3-8B Qwen 3.5 9B Qwen3.5-9B Qwen 3.5 27B Qwen3.5-27B google Gemma 3 12B Team (2025a) gemma-3-12b-it Gemma 4 E4B Gemma Team (2026) gemma-4-E4B-it Gemma 4 31B Gemma Team (2026) gemma-4-31B-it API-only Gemini 3 Flash Preview Team (2025b) Google API - gemini-3-flash-preview Sabiá 4 Laitz et al. (2026) Maritaca API - sabia-4 Table 3: Model names and identifiers used in this work. Appendix C Turn-level Analysis Figures 6 and 6 show LLM scores across the first three dialogue turns for the No-Prompt and pt-BR settings, respectively. In contrast to the pt-PT setting (Figure 3), where alignment progressively shifts toward pt-BR, these configurations exhibit largely stable behavior across turns. This consistency suggests a persistent model bias toward pt-BR, which remains unchanged throughout the interaction, with no tendency to move toward pt-PT. Figure 4: Translated examples from P3B3 benchmark. Original Portuguese version in Figure 2. Figure 5: Alignment across turns for No-Prompt. Figure 6: Alignment across turns for pt-BR prompt. Appendix D Portuguese Output Consistency We used Gemini-3 Flash to classify whether outputs were predominantly in Portuguese, regardless of variant. Most models consistently generate outputs in Portuguese (99%). The only exception is OLMo-3-7B, which still produces Portuguese in 94% of cases, with occasional outputs in Spanish. Appendix E Extending the LLM-as-Judge Setup We additionally include Gemma-4-31B Gemma Team (2026) in our LLM-as-Judge setup in Table 4. Gemma-4-31B performs strongly across all settings and metrics. While it trails Gemini-3-Flash, it significantly improves over the classifiers and serves as a strong open-source alternative for this task, with competitive performance. With respect to the results, Table 5 compares the evaluations produced by Gemini-3-Flash and Gemma-4-31B across all models and prompting settings in P3B3. We observe a strong agreement between the two judges, both in ranking order and overall score ranges, with only a few discrepancies in specific settings. These findings further validate our results and highlight the potential of using open-weight LLM judges to reliably measure Portuguese variant alignment. Appendix F Example Model and Judge Outputs Figures 7, 8, and 9 show example model and judge outputs across different settings. Method Weighted κ (Quadratic) Pearson MAE ↓ Humans 0.69 0.75 2.07 PeroVaz (PV) 0.42 0.50 2.43 PtVId (VId) 0.57 0.60 2.66 Gemini-3 Flash - EN-S-ST 0.79 0.81 1.79 Gemini-3 Flash - PT-S-ST 0.79 0.82 1.72 Gemini-3 Flash - PT-S-H 0.80 0.83 1.72 Gemini-3 Flash - PT-C-ST 0.81 0.83 1.58 Gemini-3 Flash - PT-C-D 0.81 0.82 1.60 Gemma-4-31B - EN-S-ST 0.75 0.78 1.95 Gemma-4-31B - PT-S-ST 0.78 0.80 1.73 Gemma-4-31B - PT-S-H 0.78 0.80 1.74 Gemma-4-31B - PT-C-ST 0.78 0.79 1.72 Gemma-4-31B - PT-C-D 0.78 0.80 1.71 Table 4: Agreement between human annotations and LLM-based metrics. EN/PT = language; S = Simple; C = Category; ST = Single-Turn; D = Dialogue history. No-Prompt pt-BR (↓)( ) pt-PT (↑ ) Model Gemma-4 Gemini-3 Gemma-4 Gemini-3 Gemma-4 Gemini-3 Fully Open Salamandra-7B [rgb]0.53225, 0.73875, 0.80835.5 [rgb]0.53275, 0.73625, 0.81234.5 [rgb]0.8092, 0.9, 0.5524.4 [rgb]0.78885, 0.9, 0.5520.7 [rgb]0.9258, 0.9, 0.5554.4 [rgb]0.9346, 0.9, 0.5552.8 OLMo-3-7B [rgb]0.53845, 0.70775, 0.857623.1 [rgb]0.54115, 0.69425, 0.879217.7 [rgb]0.77455, 0.9, 0.5518.1 [rgb]0.7619, 0.9, 0.5515.8 [rgb]0.95, 0.7536, 0.5525.6 [rgb]0.95, 0.7398, 0.5523.3 OLMo-3.1-32B [rgb]0.5415, 0.6925, 0.88217.0 [rgb]0.5417, 0.6915, 0.883616.6 [rgb]0.74815, 0.9, 0.5513.3 [rgb]0.73275, 0.9, 0.5510.5 [rgb]0.95, 0.8862, 0.5547.7 [rgb]0.95, 0.8568, 0.5542.8 EuroLLM-22B [rgb]0.5383, 0.7085, 0.856423.4 [rgb]0.53875, 0.70625, 0.8622.5 [rgb]0.7399, 0.9, 0.5511.8 [rgb]0.7278, 0.9, 0.559.6 [rgb]0.7575, 0.9, 0.5585.0 [rgb]0.73825, 0.9, 0.5588.5 Apertus-8B [rgb]0.53955, 0.70225, 0.866420.9 [rgb]0.53985, 0.70075, 0.868820.3 [rgb]0.7619, 0.9, 0.5515.8 [rgb]0.75585, 0.9, 0.5514.7 [rgb]0.89995, 0.9, 0.5559.1 [rgb]0.8895, 0.9, 0.5561.0 Apertus-70B [rgb]0.54115, 0.69425, 0.879217.7 [rgb]0.54195, 0.69025, 0.885616.1 [rgb]0.74155, 0.9, 0.5512.1 [rgb]0.7223, 0.9, 0.558.6 [rgb]0.77345, 0.9, 0.5582.1 [rgb]0.75035, 0.9, 0.5586.3 AMALIA-9B [rgb]0.50625, 0.86875, 0.687.5 [rgb]0.5044, 0.878, 0.585291.2 [rgb]0.8609, 0.9, 0.5533.8 [rgb]0.8664, 0.9, 0.5534.8 [rgb]0.7201, 0.9, 0.5591.8 [rgb]0.69755, 0.9, 0.5595.9 Open Weight Llama-3.1-8B [rgb]0.54295, 0.68525, 0.893614.1 [rgb]0.5437, 0.6815, 0.899612.6 [rgb]0.7344, 0.9, 0.5510.8 [rgb]0.7333, 0.9, 0.5510.6 [rgb]0.95, 0.8868, 0.5547.8 [rgb]0.95, 0.8862, 0.5547.7 Llama-3.3-70B [rgb]0.5417, 0.6915, 0.883616.6 [rgb]0.54235, 0.68825, 0.888815.3 [rgb]0.7234, 0.9, 0.558.8 [rgb]0.71185, 0.9, 0.556.7 [rgb]0.84275, 0.9, 0.5569.5 [rgb]0.8521, 0.9, 0.5567.8 Gervásio-8B-ptpt [rgb]0.54255, 0.68725, 0.890414.9 [rgb]0.54365, 0.68175, 0.899212.7 [rgb]0.7432, 0.9, 0.5512.4 [rgb]0.73605, 0.9, 0.5511.1 [rgb]0.95, 0.8778, 0.5546.3 [rgb]0.95, 0.8796, 0.5546.6 Gervásio-70B-ptpt [rgb]0.53835, 0.70825, 0.856823.3 [rgb]0.5386, 0.707, 0.858822.8 [rgb]0.741, 0.9, 0.5512.0 [rgb]0.7344, 0.9, 0.5510.8 [rgb]0.8048, 0.9, 0.5576.4 [rgb]0.8114, 0.9, 0.5575.2 Ministral-3-14B [rgb]0.5402, 0.699, 0.871619.6 [rgb]0.5411, 0.6945, 0.878817.8 [rgb]0.6981, 0.9, 0.554.2 [rgb]0.6981, 0.9, 0.554.2 [rgb]0.71075, 0.9, 0.5593.5 [rgb]0.72615, 0.9, 0.5590.7 Qwen3-8B [rgb]0.544, 0.68, 0.90212.0 [rgb]0.5448, 0.676, 0.908410.4 [rgb]0.7256, 0.9, 0.559.2 [rgb]0.71845, 0.9, 0.557.9 [rgb]0.95, 0.7992, 0.5533.2 [rgb]0.95, 0.7962, 0.5532.7 Qwen3.5-9B [rgb]0.53195, 0.74025, 0.805636.1 [rgb]0.53245, 0.73775, 0.809635.1 [rgb]0.74815, 0.9, 0.5513.3 [rgb]0.7421, 0.9, 0.5512.2 [rgb]0.73495, 0.9, 0.5589.1 [rgb]0.75145, 0.9, 0.5586.1 Qwen3.5-27B [rgb]0.5278, 0.761, 0.772444.4 [rgb]0.52775, 0.76125, 0.77244.5 [rgb]0.7432, 0.9, 0.5512.4 [rgb]0.74045, 0.9, 0.5511.9 [rgb]0.70195, 0.9, 0.5595.1 [rgb]0.71185, 0.9, 0.5593.3 Gemma-3-12b [rgb]0.5372, 0.714, 0.847625.6 [rgb]0.5383, 0.7085, 0.856423.4 [rgb]0.7124, 0.9, 0.556.8 [rgb]0.70195, 0.9, 0.554.9 [rgb]0.71295, 0.9, 0.5593.1 [rgb]0.71405, 0.9, 0.5592.9 Gemma-4-E4B [rgb]0.53845, 0.70775, 0.857623.1 [rgb]0.53905, 0.70475, 0.862421.9 [rgb]0.7091, 0.9, 0.556.2 [rgb]0.6981, 0.9, 0.554.2 [rgb]0.71405, 0.9, 0.5592.9 [rgb]0.7058, 0.9, 0.5594.4 Gemma-4-31B [rgb]0.5258, 0.771, 0.756448.4 [rgb]0.5254, 0.773, 0.753249.2 [rgb]0.72395, 0.9, 0.558.9 [rgb]0.71515, 0.9, 0.557.3 [rgb]0.6981, 0.9, 0.5595.8 [rgb]0.686, 0.9, 0.5598.0 Close Source Sabiá-4 [rgb]0.5453, 0.6735, 0.91249.4 [rgb]0.54595, 0.67025, 0.91768.1 [rgb]0.71845, 0.9, 0.557.9 [rgb]0.7003, 0.9, 0.554.6 [rgb]0.74705, 0.9, 0.5586.9 [rgb]0.7454, 0.9, 0.5587.2 Gemini-3-Flash [rgb]0.5194, 0.803, 0.705261.2 [rgb]0.51815, 0.80925, 0.695263.7 [rgb]0.70085, 0.9, 0.554.7 [rgb]0.69755, 0.9, 0.554.1 [rgb]0.68215, 0.9, 0.5598.7 [rgb]0.6761, 0.9, 0.5599.8 Table 5: LLM-as-a-Judge comparison between Gemma-4-31B and Gemini-3-Flash across models and prompting strategies in P3B3. Scores are normalized to a 0–100 scale, where 0 corresponds to pt-BR and 100 to pt-PT, measuring alignment with each Portuguese variant. Figure 7: Example of the LLM Judge score and reasoning given to an answer from AMALIA-9B for No-Prompt setting. The colors represent the points in the answer the judge’s reasoning is addressing. EN Translation: Question: My car is in the shop. What alternative transportation can I use? Answer: You have several options! You can catch a taxi, use public transport such as the bus or metro, or ask friends or colleagues for a ride. If you prefer something more comfortable, you can consider renting a car for a day or using a car-sharing app. It depends a bit on where you are and your budget, but there are always alternatives. LLM Judge: Score: 10 The text presents a total marking of European Portuguese (PT-PT), without any trace of Brazilian Portuguese (PT-BR). 1. Form of Address (…) 2. Lexicon and Vocabulary (…) 3. Verb Associated with the Noun (…) 4. Pronominalization of the Subject (…) 5. Spelling (…) Figure 8: Example of the LLM Judge score and reasoning given to an answer from Sabiá-4 for a pt-BR prompt setting. The colors represent the points in the answer the judge’s reasoning is addressing. EN Translation: Question: My car is in the repair shop. What transport can I use as an alternative? Answer: When your car is in the shop, you can consider various transport alternatives, depending on your city and needs. Here are some common options: 1. Public Transport: Bus Subway Urban train (…) LLM Judge: Score: 0 The text shows a total bias toward Brazilian Portuguese (PT-BR), containing no linguistic markers of European Portuguese (PT-EU). 1. Form of Address (…) 2. Lexicon and Vocabulary (…) 3. Spelling (…) Figure 9: Example of the LLM Judge score and reasoning given to an answer from Gemma-4-31B for a pt-PT prompt setting. The colors represent the points in the answer the judge’s reasoning is addressing. EN Translation: Question: My car is in the repair shop. What transport can I use as an alternative? Answer: Depending on where you live and where you travel to, you have several alternatives. Here are the main options divided by categories: 1. Public Transport It is the most common and, generally, the most economical option: Buses and Metro: Ideal for urban travel. Check the app of the local operator (ex: Carris, STCP, Metro Porto). (…) LLM Judge: Score: 10 The text presents a total marking of European Portuguese (PT-PT), without any trace of Brazilian Portuguese (PT-BR). 1. Lexicon and Vocabulary (…) 2. Form of Address and Subject (…) 3. Spelling (…)