Paper deep dive

Tracing Stereotypes from Representation to Output in Multilingual LLMs

Ariun-Erdene Tumurchuluun, Yusser Al Ghussin, Pinzhen Chen, Josef van Genabith, Koel Dutta Chowdhury

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 63

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 9/14/2026, 5:05:12 AM

Summary

This study investigates the internal mechanisms of stereotype representation in multilingual Large Language Models (LLMs) by comparing linear probing, attribution patching, and sparse autoencoder (SAE) feature ablation across Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B. The authors find that stereotype information is linearly decodable in middle layers (36-53% of model depth) but causally influences output only in later layers. SAE analysis reveals that most bias features are language-specific and category-specific, with only a small fraction (6-18%) being language-agnostic. The study highlights a dissociation between where stereotypes are stored and where they affect generation, emphasizing the need to measure decodability, output influence, and cross-lingual effects separately.

Entities (14)

Gemma-2-9B · model · 99%LLaMA-3.1-8B · model · 99%Qwen3-8b · model · 99%Stereotype Bias · concept · 98%Attribution Patching · method · 95%Linear Probing · method · 95%Social Categories · concept · 95%Sparse Autoencoders · method · 95%MBBQ · dataset · 92%SHADES · dataset · 92%Gemma Scope · sae-suite · 90%Llama-Multi · sae-suite · 90%Llama Scope · sae-suite · 90%Qwen-Multi · sae-suite · 90%

Relation Signals (10)

Qwen-Multi → usedon → Qwen3-8b

confidence 95% · multilingual SAEs on Qwen3-8B (Qwen-Multi)

Llama-Multi → usedon → LLaMA-3.1-8B

confidence 95% · multilingual SAEs on Llama-3.1-8B (Llama-Multi, Al Ghussin et al., 2026)

Llama Scope → usedon → LLaMA-3.1-8B

confidence 95% · English-trained SAEs on Llama-3.1-8B (Llama-Scope, He et al., 2024)

Gemma Scope → usedon → Gemma-2-9B

confidence 95% · English-trained SAEs on Gemma-2-9B (Gemma-Scope, Lieberum et al., 2024)

Linear Probing → identifieslocationof → Stereotype Bias

confidence 90% · Probe performance peaks substantially earlier than attribution in all three models, with a separation of 36-53% of model depth.

Attribution Patching → identifieslocationof → Output Influence

confidence 90% · Attribution magnitude peaks near the output, with a layer separation of 36–53% of model depth between them.

MBBQ → usedfor → Feature Identification

confidence 90% · We use MBBQ (Neplenbroek et al., 2024) for layer-level analysis and sparse feature identification.

SHADES → usedfor → Evaluation

confidence 90% · we ablate individual features and measure their effects on a separate benchmark, SHADES

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Multilingual LLMs show stereotype-related behavior that varies across languages, but behavioral scores do not show where the relevant information is represented or how it affects the output. To investigate these internal mechanisms, we compare linear probing, attribution patching, sparse autoencoders (SAEs) and feature ablation in Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B. Probe performance peaks substantially earlier than attribution in all three models, with a separation of 36-53% of model depth. Retained Llama-Scope features often match the social category on which they were selected and form recurring semantic families, but their lexical alignment and ablation effects vary across SAE suites. Only 6-18% of evaluated residual-stream features have language-agnostic effects under our criterion, and none are category-agnostic. Language-agnostic features have larger mean ablation effects in Llama-Scope, but this pattern does not repeat in the other SAE suites. Decodability, output influence, and cross-lingual ablation effects therefore need to be measured separately.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

60,404 characters extracted from source content.

Expand or collapse full text

Tracing Stereotypes from Representation to Output in Multilingual LLMs Ariun-Erdene Tumurchuluun 1 * Yusser Al Ghussin 1,3 * Pinzhen Chen 2 Josef van Genabith 1,3 Koel Dutta Chowdhury 4† 1 Saarland University, Saarland Informatics Campus 2 Queen’s University Belfast 3 German Research Center for Artificial Intelligence (DFKI) 4 University of Technology Nuremberg ariunerdene.tn@gmail.com Abstract Multilingual LLMs show stereotype-related be- havior that varies across languages, but be- havioral scores do not show where the rel- evant information is represented or how it affects the output. To investigate these in- ternal mechanisms, we compare linear prob- ing, attribution patching, sparse autoencoders (SAEs) and feature ablation in Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B. Probe perfor- mance peaks substantially earlier than attribu- tion in all three models, with a separation of 36– 53% of model depth. Retained Llama-Scope features often match the social category on which they were selected and form recurring se- mantic families, but their lexical alignment and ablation effects vary across SAE suites. Only 6– 18% of evaluated residual-stream features have language-agnostic effects under our criterion, and none are category-agnostic. Language- agnostic features have larger mean ablation ef- fects in Llama-Scope, but this pattern does not repeat in the other SAE suites. Decodability, output influence, and cross-lingual ablation ef- fects therefore need to be measured separately. 1 Introduction Multilingual Large Language Models (LLMs) ex- hibit systematic stereotype-related behavior that varies substantially across languages. A growing body of behavioral evaluations has documented such variation across models and social categories (Neplenbroek et al., 2024; Bhutani et al., 2024; Mitchell et al., 2025). Importantly, these biases are neither uniform nor simple translations of one another: both their magnitude and direction vary substantially across languages, even for parallel prompts and identical social categories. * Equal contribution. † Work conducted while at Saarland University. Code and data available athttps://github.com/ariun erdenetum/stereotype-tracing-mllm. Where is stereotype information decodable vs. used? decodablecausally used dissociation: ≈ a third and half of the model's depth Interpreting sparse features inactive latents Ablating sparse features language specificity → category specificity social category, not language, is the binding constraint SAE dense activation category-agnostic 0% 82–94% both specific general mediator 0% 6–18% lang.-agnostic Age: “aging” Disability: “autism” Gender: “male subjects” Appearance: “obesity” SES: “social class” Sexual orient.: “LGBTQ+” mid-layerslater layersearly-layers Figure 1: Framework and findings. Top: stereotype information is linearly decodable between roughly a third and half of the network before it causally shapes the output. Middle: the decomposed representation is interpretable and only category-specific. Bottom: Most bias features are language-dependent and category- specific, with only a minority being language-agnostic and category-specific. Most evidence for these differences across lan- guages comes from model outputs.However, output-level behavior provides only a partial view of model bias: models that appear unbiased under explicit evaluation can still retain biased associa- tions (Bai et al., 2025). For multilingual models, the same behavioral difference can therefore have different internal explanations. Stereotype-related information may differ in how strongly it is repre- sented across languages, in how strongly similar representations influence the prediction, or both. Distinguishing these explanations requires sep- arating where stereotype-related information is most decodable from where it most strongly influences the model’s output. Linear probing arXiv:2609.08322v1 [cs.CL] 8 Sep 2026 can identify where such information becomes lin- early accessible, but probe performance does not establish that it substantially affects the predic- tion (Alain and Bengio, 2018; Ravichander et al., 2021). Attribution-based analyses instead estimate how internal states influence the output (Nanda, 2023; Syed et al., 2024). Prior multilingual studies have found both language-specific and language- agnostic representations (Wendler et al., 2024; Deng et al., 2025; Dumas et al., 2025; Wang et al., 2025), but whether such representations have sim- ilar effects on model behavior across languages remains unclear. The central question is therefore whether decodability, estimated output influence, and feature-level intervention effects identify the same parts of stereotype-related behavior or not. We investigate these questions in Llama-3.1- 8B, Qwen3-8B, and Gemma-2-9B across four lan- guages and six social categories (Figure 1). We first compare layer-wise linear probing with attri- bution patching to test whether peak decodability coincides with peak estimated causal influence. We then use sparse autoencoders (SAEs; Bricken et al., 2023; Cunningham et al., 2023; Gao et al., 2024) to identify candidate sparse features and examine how they transfer across languages and social categories. Finally, we ablate individual features and measure their effects on a separate benchmark, SHADES (BiasShades release, Mitchell et al., 2025), sepa- rating feature discovery on MBBQ (Neplenbroek et al., 2024) from downstream evaluation. Our analysis yields three main findings. 1 Peak probe decodability precedes peak out- put patching influence. Across all three models, probe performance peaks in middle layers, whereas attribution magnitude peaks near the output, with a layer separation of36–53%of model depth be- tween them (§4 and §5). 2 Sparse features are social category-linked, but their ablation effects are not fixed. Among the 139 glossed Llama-Scope features, 123 match the social category on which they were selected and form recurring semantic families. Individual ablations can either reduce or increase measured bias, and lexical alignment varies strongly across SAE suites (§6). 3Cross-language stability does not reliably predict larger ablation effects. Across the four SAE suites,6–18%of evaluated residual-stream features are language-agnostic under our criterion, while none are category-agnostic. The language- agnostic group has larger mean ablation effects in Llama-Scope, but this pattern is absent in Llama- Multi and Qwen-Multi and reverses for some Gemma-Scope settings (§7). Overall, our results distinguish three questions that are often conflated: where stereotype-related information is most decodable, where it has its strongest estimated influence on model outputs, and whether the effects of selected sparse features transfer across languages and social categories. 2 Background and Related Work We review the behavioral evaluation of stereotype bias across languages in §2.1, followed by mecha- nistic analyses of multilingual representations and social bias in §2.2. 2.1 Multilingual Stereotype Evaluation Stereotype bias in multilingual LLMs has primarily been studied through behavioral evaluation. BBQ (Parrish et al., 2022) evaluates social biases through ambiguous and disambiguated question-answering scenarios, and MBBQ (Neplenbroek et al., 2024) extends this setup to English, Spanish, Dutch, and Turkish using parallel data items for cross-lingual comparison. SeeGULL Multilingual (Bhutani et al., 2024) covers twenty languages of geo-culturally sit- uated stereotypes, while SHADES (Mitchell et al., 2025) evaluates stereotype associations across lan- guages and social groups. A recent survey of multi- lingual bias work also notes that evaluation remains concentrated on a limited set of languages and that multilingual mitigation experiments are com- paratively scarce (Gamboa et al., 2025). Across these studies, stereotype-related behavior varies across languages rather than following a single cross-lingual pattern. Models that appear unbiased under explicit evaluation can also retain biased as- sociations (Bai et al., 2025). 2.2 Mechanistic Analyses of Multilingual Representations and Bias Different interpretability methods provide evidence about different properties of a model’s internal com- putation. Linear probes isolate information that becomes linearly accessible in hidden states (Alain and Bengio, 2018), attribution patching estimates the influence of individual model components on outputs (Nanda, 2023; Syed et al., 2024) as a first- order approximation to activation patching (Meng et al., 2023), and SAEs decompose dense activa- tions into sparse features (Bricken et al., 2023; Cun- ningham et al., 2023; Gao et al., 2024), whose latents separate into input-detecting and output- driving roles across depth (Arad et al., 2025). In multilingual models, Wendler et al. (2024) show that Llama-2 processes non-English prompts through an abstract “concept space” that lies closer to English than to the input language, Deng et al. (2025) use SAEs to identify language-specific fea- tures, and Dumas et al. (2025) use activation patch- ing to identify language-agnostic concept repre- sentations. Wang et al. (2025) find that factual knowledge remains largely language-independent through much of the network before a late transi- tion toward language-specific representations. Tu- murchuluun et al. (2025) combine probing, causal tracing, and SAE-based steering to trace multilin- gual tense representations from localization to in- tervention. For social and cultural behavior, Ne- plenbroek et al. (2025) use probing-based steering for multilingual stereotypes, Simbeck and Mahran (2025) analyze religion-linked SAE features, and Yu et al. (2025) use patching to study internal cul- tural representations and Western-dominance bias. More recently, Zou et al. (2026) combine SAE fea- ture analysis with ablation and steering for cultural knowledge. Song et al. (2026) argue that run-to- run feature consistency should be a standard SAE evaluation axis. More generally, decodability does not establish causal relevance. Probe accuracy likewise does not establish that the decoded information is used for the task (Ravichander et al., 2021), and re- cent benchmarks evaluate interpretability methods through intervention behavior rather than decoding alone (Arora et al., 2024; Huang et al., 2024). Whether decodability and causal influence sim- ilarly diverge for stereotype-related information across languages and model depth remains unclear. Our experiments make these comparisons in a mul- tilingual stereotype setting. We compare probe and attribution profiles, then test selected SAE features by direct ablation and check whether the effects repeat across languages, social categories, models, and SAE suites. 3 Experimental Setup We trace stereotype-related information from its linear decodability to its influence on model out- puts and, finally, to feature-level intervention. We investigate the following questions: •Where is stereotype information most decod- able? We train and use logistic regression mod- I saw a grandson and their grandfather last week outside the supermarket trying to book a cab on their phone. DISAMBIGUATING EVIDENCEThe grandson was struggling with the app and not really listening to the advice from the grandfather. Who was not comfortable using the phone? A Cannot answerB GrandfatherC Grandson ENESNLTR Disambig.CCBC AmbiguousBBBA Table 1: Inference on an MBBQ example (Neplen- broek et al., 2024) with Llama-3.1-8B. Bold denotes the stereotype-consistent answer (grandfather, assuming the older person is less comfortable with technology). Thecorrect answeris A (cannot answer) when ambigu- ous and C (grandson) when disambiguated. els on layer-wise hidden states to locate linearly accessible stereotype representations (§4). •Where does stereotype-related information most strongly influence the output? We use attribution patching to quantify the causal effect across model depth (§5). • What social categories do the selected SAE features represent? We use lexical anchors and available feature glosses to relate the se- lected sparse features to social categories (§6). •What are the effects and transferability of feature ablation? We systematically ablate individual features on SHADES and compare their effects across languages and social cate- gories (§7). Models. We select three decoder-only multilin- gual models of comparable sizes:Llama-3.1-8B (Grattafiori et al., 2024),Gemma-2-9B(Team et al., 2024), andQwen3-8B(Qwen Team, 2025). This controls size as a source of variation while covering three independently developed model families. Datasets. We use MBBQ (Neplenbroek et al., 2024) for layer-level analysis and sparse feature identification. MBBQ provides parallel items in En- glish, Spanish, Dutch, and Turkish across six social categories (age, disability status, gender identity, physical appearance, socioeconomic status (SES), and sexual orientation). We evaluate feature inter- ventions on a held-out set SHADES (Mitchell et al., 2025), which covers three of the four languages we study (English, Spanish, and Dutch). Table 1 showsLlama-3.1-8B’s responses to a prompt across four languages. In the ambigu- ous setup with no context, the model gives the stereotype-consistent answer (“the grandfather”) in three of four languages, and only Turkish cor- rectly refuses to commit. Concurrently, even with a disambiguating sentence explicitly identifying the grandson as the one struggling, Dutch still picks the grandfather, overriding the contextual evidence with the possible learned stereotypical association. The example shows same prompt, same multilin- gual transformer, and different behaviors in differ- ent languages. 4 Where Is Stereotype Information Decodable? We first ask where stereotype-related completion conditions become linearly distinguishable across model depth. For each MBBQ disambiguated item, we construct a contrastive pair by appending either the factual answer or the stereotype-consistent an- swer, yielding sequences of the form “<context> <Q> <correct_answer>” and “<context> <Q> <biased_answer> ”. We use disambiguated rather than ambiguous contexts because they provide ex- plicit evidence against the stereotype-consistent completion. 4.1 Linear Probing Setup At each layer, we mean-pool hidden states over the full sequence and fit aL 2 -regularized logis- tic regression probe with scikit-learn (Pedregosa et al., 2011), sweeping the inverse regularization strengthC ∈0.1, 1.0(smallerCmeans stronger regularization), and report macro-F1. Unless noted otherwise, reported values useC=1.0. The train- test split is 80/20 employing disambiguated rows of MBBQ. To test whether the resulting depth profile can be explained by probe capacity, we additionally train control tasks that assign each word type a ran- domly sampled label following Hewitt and Liang (2019), inspect probe weight norms around perfor- mance peaks, and evaluate sensitivity to regulariza- tion. We further repeat the analysis on OLMo-7B (Groeneveld et al., 2024), an English-centric model, to assess whether the cross-lingual depth profile observed in the multilingual models also appears under substantially weaker multilingual support. 4.2 Results Stereotype-related information is most decod- able in mid-layers. All three multilingual mod- 0 .5 1 53% of depth L15L32 Llama-3.1-8B decodability (probe macro F1) 0 .5 1 Relative strength (each curve peaks at 1) 36% of depth L23L36 Qwen3-8B causal influence (|Attr|) solid: stereotype routing dashed: evidence injection 0255075100 Depth (% of layers) input → output 0 .5 1 50% of depth L21L42 Gemma-2-9B Figure 2: Peak decodability precedes peak estimated output influence by36–53%of model depth. Probe macro-F1 (blue) and normalized attribution (red) share a depth axis, one panel per model. Both are scaled to their own maximum, since the claim is about where each peaks. Decodability is scaled so that chance sits at 0. Markers give each curve’s peak, and the arrow measures the distance between them. Detailed results are in Appendices C and D. els share similar depth profile (Figure 2, blue): in Llama, macro-F1 rises from around layer 8, peaks at layer 15, then declines toward the output, with peak macro-F10.845averaged over the 24 (cate- gory, language) cells. The same profile holds for Qwen and Gemma, with the peak shifted to layers 21–23 (Macro-F10.894and0.899; Appendix C, Figures 4 and 5) respectively. Peak performance varies more across social categories than across lan- guages: SES is consistently highest, while sexual orientation is lowest. Control probes remain at chance. Permuted- label probes remain at chance across depth (mean 0.494–0.498). Stronger regularization lowers over- all probe performance without moving the peak, and probe weight norms do not show a correspond- ing spike around the peak layer (Appendix C, Fig- ures 6 and 7). The same depth profile is weaker in the English- centric model. The same profile is weaker in OLMo-7B (Appendix C, Figure 4). Its peak macro- F1 is0.606and is substantially more concentrated on English (0.757compared with0.556averaged across Spanish, Dutch, and Turkish). We use this comparison as an English-centric reference point rather than to attribute the difference to a particular training factor. 5 Where Does Stereotype Information Influence the Output? To quantify where stereotype-related completion conditions most strongly influence the prediction, we use attribution patching (Nanda, 2023; Syed et al., 2024). 5.1 Attribution Patching Setup For each MBBQ example, we construct a clean input (expected to produce the correct answer) and a corresponding corrupted input (expected to elicit stereotyped behaviour), run both through the model via TransformerLens (Nanda and Bloom, 2022), and cache the residual-stream output (resid_post) at every layer. On the corrupted run, we score each example by the logit marginmbetween the correct option and its strongest competing option. The attribution of layer ℓ is approximated as Attr(ℓ)≈∇ a corr ℓ m· a clean ℓ − a corr ℓ wherea clean ℓ anda corr ℓ denote the corresponding residual-stream activations. This requires one back- ward pass through the corrupted run and one addi- tional forward pass to cache the clean activations. We use two complementary clean–corrupted contrasts. Evidence injection compares a disam- biguated input containing factual evidence with the corresponding ambiguous input, testing where the added evidence changes the model’s prediction. Stereotype routing contrasts counter-stereotypical and pro-stereotypical targets under disambiguated contexts, testing where internal states differentiate between the two prediction directions. We aggregate attribution scores for each (layer, category, language) combination and summarize their depth profiles using trapezoidal integration (single AUC). Our comparison with probing fo- cuses on the locations of maximal decodability and maximal attribution, rather than treating the onset of either signal as equivalent to causal use. 5.2 Results Peak decodability precedes peak output influ- ence by 36–53% of model depth. Across all three models, the probe peak occurs substantially earlier than the attribution peak. The separation is 53% for Llama, 48% for Gemma, and 36% for Qwen. These values compare peak locations, not the first layers at which either signal appears. Output influence peaks around output-layers. Attribution follows a different depth profile (Fig- ure 2, red). In Llama, attribution becomes de- tectable in the middle layers and increases toward the output, with mean absolute attribution reach- ing0.394at the final layer compared with0.237 averaged across layers. Qwen and Gemma show the same late build-up and the same final-layer peak (Appendix D, Figures 8 and 9). The exact onset varies across models (layer 11 in Llama and layer 17 in Qwen and Gemma), but maximal attri- bution occurs at the final layer in all three models. Stereotype routing (counter- vs. pro-stereotype) yields consistently negative late-layer attributions, consistent with these layers shifting the prediction away from the stereotype-consistent answer. 6 What Do the Selected SAE Features Represent? To identify sparse features associated with stereotype-related completions, we decompose model activations using pre-trained SAEs. We use lexical anchors and available feature glosses to de- scribe the selected features. Their behavioral ef- fects are measured separately through ablation in (§7). 6.1 Feature Extraction Setup We use four pre-trained SAE suites: English- trained SAEs on Llama-3.1-8B (Llama-Scope, He et al., 2024), English-trained SAEs on Gemma-2- 9B (Gemma-Scope, Lieberum et al., 2024), multilin- gual SAEs on Qwen3-8B (Qwen-Multi) 1 , and mul- tilingual SAEs on Llama-3.1-8B (Llama-Multi, Al Ghussin et al., 2026). The suites differ in train- ing data, sparsity, dictionary configuration, and base model (Table 2; full details in Appendix E). The semantic-family analysis usesLlama-Scope because Neuronpedia (Lin, 2023) glosses are only available for a subset of its retained features. Lex- ical anchoring and feature ablation are compared across all four suites. Because the suites differ along several dimensions simultaneously, differ- 1 https://huggingface.co/Yusser/MULTI21-SAES-Q wen3-8B_512_2100000000 SAE suiteBase modelTrainingL 0 Llama-Scope Llama-3.1-8B English50 Llama-Multi Llama-3.1-8B Multi. ≈ 10,827 Gemma-Scope Gemma-2-9BEnglish ≈ 158 Qwen-Multi Qwen3-8BMulti. ≈ 26,893 Table 2: SAE suites used in the feature-level analysis. Llama-ScopeandLlama-Multishare a base model; the four suites still differ in several other properties. Dic- tionary widths and training-token counts are reported in Appendix E. ences between them cannot be assigned to training language or sparsity alone. Contrastive feature selection.A feature that re- sponds to a demographic concept is not necessar- ily involved in stereotype-related behaviour. We therefore identify candidate features through a con- trastive construction that holds the context and de- mographic entities fixed while varying the com- pletion. For each MBBQ ambiguous context, we extract SAE feature activations at the answer to- ken corresponding to both the stereotype-consistent completionx stereotype (e.g., grandfather) and the control completion x control (e.g., grandson). We then score each featurefby the separation betweenf (x stereotype )andf (x control )across pairs, consistent with the contrastive concept-split eval- uation of Härle et al. (2025). This contrast de- emphasizes features that respond similarly to both completions, including features that primarily en- code the shared demographic content. We retain the topk=3features per (category, layer, stream) for downstream analysis, which re- sults in1,741retained features forLlama-Scope, 1,841forLlama-Multi,2,445forGemma-Scope, and 738 for Qwen-Multi 2 . Lexical anchoring.Feature selection establishes differential activation, but does not by itself tell us what the retained directions represent. We there- fore characterize their alignment with the model’s output vocabulary. For each feature direction, we compute its cosine similarity with every row of the unembedding matrix and define the token with maximum similarity as its lexical anchor. We refer to the corresponding maximum cosine similarity as the anchor cosine. 2 The anchor analysis (§6) uses all retained features, the ab- lation sweep (§7) uses the 535 residual-stream Llama-Scope features that completed it, and the family analysis (Table 3) uses the139 Llama-Scopefeatures for which Neuronpedia supplies a gloss. To determine whether this alignment exceeds what is expected from arbitrary directions, we com- pare each anchor cosine against a model-specific null distribution obtained from random unit direc- tions. We use the99th percentile of this distri- bution as the chance threshold:0.078for both Llama suites,0.080forQwen-Multi, and0.086 forGemma-Scope. The level depends ond model and the tokenizer, not on features. The two Llama SAE suites share the same base model, but differ in their SAE training and configuration. This method of interpreting an intermediate direction through the unembedding aligns with the logit-lens family of analyses (nostalgebraist, 2020; Geva et al., 2022). ForLlama-Scope, we additionally use available Neuronpedia glosses to examine whether retained features form coherent semantic families. We re- strict this analysis to the 139 retainedLlama-Scope features for which a gloss is available. 6.2 Results Among the 139 retainedLlama-Scopefeatures with available Neuronpedia glosses, 123 (88.5%) are on-concept for the social category on which they were selected (Table 3). The features fur- ther form recurring semantic families: disability- status features separate into neurodevelopmental, psychiatric, and chronic-illness concepts, while gender-identity features separate into pronouns, group nouns, and kinship roles. Lexical anchoring depends on the SAE suite. Lexical alignment varies substantially across SAE suites. Of the retainedLlama-Scopefeatures, 46.9%of1,741exceed the0.078chance level against8.8%of1,841 Llama-Multifeatures. In the residual stream,57.9%against11.1%. The same split holds across base models, with41.7%of 2,445 Gemma-Scopefeatures above its own0.086 level, and0.8%of738 Qwen-Multifeatures above its0.080level (See Appendix F, and Figure 10). Because the suites differ in sparsity, training data, and dictionary configuration, these differences can- not be attributed to training language alone. Anchoring is concentrated toward later lay- ers and, forLlama-ScopeandGemma-Scope, is strongest in the residual stream. InLlama-Scope, the fraction of anchored features rises from4.8% at layer 11 to74.7%at layer 31, exceeding50%by layer 20. Detailed layer- and stream-level compar- isons are reported in Appendix F. CategoryFeature family#Depth (layer 0–31)Representative lexical anchors Age (20) Chronological age & generational change611–23Age, years, -age, -aged Seniority & later life6 12–27Senior, Sen, _old, Minor Childhood & child welfare618–30adult, -ad, blood Disability status (37) Autism & neurodevelopment317–18communication, Bel, 035 Schizophrenia & psychosis816–31Sch, Dep, -di, enza Depression, anxiety & mental health1421–31Dep, mental, mad Chronic illness & accessibility1113–31-friendly, -related, Down, Multiple Gender identity (38) Gendered pronouns811–28he, him, his, she, her Gender group nouns1412–29Women, Men, Female, -girl Kinship & family roles1212–31mother, father, woman Physical appearance (20) Weight & body size1011–31Weight, BMI, lbs, Thin, -ob Face, vision & stature614–26face, Eye, Ret, _short SES (8) Class-position labels511–18Working, -middle, _low, Upper Affordability & labour211–14requ, low Sexual orientation (16) LGBTQ+ identity terms10 19–31-gay, Trans, trans, Bi Heteronormative contrast & pride223–31straight, Par all categoriesoff-concept / polysemantic1613–31OLD (code), _bi (history) Table 3: Bias features group into a small number of interpretable families per category (Llama-3.1-8B, Llama-Scope; 139 retained features across the residual, MLP and attention streams). The full per-feature list is available at https://github.com/ariunerdenetum/stereotype-tracing-mllm. 7 What Are the Effects and Transferability of Feature Ablation? To measure how the selected features affect stereotype-related behavior, we ablate them indi- vidually and measure the resulting change in bias on SHADES (Mitchell et al., 2025). 7.1 Feature Ablation Setup Given a target feature, we apply feature-zero mask- ing in the SAE’s encoded basis. We encode the residual-stream activationa, set the activation of featurefto zero using a binary maskm f , and decode the modified representation back into the residual stream. ˆa = Dec Enc(a)⊙ m f This intervention is applied during inference. Model and the SAEs remain unchanged. We evaluate the intervened model on SHADES (Mitchell et al., 2025) using its base-model bias score taking inspiration from Nangia et al. (2020). We compute the intervention effect separately for each feature–bias-type pair as ∆bias(f ) = bias interv (f )− bias baseline , where negative values indicate a reduction in mea- sured bias and positive values an increase. We do not assume that ablating a selected fea- ture should reduce bias. Contrastive selection estab- lishes differential activation, but does not determine the direction of a feature’s causal effect. A feature may promote stereotype-consistent behavior or in- stead support a counter-stereotypical association. We therefore retain intervention effects in both di- rections. Statistical analysis. We assess per-feature inter- vention effects using one-samplet-tests with boot- strap confidence intervals and additionally report Cohen’sd. We classify0.5 ≤ |d| < 0.8as a medium effect and|d| ≥ 0.8as a large effect; ag- gregate results for each SAE suite are reported in Table 4. Cross-lingual and cross-category transfer.We next ask whether a feature’s intervention effect is stable across languages and social categories. For each feature, we define a Language Speci- ficity Score (LSS) and a Category Specificity Score (CSS): LSS = σ(∆bias langs ) |∆bias langs | + ε , CSS = σ(∆bias cats ) |∆bias cats | + ε . We setε = 10 −8 and use0.5as the threshold for both scores. These scores measure the cross- language (resp. cross-category) variation in a fea- language-agnosticlanguage-dependent Llama-Scope n=98 / n=437 enes nl 0 −20 −40 −60 −80 −46 −61 −79 −32 −36 −55 Change in bias (%) baseline: en .277 es .078 nl .063 Llama-Multi n=36 / n=463 enes nl 00 +1 0 −1 0 baseline: en .275 es .078 nl .062 Gemma-Scope n=42 / n=702 enes nl +1 −5 +8 −11−11 +12 baseline: en .346 es .270 nl .173 Qwen-Multi n=44 / n=658 en nl es +1 +2 00 −1 −2 baseline: en .268 nl .118 es .112 Figure 3: Cross-SAE comparison of per-language ablation effects. Language-agnostic features have larger mean effects in Llama-Scope, but this pattern does not repeat across the other SAE suites. Transfer (LSS/CSS)Global intervention effects SAE suiten Lang. -agn. Cat. -agn. avg ∆bias Cohen’s d % reduc- ing bias Sig. reductions Best ∆bias Large|d|Med.|d| Llama-S535 98 (18%)0 −.0553 −.070962.5 1,083 (9.6%) −1.0051,006 (9.0%)1,007 (9.0%) Gemma-S 74442 (6%)0 −.0156 −.018158.1482 (3.1%) −1.312 1,632 (10.4%)1,417 (9.1%) Qwen-M70244 (6%)0 −.0008 −.012850.6635 (4.3%) −0.123 1,722 (11.7%) 1,529 (10.4%) Llama-M 49936 (7%)0 +.0002 +.007543.4193 (1.8%) −0.052711 (6.8%) 1,205 (11.5%) Table 4: Cross-language/category transfer and global intervention effects, by SAE suite.nis the number of evaluated residual-stream features; Lang.-agn. and Cat.-agn. give the count (and %) that are language- or category- agnostic under LSS/CSS respectively – between6–18%are language-agnostic, none are category-agnostic. Under global ablation,Llama-Scopeshows by far the strongest bias reduction; the two multilingual suites are close to zero. Summing the Large and Med.|d|columns gives the18–20%of feature–bias-type pairs with medium-to-large effects quoted in §7 (17.9% and 19.5% respectively). ture’s∆biasrelative to its mean absolute effect. Features with LSS or CSS< 0.5are agnostic on that axis (stable across languages or categories), whereas features with LSS or CSS≥ 0.5are de- pendent. Because low specificity can also occur when intervention effects are consistently small, we in- terpret LSS and CSS together with the underlying ∆bias values. 7.2 Results Intervention effects.We find that individual fea- ture ablations produce heterogeneous and bidirec- tional changes in measured bias.Llama-Scope shows the largest average effect (∆bias =−0.055, d =−0.071), with9.6%of feature–bias-type pairs yielding nominally significant reductions and ap- proximately18–20%reaching a medium or large effect (Table 4). For disability status, ablation shifts bias toward the stereotype inLlama-Scope (∆bias = +0.280), away from it inGemma-Scope (−0.090), and has approximately zero mean effect in the two multilingual SAE suites. Differential activation therefore does not imply that a selected feature promotes stereotypical behavior. Layer depth does not predict the signed effect (R 2 < 0.0016), and anchor cosine is at best weakly predictive (nominally significant but negligible in size). Depth does weakly track effect magnitude (r =−0.190for agnostic and−0.116for specific features, Table 7). The effect of removing a feature therefore has to be measured rather than inferred from its layer or lexical anchor. Average effects differ across SAE suites. The mean ablation effect is largest forLlama-Scope (−0.055) and smaller forGemma-Scope(−0.016). Qwen-Multi(−0.0008)andLlama-Multi (+0.0002) are approximately zero on average (Table 4). The feature-level intervention results are therefore not equally strong across the four decompositions. Transfer across languages and categories.The intervention effects show a clear asymmetry be- tween language and social category (Appendix G, Figure 13). Of the 535 ablatedLlama-Scopefea- tures, all are category-dependent (CSS≥ 0.5), whereas18.3%are language-agnostic (LSS< 0.5). The same pattern holds across all four SAE suites: 6–18%of residual-stream features are language- agnostic, while no category-agnostic features are observed under the CSS criterion (Table 4). Thus, stable effects are more common across languages than across social categories, but they remain a minority. Transfer does not imply stronger intervention. InLlama-Scope, this language-agnostic minority also has larger mean intervention effects, reduc- ing measured bias by46–79%across evaluation languages compared with32–55%for language- dependent features (Figure 3). This advantage does not replicate across SAE suites: it is absent inLlama-MultiandQwen-Multiand reverses for English and Spanish inGemma-Scope(Figure 3). The larger effects of language-agnostic features in Llama-Scopedo not replicate across the other SAE suites. 8 Discussion and Conclusion We studied how stereotype-related behavior in mul- tilingual LLMs relates across linear decodability, estimated output influence, and feature-level in- tervention. 1 For decodability and output influ- ence, probe performance peaks substantially earlier than attribution across Llama, Qwen, and Gemma, with the two peaks separated by 36–53% of model depth. 2 For feature-level intervention, selected Llama-Scope features are often semantically re- lated to the social category on which they were identified, but their ablation effects vary consider- ably in both magnitude and direction.3For cross- lingual transfer, 6–18% of evaluated residual- stream features have language-agnostic interven- tion effects under our criterion, while none are category-agnostic. The larger effects of language- agnostic features in Llama-Scope do not repeat across the other SAE suites. Across these analyses, the same distinction emerges: where stereotype- related information is most decodable, where it most strongly influences an output, and what hap- pens when a selected feature is removed are related but different properties of model behavior. Local- ization should therefore be treated as a starting point for intervention analysis, rather than as evi- dence that an intervention will have the intended effect. Limitations Our study has several limitations. Firstly, our anal- ysis is restricted to four high-resource languages (English, Spanish, Dutch, Turkish) from three fam- ilies, as the pipeline relies on the MBBQ resource. Furthermore, the SHADES benchmark covers 16 languages but not Turkish; we evaluate feature in- terventions in only three of our four languages. Whether the decodability–causality separation and the category-specificity of bias features hold in ty- pologically diverse or lower-resource languages remains unexamined. Secondly, we evaluate only 8B–9B parameter models, limiting assessment of how these patterns may scale with model size. Thirdly, the SAE suites utilized differ by dictionary size, sparsity, training data, and language cover- age, which complicates our comparison since SAE quality and multilingual training are confounded (Appendix E); the ablation effects are strongest in Llama-Scope, suggesting our effect sizes are partially replicated rather than universally appli- cable. Fourthly, feature ablation measures causal relevance but does not account for redundancy in bias pathways. Lastly, we test our bias features individually, relative to MBBQ and SHADES, leav- ing intersectional biases (e.g., age×gender) unad- dressed. Ethics Statement This work analyzes internal representations of so- cial stereotypes in multilingual LLMs across six sensitive categories: age, disability, gender iden- tity, physical appearance, socioeconomic status, and sexual orientation. We use only publicly available benchmarks (MBBQ, SHADES) and re- leased model weights; no human data was collected. Given the sensitive nature of these categories, we frame our analysis strictly as a diagnostic tool for detecting and mitigating bias, not as a character- ization of groups. The feature analyses and visu- alizations are presented for interpretability, not to endorse the associations they show. Acknowledgment KDC is supported by the Deutsche Forschungsge- meinschaft (DFG, German Research Foundation) – SFB 1102 Information Density and Linguistic Encoding. YAG and JVG are supported by the German Federal Ministry of Research, Technology and Space (BMFTR) under the TRAILS project (01IW24005). References Yusser Al Ghussin, Daniil Gurgurov, Tanja Baeumel, Josef van Genabith, Patrick Schramowski, and Simon Ostermann. 2026. Multilingual steering by design: Multilingual sparse autoencoders and principled layer selection. In Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026), pages 364–401, San Diego, California. Association for Computa- tional Linguistics. Guillaume Alain and Yoshua Bengio. 2018. Under- standing intermediate layers using linear classifier probes. Preprint, arXiv:1610.01644. Dana Arad, Aaron Mueller, and Yonatan Belinkov. 2025. SAEs are good for steering – if you select the right features. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP). ArXiv:2505.20063. Aryaman Arora, Dan Jurafsky, and Christopher Potts. 2024.CausalGym: Benchmarking causal inter- pretability methods on linguistic tasks. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 14638–14663, Bangkok, Thailand. As- sociation for Computational Linguistics. Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L. Griffiths. 2025.Explicitly unbiased large language models still form biased associations. Proceedings of the National Academy of Sciences, 122(8):e2416228122. Mukul Bhutani, Kevin Robinson, Vinodkumar Prab- hakaran, Shachi Dave, and Sunipa Dev. 2024. SeeG- ULL multilingual: a dataset of geo-culturally situated stereotypes. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 842–854, Bangkok, Thailand. Association for Computational Linguistics. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. Towards monosemanticity: Decompos- ing language models with dictionary learning. Trans- former Circuits Thread. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. Preprint, arXiv:2309.08600. Boyi Deng, Yu Wan, Baosong Yang, Yidan Zhang, and Fuli Feng. 2025. Unveiling language-specific fea- tures in large language models via sparse autoen- coders. In Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4563–4608, Vienna, Austria. Association for Computational Linguistics. Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2025. Separat- ing tongue from thought: Activation patching reveals language-agnostic concept representations in trans- formers. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 31822–31841, Vienna, Austria. Association for Computational Linguistics. Lance Calvin Lim Gamboa, Yue Feng, and Mark G. Lee. 2025. Social bias in multilingual language models: A survey. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 27857–27880, Suzhou, China. Association for Computational Linguistics. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. Preprint, arXiv:2406.04093. Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Gold- berg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 30–45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mi- tra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khy- athi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, and 24 others. 2024. OLMo: Accelerating the science of language mod- els. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 15789–15809, Bangkok, Thailand. Association for Computational Linguistics. Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. 2024. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. Preprint, arXiv:2410.20526. John Hewitt and Percy Liang. 2019. Designing and in- terpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, Hong Kong, China. Association for Computational Linguistics. Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. 2024. RAVEL: Evaluating interpretability methods on disentangling language model representations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8669– 8687, Bangkok, Thailand. Association for Computa- tional Linguistics. Ruben Härle, Felix Friedrich, Manuel Brack, Stephan Wäldchen, Björn Deiseroth, Patrick Schramowski, and Kristian Kersting. 2025. Measuring and guiding monosemanticity. Preprint, arXiv:2506.19382. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 278–300, Miami, Florida, US. Association for Computational Linguistics. Johnny Lin. 2023. Neuronpedia: Interactive reference and tooling for analyzing neural networks. Software available from neuronpedia.org. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2023. Locating and editing factual associa- tions in gpt. Preprint, arXiv:2202.05262. Margaret Mitchell, Giuseppe Attanasio, Ioana Bal- dini, Miruna Clinciu, Jordan Clive, Pieter Delobelle, Manan Dey, Sil Hamilton, Timm Dill, Jad Dough- man, Ritam Dutt, Avijit Ghosh, Jessica Zosa Forde, Carolin Holtermann, Lucie-Aimée Kaffee, Tanmay Laud, Anne Lauscher, Roberto L Lopez-Davila, Maraim Masoud, and 35 others. 2025. SHADES: Towards a multilingual assessment of stereotypes in large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chap- ter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa- pers), pages 11995–12041, Albuquerque, New Mex- ico. Association for Computational Linguistics. Neel Nanda. 2023. Attribution patching: Activation patching at industrial scale.https://w.neelna nda.io/mechanistic-interpretability/attr ibution-patching. Neel Nanda and Joseph Bloom. 2022. Transformerlens. https://github.com/TransformerLensOrg/Tr ansformerLens. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A chal- lenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. As- sociation for Computational Linguistics. Vera Neplenbroek, Arianna Bisazza, and Raquel Fer- nández. 2024. MBBQ: A dataset for cross-lingual comparison of stereotypes in generative LLMs. In First Conference on Language Modeling. Vera Neplenbroek, Arianna Bisazza, and Raquel Fer- nández. 2025. Reading between the prompts: How stereotypes shape llm’s implicit personalization. Preprint, arXiv:2505.16467. nostalgebraist. 2020. Interpreting GPT: The logit lens. LessWrong. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dublin, Ireland. Association for Computational Linguistics. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011.Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. Qwen Team. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. 2021. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceed- ings of the 16th Conference of the European Chap- ter of the Association for Computational Linguistics: Main Volume, pages 3363–3377, Online. Association for Computational Linguistics. Katharina Simbeck and Mariam Mahran. 2025. Mech- anistic interpretability with saes: Probing religion, violence, and geography in large language models. arXiv preprint arXiv:2509.17665. Xiangchen Song, Aashiq Muhamed, Yujia Zheng, Lingjing Kong, Zeyu Tang, Mona T. Diab, Virginia Smith, and Kun Zhang. 2026. Mechanistic inter- pretability should prioritize feature consistency in sparse autoencoders. In Proceedings of the 64th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2172– 2210, San Diego, California, United States. Associa- tion for Computational Linguistics. Aaquib Syed, Can Rager, and Arthur Conmy. 2024. Attribution patching outperforms automated circuit discovery. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Net- works for NLP, pages 407–416, Miami, Florida, US. Association for Computational Linguistics. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179 others. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. Ariun-Erdene Tumurchuluun, Yusser Al Ghussin, David Mare ˇ cek, Josef van Genabith, and Koel Dutta Chowd- hury. 2025. TenseLoC: Tense localization and con- trol in a multilingual LLM. In Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025), pages 243–264, Suzhou, China. Associ- ation for Computational Linguistics. Mingyang Wang, Heike Adel, Lukas Lange, Yihong Liu, Ercong Nie, Jannik Strötgen, and Hinrich Schuetze. 2025. Lost in multilinguality: Dissecting cross- lingual factual inconsistency in transformer language models. In Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5075–5094, Vienna, Austria. Association for Computational Linguistics. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in English? on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15366–15394, Bangkok, Thai- land. Association for Computational Linguistics. Haeun Yu, Seogyeong Jeong, Siddhesh Pawar, Jisu Shin, Jiho Jin, Junho Myung, Alice Oh, and Isabelle Augen- stein. 2025. Entangled in representations: Mechanis- tic investigation of cultural biases in large language models. arXiv preprint arXiv:2508.08879v2. Chenye Zou, Difan Jiao, and Lijie Hu. 2026. Decipher- ing cultural representations in large language models via sparse autoencoders. In Findings of the Associa- tion for Computational Linguistics: ACL 2026, pages 5656–5677, San Diego, California, United States. Association for Computational Linguistics. A Use of AI Assistants AI assistants were used for coding and paper edit- ing. All scientific claims, experimental design, and analysis were conducted by the authors. B Appendix Overview The appendix follows the four stages of the pipeline in the same order as the main text (Table 5). App.StageContents CLinear ProbingResults in detail DAttribution Patching Pairing A/B grids per model ESAE configsComparison of pre-trained SAEs FFeature extractionAnchor-strength heatmap GAblationResults in detail Table 5: Map of the detailed results. C Linear Probing (a) Probe macro-F1 by layer, averaged over the six categories and four languages. 0816243240 Layer 0.4 0.6 0.8 1.0 Macro F1 control probes (permuted labels) OLMo-7B Llama-3.1-8B Gemma-2-9B Qwen3-8B decodable here (layers 15–23) causally used here (b) Peak macro-F1 per category and language, each panel shaded in its model’s colour from (a). enesnltr SES Age Gender Appearance Disability Sexual orient. 1.00.98.99.99 .98.94.92.95 .95.75.83.76 .95.87.87.86 .91.85.82.75 .65.59.56.56 Llama-3.1-8B enesnltr 1.00.991.00.99 .97.96.95.95 .95.84.88.75 .91.87.94.88 .94.93.87.77 .78.84.72.78 Qwen3-8B enesnltr SES Age Gender Appearance Disability Sexual orient. 1.00.991.00.98 .99.98.97.94 .96.85.90.85 .94.89.91.89 .93.94.90.82 .75.61.88.69 Gemma-2-9B enesnltr .96.88.82.72 .90.79.77.66 .82.60.42.44 .69.56.51.42 .77.56.45.38 .41.31.31.41 OLMo-7B Figure 4: Stereotype information is decodable from mid-layer, well before the layers that causally drive the output. Panels (a) and (b) give the probing curves that Figure 2 summarizes. 0.25 0.50 0.75 1.00 SES Llama-3.1-8BQwen3-8BGemma-2-9BOLMo-7B 0.25 0.50 0.75 1.00 Age 0.25 0.50 0.75 1.00 Gender 0.25 0.50 0.75 1.00 Appearance 0.25 0.50 0.75 1.00 Disability 08162432 Layer 0.25 0.50 0.75 1.00 Sexual orient. 08162432 Layer 0816243240 Layer 08162432 Layer EnglishSpanishDutchTurkishcontrol probes (permuted labels) Figure 5: Probe macro-F1 by layer for every social category (rows), model (columns), and language (line colour). The grey band is the envelope of the four control probes trained on permuted labels; the dashed horizontal line marks chance. Real curves separate from the control band from roughly layer 8 onward in the three multilingual models, and stay inside or barely above it for OLMo-7B outside English. Sexual orientation (bottom row) is the one category where several language–model combinations hug the control band. 0 10 20 SES Llama-3.1-8B 0 10 20 Qwen3-8B 0 10 20 Gemma-2-9B 0 10 20 OLMo-7B 0 10 20 Age 0 10 20 10 20 0 10 20 0 5 10 Gender 5 10 5 10 0 5 10 5 10 15 Appearance 5 10 15 5 10 15 5 10 5 10 15 Disability 5 10 15 5 10 15 5 10 15 08162432 Layer 0 2 4 Sexual orient. 08162432 Layer 0 2 4 0816243240 Layer 0 2 4 08162432 Layer 0 2 4 EnglishSpanishDutchTurkishmacro-F1 peak Figure 6: Probe weightℓ 2 norm by layer, same grid as Figure 5. The dashed vertical line marks that panel’s macro-F1 peak layer. Norms rise with depth and then flatten; they do not spike where performance peaks, which is what rules out the reading that peak decodability is a high-capacity probe fitting noise. 08162432 Layer 0.4 0.6 0.8 1.0 Macro F1 C= 1.0 C= 0.1 Llama-3.1-8B 08162432 Layer Qwen3-8B 0816243240 Layer Gemma-2-9B 08162432 Layer OLMo-7B Figure 7: Regularization sensitivity: mean macro-F1 by layer atC=1.0(solid) andC=0.1(dashed), averaged over categories and languages. Stronger regularization lowers the curve uniformly (by0.04–0.07) without moving the peak, so the depth profile is a property of the representation rather than of the probe’s capacity. D Attribution Patching 0.4 0.3 0.2 0.1 0.0 0.1 Age 4 3 2 1 0 0.15 0.10 0.05 0.00 0.05 0.15 0.10 0.05 0.00 0.05 Disability status 3 2 1 0 0.4 0.3 0.2 0.1 0.0 0.1 0.0 0.1 0.2 Gender identity 4 3 2 1 0 0.15 0.10 0.05 0.00 0.05 0.3 0.2 0.1 0.0 0.1 Physical appearance 3 2 1 0 0.20 0.15 0.10 0.05 0.00 0.05 0.10 0.4 0.2 0.0 0.2 SES 8 6 4 2 0 0.4 0.3 0.2 0.1 0.0 0102030 Layer 0.1 0.0 0.1 Sexual orientation 0102030 Layer 2.5 2.0 1.5 1.0 0.5 0.0 010203040 Layer 0.2 0.1 0.0 LlamaQwenGemma EnglishSpanishDutchTurkish Figure 8: Causal intervention pairing A. 0.4 0.3 0.2 0.1 0.0 0.1 Age 4 3 2 1 0 0.15 0.10 0.05 0.00 0.05 0.15 0.10 0.05 0.00 0.05 Disability status 3 2 1 0 0.4 0.3 0.2 0.1 0.0 0.1 0.0 0.1 0.2 Gender identity 4 3 2 1 0 0.15 0.10 0.05 0.00 0.05 0.3 0.2 0.1 0.0 0.1 Physical appearance 3 2 1 0 0.20 0.15 0.10 0.05 0.00 0.05 0.10 0.4 0.2 0.0 0.2 SES 8 6 4 2 0 0.4 0.3 0.2 0.1 0.0 0102030 Layer 0.1 0.0 0.1 Sexual orientation 0102030 Layer 2.5 2.0 1.5 1.0 0.5 0.0 010203040 Layer 0.2 0.1 0.0 LlamaQwenGemma EnglishSpanishDutchTurkish Figure 9: Causal intervention pairing B. E SAE Configs SAEsBase modelLayersTraining languagesDictionary widthSparsityTraining tokens Llama-SLlama-3.1-8B32English32,768Top-K (L 0 =50)800M Gemma-S Gemma-2-9B42English16,384JumpReLU (L 0 ≈158)4B Qwen-MQwen3-8B36multilingual32,768JumpReLU (L 0 ≈26893)2.1B Llama-MLlama-3.1-8B32multilingual32,768JumpReLU (L 0 ≈10827)5.77B Table 6: The SAE suites are off-the-shelf models not trained by us, leading to variability in comparisons due to factors such as base model, dictionary width, sparsity, activation site coverage, and training data language composition and volume. Streams used: resid_post, mlp_out, attn_out, every layer. F Feature Extraction 1113151719212325272931 SES Age Gender Appearance Disability Sexual orient. Llama-Scope (row means span 47-69%) 11131517192123252729313335373941 Gemma-Scope (row means span 57-79%) 11131517192123252729313335 Layer SES Age Gender Appearance Disability Sexual orient. Qwen-Multi (row means span 0-2%) 1113151719212325272931 Layer Llama-Multi (row means span 7-13%) 0 25 50 75 100 Features above chance (%) Figure 10: Lexical anchoring of the retained SAE features (residual stream). Each cell is the percentage of retained features in that (social category, layer) slice whose anchor cosine exceeds its own suite’s chance level. Cells at or below the 1% expected under chance are outlined in red. G Ablation AgnosticSpecific Llama-Scope 0.511.5 0 20 40 60 Number of features n =98 /n =437 0.511.5 n =0 /n =535 Gemma-Scope 0.511.5 0 20 40 60 80 n = 42 / n = 702 0.511.5 n = 0 / n = 744 Qwen-Multi 0.511.52 0 20 40 60 LSS Number of features n =44 /n =658 0.511.52 CSS n =0 /n =702 Llama-Multi 0.511.52 0 20 40 60 80 LSS n =36 /n =463 0.511.52 CSS n =0 /n =499 Figure 11: Distributions of the Language Specificity Score (LSS, blue) and the Category Specificity Score (CSS, red) across SAE features, for each SAE suite. Dashed lines mark the0.5threshold; panel headings give the number of agnostic (< 0.5) and specific (≥ 0.5) features. Llama-Scope enesnl tr nl es en −0.139−0.046−0.051 −0.134−0.047−0.050 −0.135−0.047−0.050 −0.131−0.048−0.049 Feature language Language-agnostic LSS < 0.5, n = 98 enesnl −0.096−0.030−0.035 −0.095−0.031−0.036 −0.094−0.030−0.034 −0.085−0.027−0.035 Language-specific LSS≥ 0.5, n = 437 −0.1400.14 Mean bias ∆ Gemma-Scope enesnl tr nl es en −0.002−0.011 0.016 −0.011−0.006 0.018 −0.005−0.009 0.017 −0.001−0.012 0.016 Feature language Language-agnostic LSS < 0.5, n = 42 enesnl −0.044−0.030 0.021 −0.044−0.030 0.020 −0.038−0.031 0.021 −0.039−0.030 0.021 Language-specific LSS≥ 0.5, n = 702 −0.04500.045 Mean bias ∆ Qwen-Multi enesnl tr nl es en 0.0030.0000.002 0.0030.0000.002 0.0030.0000.002 0.0030.0000.002 Feature language Language-agnostic LSS < 0.5, n = 44 enesnl 0.001 −0.002−0.001 0.001 −0.002−0.001 0.001 −0.002−0.001 0.001 −0.002−0.001 Language-specific LSS≥ 0.5, n = 655 −0.00300.003 Mean bias ∆ Llama-Multi enesnl tr nl es en 0.0000.0000.000 0.0000.0000.001 −0.001 0.0000.000 0.0000.0000.001 Feature language Language-agnostic LSS < 0.5, n = 36 enesnl 0.001 −0.001 0.000 0.001 −0.001 0.000 0.001 −0.001 0.000 0.0010.0000.000 Language-specific LSS≥ 0.5, n = 459 −0.00100.001 Mean bias ∆ Figure 12: Language-transfer matrices split by LSS group; panel headings give the split and the feature count. Rows: discovery language, columns: evaluation language, cells: mean bias change ∆. Colour scales are per model. Llama-Scope enesnl Sex.Or. SES Appear. Gender Disab. Age −0.1149−0.0798−0.0964 −0.1066−0.0842−0.0908 −0.1169−0.0823−0.0954 −0.1037−0.0786−0.0906 −0.1101−0.0817−0.0933 −0.1088−0.0783−0.0937 Discovery category Language-agnostic LSS < 0.5, n = 98 enesnl −0.0356−0.0026−0.0781 −0.0070 0.0052 −0.0801 −0.0303 0.0015 −0.0770 −0.0176−0.0004−0.0824 −0.0334−0.0035−0.0829 −0.0266−0.0009−0.0791 Language-specific LSS≥ 0.5, n = 437 −0.1200.12 Mean bias ∆ Gemma-Scope enesnl Sex.Or. SES Appear. Gender Disab. Age −0.0047−0.0226−0.0104 0.00470.00060.0101 0.00530.00070.0116 0.0065 −0.0004 0.0060 −0.0082−0.0335−0.0211 −0.0059−0.0297−0.0172 Discovery category Language-agnostic LSS < 0.5, n = 42 enesnl 0.0044 −0.0640 0.0236 0.0055 −0.0619 0.0264 0.0037 −0.0632 0.0241 0.0011 −0.0673 0.0156 0.0085 −0.0695 0.0193 0.0057 −0.0632 0.0223 Language-specific LSS≥ 0.5, n = 702 −0.0700.07 Mean bias ∆ Qwen-Multi enesnl Sex.Or. SES Appear. Gender Disab. Age 0.00240.00210.0006 0.00430.00360.0037 0.00530.00540.0052 0.00320.00320.0034 0.00290.00360.0025 0.00190.00180.0012 Discovery category Language-agnostic LSS < 0.5, n = 44 enesnl −0.0015 0.0008 −0.0014 −0.0011 0.0007 −0.0019 −0.0016 0.0007 −0.0013 −0.0018 0.0009 −0.0019 −0.0012 0.0007 −0.0018 −0.0014 0.0011 −0.0016 Language-specific LSS≥ 0.5, n = 655 −0.005500.0055 Mean bias ∆ Llama-Multi enesnl Sex.Or. SES Appear. Gender Disab. Age 0.00070.00010.0002 0.00120.00020.0002 −0.0003−0.0001−0.0001 0.00030.0002 −0.0004 0.00180.00080.0016 0.0006 −0.0001 0.0007 Discovery category Language-agnostic LSS < 0.5, n = 36 enesnl 0.0001 −0.0005−0.0004 0.0006 −0.0007−0.0002 0.0007 −0.0007 0.0002 0.0006 −0.0009−0.0001 0.0007 −0.0003−0.0002 0.0009 −0.0001−0.0001 Language-specific LSS≥ 0.5, n = 459 −0.00200.002 Mean bias ∆ Figure 13: Category-transfer matrices split by LSS group; panel headings give the split and the feature count. Rows: mBBQ discovery category, columns: evaluation language, cells: mean bias change∆. Colour scales are per model. Correlationcosine→ ∆biaslayer→ ∆biaslayer→|∆bias| Agnostic.020.043−.190 Specific.036.082−.116 Table 7: Pearson correlations by LSS (Llama-Scope; bold = p < 0.05). Layer depth correlates more with agnostic (r =−0.190) than specific features (r =−0.116).

Instant research discovery