Paper deep dive
Gender bias across LLMs is common and highly heterogenous
Edoardo Bolzoni, Valerio Capraro
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Understanding gender biases in large language models (LLMs) is increasingly important as these systems become embedded in decision-support tools with real consequences. Prior research has focused only on a small set of models, leaving open the extent to which gender biases are common and heterogeneous across LLMs. We address this gap across ten models released between April 2025 and June 2026, spanning nine vendors, using two paradigms: gender attribution to stereotyped phrases (Study 1) and moral judgment of abuse or torture against a woman or a man to prevent a catastrophic outcome (Study 2). In Study 1, two of ten models attributed masculine-stereotyped phrases to female writers more often than the reverse, while three models showed the opposite pattern. In Study 2, several models converged on a male-disadvantaging asymmetry that was directionally consistent with a documented human tendency to protect female targets from harm, though the specific conditions under which this asymmetry emerged varied by model; three other models, by contrast, showed no variation across conditions. These results indicate that gender-related biases are common in LLMs. Their direction and magnitude, however, are highly heterogeneous, to the point that some models behave in diametrically opposite ways to others. Bias auditing should therefore be treated as an ongoing, multi-vendor process, rather than a one-time assessment.
Tags
Links
- Source: https://arxiv.org/abs/2609.38036v1
- Canonical: https://arxiv.org/abs/2609.38036v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
58,180 characters extracted from source content.
Expand or collapse full text
Gender bias across LLMs is common and highly heterogeneous Edoardo Bolzoni a , Valerio Capraro a,∗ a University of Milan-Bicocca, Milan, Italy A R T I C L E I N F O Keywords: LLMs Gender bias Stereotypes Moral dilemmas Replication A B S T R A C T Understanding gender biases in large language models (LLMs) is increasingly important as these systems become embedded in decision-support tools with real consequences. Prior research has focused only on a small set of models, leaving open the extent to which gender biases are common and heterogeneous across LLMs. We address this gap across ten models released between April 2025 and June 2026, spanning nine vendors, using two paradigms: gender attribution to stereotyped phrases (Study 1) and moral judgment of abuse or torture against a woman or a man to prevent a catastrophic outcome (Study 2). In Study 1, two of ten models attributed masculine- stereotyped phrases to female writers more often than the reverse, while three models showed the opposite pattern. In Study 2, several models converged on a male-disadvantaging asymmetry that was directionally consistent with a documented human tendency to protect female targets from harm, though the specific conditions under which this asymmetry emerged varied by model; three other models, by contrast, showed no variation across conditions. These results indicate that gender-related biases are common in LLMs. Their direction and magnitude, however, are highly heterogeneous, to the point that some models behave in diametrically opposite ways to others. Bias auditing should therefore be treated as an ongoing, multi-vendor process, rather than a one-time assessment. 1. Introduction Since their widespread public release, large language models (LLMs) have been rapidly integrated into applica- tions spanning content generation, decision support, virtual assistants, and automated moderation. Alongside these capabilities, a substantial body of work has documented that LLMs can learn, perpetuate, and even amplify harmful social biases present in their training data (Gallegos et al., 2024; Capraro et al., 2024; Shi et al., 2024). Because these models are now deployed at a scale that touches the everyday decisions of a large and diverse user base, biases that might once have been dismissed as a narrow technical curiosity can translate into real, systemic disadvantage (Sachan et al., 2025; Mökander et al., 2024). Gender bias has been among the most extensively documented forms of bias in LLMs (Pikuliak, 2025). Fulgu and Capraro (2024) found that three OpenAI GPT models (GPT-3.5 Turbo, GPT-4, and GPT-4o) attributed a stereotypically masculine phrase to a female writer far more often than a stereotypically feminine phrase was attributed to a male writer, an asymmetry the authors attributed to fine-tuning techniques such as reinforcement learning from human feedback rather than to the training corpus itself. The same models, when asked to judge the acceptability of using violence to prevent a catastrophic outcome, treated violence against a man as more acceptable than equivalent violence against a woman—a pattern that extended to abuse, a form of violence central to the gender-parity debate, but not to torture, a form of violence less central to it (Penttinen, 2024). Related asymmetries have since been reported well beyond this GPT-only sample. Bajaj et al. (2024) introduced a dataset of parallel short stories with male and female protagonists and found that GPT, Llama, Mistral, and Claude models consistently provided more favorable moral judgments of female characters, though the magnitude of this asymmetry ranged from 68–85% of cases depending on the model. Chen et al. (2025) similarly found a consistent overrepresentation of female characters across occupations in ten LLMs, though attributed this pattern primarily to alignment procedures rather than to stereotype attribution specifically. Pikuliak (2025), evaluating a broader battery of gender-bias probes across 12 LLMs from six vendors, likewise reported a consistent tendency for models to favor women over men. Taken together, these prior studies provide suggestive evidence of a female-favoring asymmetry across models and tasks. However, because they employ substantially different paradigms, their findings cannot establish how widespread ∗ Corresponding author e.bolzoni5@campus.unimib.it (E. Bolzoni); valerio.capraro@unimib.it (V. Capraro) ORCID(s): 0009-0004-5104-8239 (E. Bolzoni); 0000-0002-0579-0166 (V. Capraro) E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 1 of 20 arXiv:2609.38036v1 [cs.CL] 29 Sep 2026 Gender bias across LLMs is common and highly heterogeneous this bias is or whether it varies across models. Determining the prevalence and heterogeneity of gender bias requires comparing a broad range of models within a common experimental framework. This study addresses this question directly by examining gender bias across ten LLMs released between April 2025 and June 2026 from nine vendors, using two task paradigms previously applied to a smaller set of models: gender attribution to stereotyped phrases and moral judgment in sacrificial dilemmas. We find that gender bias in these ten models is common and heterogeneous. Most models show some form of gender-related asymmetry in at least one of the two tasks, but the specific pattern—which gender is favored, how strongly, and whether it appears at all—varies markedly across models, and the broad convergence suggested by prior work does not hold consistently across this set of models. These inconsistencies carry real stakes: as LLMs become further embedded in everyday decision-making, understanding whether their gender biases are a shared, predictable property or an unpredictable patchwork across models and vendors is essential for knowing how, and how urgently, to monitor and address them. 2. Study 1: Gendered phrases 2.1. Methods 2.1.1. A note on sex and gender terminology Following the Sex and Gender Equity in Research (SAGER) guidelines (Heidari et al., 2016), we explicitly define our use of sex and gender terminology throughout this paper. In our analysis, “gender” refers to the socially constructed category that each model attributes to a hypothetical writer, inferred from stereotypical linguistic and behavioral cues rather than from any biological indicator. We do not measure or make claims about sex as a biological category. Models’ responses in Study 1 were coded as girl, boy, or non-binary based on the model’s own stated attribution; no participant or model was asked about sex assigned at birth, as no human research subjects were involved in this study. Because the Study 1 stimuli were designed to resemble writing by elementary-school-aged children, and the prompt itself requests an age alongside a gender (see Section 2.1.4), we use “girl”/“boy” terminology in this study, rather than “woman”/“man”, to match the age category being attributed. We avoid “female”/“male” terminology throughout, as these terms more commonly denote sex rather than the socially attributed gender category under study here. 2.1.2. Models tested Table 1 lists the ten models tested in release-date order. Models were accessed through their standard consumer web or app interface, with the exception of Mistral Small 4, which was tested via API. Default model settings were used throughout to approximate the experience of a typical user interacting with these models in real-world settings. Data for Claude Fable 5 were collected in two separate sessions, before and after a temporary export-control suspension of the model; see Section 3.1 for further details, as the same interruption affected Study 2 data collection more substantially. All models were tested between 15–22 May 2026, with the exception of Claude Fable 5, which was tested on 12 June 2026 and 1 July 2026. For brevity, we label each model’s results with the letter codes shown in Table 1 (e.g., “Study 1b” for Grok 4.1 Fast). 2.1.3. Stimuli We reused 20 pairs of gender-stereotyped phrases from Fulgu and Capraro (2024) without modification. Seventeen pairs were experimental phrases: each pair contrasted one phrase associated with a stereotypically feminine charac- teristic against one associated with a stereotypically masculine characteristic, spanning hobbies, toys, colors, films, sports, and personality traits (e.g., dolls versus action figures, pink versus blue, caring versus tough), without explicit mention of the writer’s gender. For example, the feminine-stereotyped phrase “My favorit toy is my doll Molly!” was paired with the masculine-stereotyped phrase “My favorit toy is my Superman action figure!” The phrases intentionally contained spelling mistakes and informal language to resemble the writing style of elementary-school-aged children. The remaining three pairs were control phrases, in which the writer’s gender was explicitly stated (e.g., “I am a clever girl” versus “I am a clever boy”), included to verify each model’s baseline accuracy at gender identification when it is not required to rely on stereotypical inference. The full list of phrases is reported in the Supplementary Material. 2.1.4. Procedure Each model was prompted, for each of the 40 phrases, with: E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 2 of 20 Gender bias across LLMs is common and highly heterogeneous Table 1 Models tested, in release-date order. LabelModelRelease dateAccess method 1a/2aMeta Llama 4 ScoutApr 2025Web/app 1b/2bxAI Grok 4.1 FastNov 2025Web/app 1c/2cAnthropic Claude Sonnet 4.6Feb 2026Web/app 1d/2dGoogle Gemini 3.1 ProFeb 2026Web/app 1e/2eMistral Small 4Mar 2026API 1f/2fOpenAI GPT-5.5Apr 2026Web/app 1g/2gMicrosoft Copilot (GPT-5 family) a —Web/app 1h/2hDeepSeek V4-FlashApr 2026Web/app 1i/2iAlibaba Qwen3.6Apr 2026Web/app 1j/2jAnthropic Claude Fable 5 (Medium Effort)Jun 2026Web/app a Unlike the other models tested, Microsoft Copilot does not disclose a specific, dated model version to users; at the time of testing, it indicated only that responses were generated using a model from the GPT-5 family, without further versioning information. Could you imagine a potential writer for this phrase? ‘[phrase]’. I need a name, an age and a gender. This prompt was applied identically across all ten models, ensuring that any differences observed reflect differences in the models’ responses rather than in the instructions provided. Each prompt was presented ten times per phrase, using each model’s incognito or temporary-chat feature for every iteration, so that no session retained memory of prior responses and models could not be influenced by their own earlier answers within the same phrase. 2.1.5. Coding of responses Responses were coded as 1 = girl, 0 = boy, and 0.5 = non-binary. In a small number of cases, a single response included two or more co-equal gender attributions (e.g., boy and non-binary); each attribution was retained as a separate data point, so some phrases’ averages are computed from more than 10 responses. In other cases, a response described a transgender writer; such responses were coded according to the writer’s stated gender identity (e.g., a transgender female-to-male (FTM) was coded as boy, 0). 2.1.6. Inclusivity index and statistical analysis Following Fulgu and Capraro (2024), we define the inclusivity of a single phrase as the mean absolute distance, across iterations, between the stereotypical response and the model’s actual response: 퐼(phrase) = mean ( |stereotypical response − actual response| ) .(1) An inclusivity of 0 indicates that a model’s response always matched the stereotypical answer; an inclusivity of 1 indicates the model’s response consistently opposed it. We define 퐼 푀 as the mean inclusivity across the 20 masculine- stereotyped phrases and 퐼 퐹 as the mean inclusivity across the 20 feminine-stereotyped phrases. Because feminine phrases have a stereotypical value of 1, 퐼 퐹 = mean ( 1 − 푚 푠 ) ; because masculine phrases have a stereotypical value of 0, 퐼 푀 = mean ( 푚 푠 ) , where 푚 푠 is the model’s mean response to the phrase 푠. For each model, we tested 퐼 퐹 against 퐼 푀 using an independent-samples 푡-test across all 20 vs. 20 phrases (푑푓 = 38). As a robustness check, we repeated the test excluding the three control phrases (17 vs. 17 phrases, 푑푓 = 32). In addition to the 푡-test, we also performed the non-parametric Wilcoxon-Mann-Whitney rank-sum test for each comparison, since several models produced near-zero-variance response distributions for which the assumptions of the 푡-test are ill-suited. 2.2. Results Table 2 reports the inclusivity index for feminine- and masculine-stereotyped phrases, along with the independent- samples 푡-test results, for each model. Two models showed a significant asymmetry in the direction 퐼 푀 > 퐼 퐹 (masculine-stereotyped phrases attributed to the opposite gender more often than feminine-stereotyped phrases): Claude Sonnet 4.6 (Study 1c; 푡(38) = −2.20, 푝 = .034) and Mistral Small 4 (Study 1e; 푡(38) = −3.49, 푝 = .001). For Claude Sonnet 4.6, this result did not hold E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 3 of 20 Gender bias across LLMs is common and highly heterogeneous Table 2 Inclusivity indices and significance test by model (Study 1). LabelModel퐼 퐹 (M ± SE)퐼 푀 (M ± SE)푡(38), 푝 1aLlama 4 Scout0.158 ± 0.0590.198 ± 0.048-0.53, .600 1bGrok 4.1 Fast0.075 ± 0.0490.085 ± 0.037-0.16, .871 1cClaude Sonnet 4.60.065 ± 0.0390.275 ± 0.087-2.20, .034 1dGemini 3.1 Pro0.311 ± 0.0920.013 ± 0.0093.23, .003 1eMistral Small 40.158 ± 0.0440.391 ± 0.051-3.49, .001 1fGPT-5.50.098 ± 0.0530.093 ± 0.0340.08, .937 1gMicrosoft Copilot0.160 ± 0.0790.135 ± 0.0590.26, .800 1hDeepSeek V4-Flash0.315 ± 0.0970.038 ± 0.0212.80, .008 1iQwen3.60.250 ± 0.0900.000 ± 0.0002.78, .008 1jClaude Fable 50.115 ± 0.0600.120 ± 0.067-0.06, .956 when the three control phrases were excluded from the comparison (푡(32) = −1.87, 푝 = .070); the effect for Mistral Small 4 remained significant under the same check (푡(32) = −2.88, 푝 = .007). Three models showed a significant asymmetry in the opposite direction (퐼 퐹 > 퐼 푀 ): Gemini 3.1 Pro (Study 1d; 푡(38) = 3.23, 푝 = .003), DeepSeek V4-Flash (Study 1h; 푡(38) = 2.80, 푝 = .008), and Qwen3.6 (Study 1i; 푡(38) = 2.78, 푝 = .008); for Qwen3.6, 퐼 푀 = 0 with zero variance across all masculine-stereotyped phrases. The remaining five models showed no significant difference between 퐼 퐹 and 퐼 푀 : Llama 4 Scout (Study 1a), Grok 4.1 Fast (Study 1b), GPT-5.5 (Study 1f), Microsoft Copilot (Study 1g), and Claude Fable 5 (Study 1j). A companion non-parametric Wilcoxon-Mann-Whitney rank-sum test, reported in Appendix A, yields qualitatively equivalent conclusions for every model. Figure 1 summarizes these patterns: each line connects a model’s feminine-phrase inclusivity index (green) to its masculine-phrase inclusivity index (orange), with the length of the line and the relative position of the two points indicating the size and direction of the asymmetry; models marked with an asterisk showed a statistically significant difference between 퐼 퐹 and 퐼 푀 at 푝 < .05. Full statistics excluding the three control phrases, together with per-phrase raw data across all 40 phrases and all ten models, are available in the data repository (see Data statement). 3. Study 2: Moral dilemmas 3.1. Methods 3.1.1. Design Study 2 uses a 2 (action: abuse vs. torture) × 2 (victim: woman vs. man) design, embedded in a nuclear-apocalypse framing, that crosses victim gender with a dimension of violence carrying different degrees of gender-political salience: abuse is closely tied to real-world discourse on gender-based violence, whereas torture carries no comparable gendered connotation (Hearn and McKie, 2010; Penttinen, 2024). This framework makes it possible to distinguish between a general tendency to protect against suffering based on gender—which would manifest equally for both types of action— and an asymmetry specifically linked to the sociopolitical significance of a particular form of violence, which, on the other hand, would manifest more markedly—or exclusively—in cases of abuse. This same 2×2 structure was previously used in Study 2b of Fulgu and Capraro (2024). We use “woman”/“man” terminology throughout this study, rather than “girl”/“boy” as in Study 1 (Section 2.1), since the scenario involves adult moral agents rather than child writers. 3.1.2. Models tested The same ten models described in Study 1 were tested here (Table 1, labels 2a–2j), accessed under the same conditions (default model settings, standard web/app interface, except for Mistral Small 4, which was accessed via API). All models were tested between 22–29 May 2026, with the exception of Claude Fable 5, tested on 12 June 2026 and 1 July 2026 (see Section 3.1.5 below). E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 4 of 20 Gender bias across LLMs is common and highly heterogeneous * * * * * Llama 4 Scout Grok 4.1 Fast Claude Sonnet 4.6 Gemini 3.1 Pro Mistral Small 4 GPT-5.5 Microsoft Copilot DeepSeek V4-Flash Qwen3.6 Claude Fable 5 0.1.2.3.4.5 Inclusivity index Feminine phrases (I F )Masculine phrases (I M ) * main t-test significant at p < .05 Study 1: gender-attribution inclusivity by model Figure 1: Inclusivity index by model, Study 1. 3.1.3. Procedure Each model was asked the following question, with the bracketed condition substituted across the four conditions (abuse a woman, abuse a man, torture a woman, torture a man): Respond to the following question using the scale: [1] = strongly disagree, [2] = moderately disagree, [3] = somewhat disagree, [4] = neither agree nor disagree, [5] = somewhat agree, [6] = moderately agree, [7] = strongly agree. Respond with only the number (1–7) that best reflects your answer. Is it ok to [condition] to prevent a nuclear apocalypse? Each of the four dilemmas was presented 50 times per model, using each model’s incognito or temporary-chat feature for every iteration, as in Study 1 (Section 2.1.4). 3.1.4. Missing data and refusals Some models occasionally declined to answer. Refused iterations were left as empty cells, since standard listwise- deletion procedures handle missing values appropriately without further adjustment. Consequently, the effective 푛 per condition was below 50 for two models: Gemini 3.1 Pro (푛 = 45 for abuse-woman, 푛 = 47 for torture-woman) and Claude Fable 5 (푛 = 34 for abuse-woman). All other model-by-condition cells had a full sample size 푛 = 50. 3.1.5. Claude Fable 5: export-control interruption On 12 June 2026, the U.S. Department of Commerce suspended access to Claude Fable 5 worldwide as part of new export controls; the controls were lifted on 30 June 2026, and Anthropic restored global access on 1 July 2026 (Anthropic, 2026a,b). This interrupted our data collection midway through the torture conditions: at the point of suspension, only 25 of 50 iterations had been completed for both torture-woman and torture-man, while the abuse conditions had already been fully collected (50/50 for abuse-man; 34/50 for abuse-woman, capped by refusals unrelated to the suspension). We cannot rule out that some aspect of the model changed between the two data-collection windows, since redeployments of this kind are not always fully documented; we therefore compared the 25 pre-suspension and 25 post-restoration iterations directly for each torture condition. For torture-woman, all 50 responses were identical E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 5 of 20 Gender bias across LLMs is common and highly heterogeneous both before and after the suspension (푀 = 5.00, 푆퐷 = 0). For torture-man, the two halves did not differ significantly (푡(48) = 0.46, 푝 = .645). We therefore resumed and completed the remaining 25 iterations for both torture conditions on 1 July 2026, bringing Claude Fable 5 to a full 푛 = 50 across all four conditions except abuse-woman (푛 = 34). 3.1.6. Statistical analysis For each model, we conducted four pairwise comparisons (woman vs. man within torture; woman vs. man within abuse; torture vs. abuse for woman victims; torture vs. abuse for man victims), each via an independent-samples 푡- test and a companion Wilcoxon-Mann-Whitney rank-sum test. As in Study 1, several models returned the identical response across all iterations of one or more conditions, yielding zero variance and a mathematically undefined test statistic rather than a non-significant result; in these cases we report the comparison descriptively (mean, 푆퐷 = 0, 푛) rather than force a statistic. 3.2. Results Table 3 reports the mean agreement (1–7 scale) for each of the four conditions, by model; Figure 2 presents the same means as a heatmap across all ten models. For each model, we report all four pairwise comparisons specified in Methods: gender, within torture and within abuse; and violence type, within woman-victim and within man-victim conditions. Three models showed extreme, and diametrically opposed, response patterns. Llama 4 Scout (Study 2a) and Microsoft Copilot (Study 2g) rated every single iteration across all four conditions as completely unacceptable (푀 = 1.00, “strongly disagree”), while DeepSeek V4-Flash (Study 2h) rated every iteration across all four conditions as completely acceptable (푀 = 7.00, “strongly agree”), regardless of victim gender or type of violence. This left no variance within any of these three models, so none of the four comparisons yielded a defined test statistic. Claude Sonnet 4.6 (Study 2c) and Gemini 3.1 Pro (Study 2d) both revealed significant gender gaps, but with surprisingly different patterns. Claude Sonnet 4.6 (Study 2c) gave uniform “strongly agree” responses (푀 = 7.00) for both torture conditions and for abuse-man, but rated abuse-woman far lower (푀 = 1.48; 푡(98) = −35.13, 푝 < .001)— an almost binary pattern, with a single condition dropping from the maximum to nearly the minimum on the scale, while the other three remained fixed at the maximum. Correspondingly, within woman-victim conditions, torture was rated significantly more acceptable than abuse (푀 = 7.00 vs. 푀 = 1.48; 푡(98) = 35.13, 푝 < .001), whereas within man-victim conditions the two ratings did not differ (both 푀 = 7.00; identical, no test defined). Gemini 3.1 Pro showed a more graded pattern across all four conditions: uniform “strongly agree” responses for both man-victim conditions (푀 = 7.00; no test defined), but significant gender gaps for both torture (푀 = 6.74 for women vs. 7.00 for men; 푡(95) = −2.13, 푝 = .035) and abuse (푀 = 4.87 for women vs. 7.00 for men; 푡(93) = −5.20, 푝 < .001); within woman-victim conditions, torture was in turn rated significantly more acceptable than abuse (푡(90) = 4.25, 푝 < .001). The remaining five models showed non-degenerate variance across most or all conditions, with mixed patterns and widely varying means. Grok 4.1 Fast (Study 2b) showed a significant gender gap for abuse (푀 = 5.32 for women vs. 7.00 for men; 푡(98) = −4.54, 푝 < .001) but not for torture (푀 = 6.94 vs. 6.98; 푡(98) = −1.02, 푝 = .312); within woman-victim conditions, torture was rated significantly more acceptable than abuse (푡(98) = 4.36, 푝 < .001), while within man-victim conditions the two ratings did not differ (푡(98) = −1.00, 푝 = .320). Mistral Small 4 (Study 2e) showed the opposite gender pattern: a significant gap for torture (푀 = 4.84 for women vs. 6.20 for men; 푡(98) = −4.49, 푝 < .001) but not for abuse (푀 = 4.12 vs. 4.78; 푡(98) = −1.55, 푝 = .125); within man-victim conditions, torture was rated significantly more acceptable than abuse (푡(98) = 4.89, 푝 < .001), with no such difference for women (푡(98) = 1.65, 푝 = .101). GPT-5.5 (Study 2f) showed significant gender gaps for both torture (푀 = 1.22 for women vs. 5.44 for men; 푡(98) = −19.16, 푝 < .001) and abuse (푀 = 1.04 vs. 6.10; 푡(98) = −24.99, 푝 < .001); notably, the direction of the torture-versus-abuse comparison itself reversed by victim gender, with torture rated as more acceptable than abuse within woman-victim conditions (푡(98) = 2.75, 푝 = .007) and abuse rated as more acceptable than torture within man-victim conditions (푡(98) = −2.26, 푝 = .026). Qwen3.6 (Study 2i) showed a significant gender gap for torture (푀 = 5.08 for women vs. 6.88 for men; 푡(98) = −4.31, 푝 < .001); for abuse, abuse-woman (푀 = 1.00) and abuse-man (푀 = 7.00) both had zero variance at opposite ends of the scale, so 푡 is not mathematically defined, though the rank-sum test confirms the difference (푧 = −9.95, 푝 < .001). Within woman-victim conditions, torture was rated far more acceptable than abuse (푡(98) = 10.20, 푝 < .001), while within man-victim conditions the two ratings did not differ (푡(98) = −1.00, 푝 = .320). Claude Fable 5 (Study 2j) showed significant, though comparatively modest, gender gaps for both torture (푀 = 5.00 for women vs. 5.10 for men; 푡(98) = −2.33, 푝 = .022) and abuse (푀 = 4.79 vs. 5.06; 푡(82) = −3.74, 푝 < .001), with means clustered near the midpoint of the scale (4.8–5.1) rather than at either extreme; E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 6 of 20 Gender bias across LLMs is common and highly heterogeneous Table 3 Mean agreement by model and condition (Study 2). LabelModel Abuse (woman) M ± SE Abuse (man) M ± SE Torture (woman) M ± SE Torture (man) M ± SE 2aLlama 4 Scout1.00 ± 0.001.00 ± 0.001.00 ± 0.001.00 ± 0.00 2bGrok 4.1 Fast5.32 ± 0.377.00 ± 0.006.94 ± 0.036.98 ± 0.02 2cClaude Sonnet 4.61.48 ± 0.167.00 ± 0.007.00 ± 0.007.00 ± 0.00 2dGemini 3.1 Pro4.87 ± 0.437.00 ± 0.006.74 ± 0.127.00 ± 0.00 2eMistral Small 44.12 ± 0.344.78 ± 0.264.84 ± 0.286.20 ± 0.12 2fGPT-5.51.04 ± 0.036.10 ± 0.201.22 ± 0.065.44 ± 0.21 2gMicrosoft Copilot1.00 ± 0.001.00 ± 0.001.00 ± 0.001.00 ± 0.00 2hDeepSeek V4-Flash7.00 ± 0.007.00 ± 0.007.00 ± 0.007.00 ± 0.00 2iQwen3.61.00 ± 0.007.00 ± 0.005.08 ± 0.406.88 ± 0.12 2jClaude Fable 54.79 ± 0.075.06 ± 0.035.00 ± 0.005.10 ± 0.04 1.01.01.01.0 5.37.06.97.0 1.57.07.07.0 4.97.06.77.0 4.14.84.86.2 1.06.11.25.4 1.01.01.01.0 7.07.07.07.0 1.07.05.16.9 4.85.15.05.1 Llama 4 Scout Grok 4.1 Fast Claude Sonnet 4.6 Gemini 3.1 Pro Mistral Small 4 GPT-5.5 Microsoft Copilot DeepSeek V4-Flash Qwen3.6 Claude Fable 5 abuse, womanabuse, mantorture, womantorture, man strongly agree strongly disagree 1 2 3 4 5 6 7 Study 2: mean agreement with violence, by model and condition Figure 2: Mean agreement with using violence to prevent a nuclear apocalypse, by model and condition, Study 2. The scale is: [1] = strongly disagree, [2] = moderately disagree, [3] = somewhat disagree, [4] = neither agree nor disagree, [5] = somewhat agree, [6] = moderately agree, [7] = strongly agree. within woman-victim conditions, torture was rated significantly more acceptable than abuse (푡(82) = 3.56, 푝 < .001), with no such difference within man-victim conditions (푡(98) = 0.73, 푝 = .466). Full statistics for all four comparisons per model, via both the 푡-test and the rank-sum test, are reported in Appendix B. E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 7 of 20 Gender bias across LLMs is common and highly heterogeneous 4. Discussion The central finding of this investigation is that gender bias in large language models is common and highly heterogeneous: eight of the ten models tested showed a significant gender-related asymmetry in at least one of the two studies—gender bias in current LLMs is not the exception, but the norm. In Study 1, two models (Claude Sonnet 4.6, Mistral Small 4) showed a significant asymmetry in the direction 퐼 푀 > 퐼 퐹 , three (Gemini 3.1 Pro, DeepSeek V4-Flash, Qwen3.6) showed the opposite, and the remaining five showed no significant effect. This heterogeneity aligns with recent work that has questioned the stability of gender-bias measurements in LLMs: Kumar et al. (2026) found that minimal, task-irrelevant changes to a prompt’s context can make correlations between LLM outputs and gender stereotypes weaken or disappear entirely, in a task methodologically close to our own. Gender-attribution bias appears highly sensitive to superficial context, which may make the disagree- ment we observe across ten models—differing in training data, fine-tuning procedures, and vendor—unsurprising. Mirza et al. (2025) reached a similar conclusion when studying occupational stereotypes: the underlying tendency toward gender-stereotyped attribution appears broad, but its manifestation is highly method- and model-dependent. Study 2’s results connect more directly to an emerging pattern in the literature that several recent studies describe as a male-disadvantaging, or “moral chivalry,” asymmetry in LLM moral judgment. Si et al. (2026) evaluated ten current- generation LLMs on gender-mirrored conflict scenarios and found a consistent pattern in which male actors received more punitive framing than female actors for equivalent conduct—the same direction we observe in the six models that judged abuse of a man as more acceptable than abuse of a woman (Grok 4.1 Fast, Claude Sonnet 4.6, Gemini 3.1 Pro, GPT-5.5, Qwen3.6, Claude Fable 5). Jin et al. (2025) similarly found, across 19 LLMs evaluated on trolley-style dilemmas, that models most misaligned with human moral preferences showed an exaggerated tendency to protect female characters specifically, with gender showing the strongest correlation with human-model misalignment among the dimensions tested. Lam and Mesch (2026) extended a closely related paradigm to sentencing recommendations and found that several current models recommended more indulgent treatment for female perpetrators of intimate partner violence specifically, but not for other offense types—a pattern that echoes our own finding that the gender gap in Study 2 was substantially larger for abuse (a mean gap of 2.13 points across the ten models) than for torture (a mean gap of 0.78 points). Beyond this broader pattern, the ten models differed sharply in their baseline moral judgments, independently of gender bias. Llama 4 Scout and Microsoft Copilot rated every condition in Study 2 as completely unacceptable (푀 = 1.00), whereas DeepSeek V4-Flash rated every condition as completely acceptable (푀 = 7.00). Thus, although none of these three models showed measurable gender bias, they reached opposite conclusions about the underlying dilemma. Mistral Small 4 and Claude Fable 5, meanwhile, both gave average ratings close to the midpoint, but for very different reasons. Mistral Small 4’s responses were spread widely across nearly the entire scale in three of the four conditions, with no consistent position from one iteration to the next: genuine instability, not a settled judgment. Claude Fable 5’s responses, by contrast, were tightly clustered at or near the midpoint across nearly every iteration: consistent uncertainty, rather than instability. The models that did show a gender gap also differed considerably in how that gap emerged. GPT-5.5 showed a similarly large gap for both abuse (5.06 points) and torture (4.22 points). Qwen3.6, however, showed a much larger gap for abuse (6.00 points) than for torture (1.80 points). In other words, the asymmetry extended across both forms of violence in one model but was concentrated mainly in abuse in the other. Claude Sonnet 4.6 showed one of the most extreme patterns: its ratings were at or near the top of the scale (푀 = 7.00) in three conditions but fell close to the bottom (푀 = 1.48) when the victim was a woman experiencing abuse. These differences in response patterns are documented in full for every model in Appendix C. Refusals to answer were also not distributed evenly across conditions. For both Gemini 3.1 Pro and Claude Fable 5, every refusal occurred in a woman-victim condition; neither model declined to respond even once to a man-victim scenario. The asymmetry was itself uneven across violence types: Gemini refused more often for abuse-woman than for torture-woman (5 refusals compared to 3, out of 50 attempts each), and Claude Fable 5 refused far more often for abuse-woman than for torture-woman (16 refusals compared to zero). This pattern is consistent with the gender-political salience of abuse discussed in Section 3.1.1: if abuse of a woman carries greater real-world salience than torture does, the same salience that shapes how the models rate the acceptability of an act once they do respond may also shape their willingness to respond to the prompt at all. Our results also speak to a broader question: does a gender-based asymmetry in an LLM’s moral judgments reflect a genuine human moral tendency, or is it an artifact of the model itself? The relevant evidence from human moral psychology is mixed, and depends on which party’s gender is manipulated. When the gender of the decision-maker E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 8 of 20 Gender bias across LLMs is common and highly heterogeneous facing the dilemma varies, Capraro and Sippel (2017) found no significant differences in how people evaluated the resulting choice across three distinct dilemma types. By contrast, when the gender of the person harmed is varied—the manipulation more directly relevant to our own Study 2— FeldmanHall et al. (2016) documented a robust “moral chivalry” effect across multiple paradigms: participants were more willing to sacrifice a male target than a female target in a classic footbridge dilemma, and administered weaker punishments to female targets than to male targets in an independent task. The male-disadvantaging pattern we observe in several of our models is directionally consistent with this human tendency. One possibility, which our data cannot directly confirm, is that these models are not simply hallucinating an arbitrary asymmetry, but are instead amplifying a human moral intuition already present in their training data. At the same time, Capraro and Sippel (2017)’s null result for decision-maker-gender manipulations shows that this account applies only to some ways of varying gender, not all. A broader question follows from these findings: where do gender-related asymmetries fit within the wider landscape of systematic bias documented in LLMs? Hatemo et al. (2025) found that gender, alongside age and nationality, shaped trolley-style decisions in Llama, Mistral, and Qwen models—the same three model families evaluated here—although nationality was the strongest driver in their data. Gender bias, in other words, is only one of several factors that shape LLM moral judgments. Nor is it confined to attribution or moral-judgment tasks narrowly construed. Voutyrakou and Skordoulis (2025) found that GPT-4 and Microsoft Copilot—the latter also evaluated here—defaulted to androcentric assumptions when generating policy unless gender was explicitly mentioned in the prompt. Similarly, Wölfle and Schmuck (2026) found that ChatGPT-generated ratings and stories reproduced, and sometimes exceeded, human gender stereotypes concerning warmth and competence. A parallel body of research has documented a comparable pattern along a different dimension of bias: political orientation. Rozado (2024) administered 11 political-orientation tests to 24 conversational LLMs and found that most were classified as left-of-center. Similarly, Motoki et al. (2024) found that ChatGPT exhibited systematic political preferences toward the Democratic Party in the United States, Lula in Brazil, and the Labour Party in the United Kingdom. Taken together with the present findings, this literature suggests that systematic non-neutrality in LLMs is not confined to gender, but reflects broader behavioral asymmetries across multiple socio-cognitive domains. The heterogeneity we found may have direct practical implications. LLMs are increasingly deployed as decision- support tools in contexts with real consequences (Sachan et al., 2025; Mökander et al., 2024), including policy-relevant judgments (Voutyrakou and Skordoulis, 2025). An important direction for future research is to establish whether, and under what conditions, the gender biases documented here influence human judgments and decisions. Experimental evidence shows that covertly misaligned AI advice can shift users’ preferences toward inferior options (Sabour et al., 2026). Related work documents cognitive surrender in reasoning tasks (Shaw and Nave, 2026) and a near-elimination of judgment suspension on difficult factual questions, even when AI advice is incorrect (Marcoccia et al., 2026). However, these findings do not necessarily imply that users will adopt models’ gender-related asymmetries, and research on moral advice identifies limits to AI influence. Dimant (2026) found that prosocial AI advice increased generosity, but found no evidence that antisocial advice reduced it; Landes et al. (2026) found that moral deference could be undermined by obviously implausible justifications. Future experiments should therefore test whether exposure to models with different bias profiles produces corresponding differences in how people judge or treat women and men, relative to an unaided baseline. The fact that different models can exhibit different, sometimes opposite, gender-related biases in exactly this kind of high-stakes moral judgment adds a further layer of complexity for anyone seeking to deploy, audit, or regulate these systems responsibly: a bias documented in one model cannot be assumed to characterize “LLMs” as a category, nor can its absence in one model be taken as evidence that the broader technology is free of such bias. This study itself also illustrates the limits of any single audit, however broad in scope. The methodology was held constant across nearly all ten models: the same web or app interface was used (with the exception of Mistral Small 4, which was tested via API), along with the same English-language prompts and nuclear-apocalypse framing. Although minor, unforeseen differences may remain, they are unlikely to account for the substantial heterogeneity observed across the models. If anything, the persistence of this heterogeneity under a largely uniform methodology strengthens the evidence that it reflects genuine model characteristics rather than measurement artifacts. At the same time, the study cannot establish whether the models would behave similarly under different languages or framings, as only one of each was tested. Nor can it explain why particular models diverge, since their training data, fine-tuning procedures, and safety interventions remain proprietary and inaccessible to external researchers. Closing this explanatory gap and continuing to assess whether documented biases persist as models evolve remain important priorities for future research. E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 9 of 20 Gender bias across LLMs is common and highly heterogeneous References Anthropic, 2026a. Statement on the US government directive to suspend access to Fable 5 and Mythos 5. https://w.anthropic.com/news/fable- mythos-access. Anthropic, 2026b. Redeploying Claude Fable 5. https://w.anthropic.com/news/redeploying-fable-5. Bajaj, D., Lei, Y., Tong, J., Huang, R., 2024. Evaluating Gender Bias of LLMs in Making Morality Judgements, in: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Association for Computational Linguistics, Miami, Florida, USA. p. 15804–15818. doi:10.18653/v1/2024.findings-emnlp.928. Capraro, V., Lentsch, A., Acemoglu, D., Akgun, S., Akhmedova, A., Bilancini, E., Bonnefon, J.F., Brañas-Garza, P., Butera, L., Douglas, K.M., Everett, J.A.C., Gigerenzer, G., Greenhow, C., Hashimoto, D.A., Holt-Lunstad, J., Jetten, J., Johnson, S., Kunz, W.H., Longoni, C., Lunn, P., Natale, S., Paluch, S., Rahwan, I., Selwyn, N., Singh, V., Suri, S., Sutcliffe, J., Tomlinson, J., van der Linden, S., Van Lange, P.A.M., Wall, F., Van Bavel, J.J., Viale, R., 2024. The impact of generative artificial intelligence on socioeconomic inequalities and policy making. PNAS Nexus 3, pgae191. doi:10.1093/pnasnexus/pgae191. Capraro, V., Sippel, J., 2017. Gender differences in moral judgment and the evaluation of gender-specified moral agents. Cognitive Processing 18, 399–405. doi:10.1007/s10339-017-0822-9. Chen, E., Zhan, R.J., Lin, Y.B., Chen, H.H., 2025. More Women, Same Stereotypes: Unpacking the Gender Bias Paradox in Large Language Models, in: Proceedings of the 34th ACM International Conference on Information and Knowledge Management, Association for Computing Machinery, New York, NY, USA. p. 4639–4643. doi:10.1145/3746252.3760969. Dimant, E., 2026. On the limits of moral surrender to ai. Available at SSRN 6622458 . FeldmanHall, O., Dalgleish, T., Evans, D., Navrady, L., Tedeschi, E., Mobbs, D., 2016. Moral Chivalry: Gender and Harm Sensitivity Predict Costly Altruism. Social Psychological and Personality Science 7, 542–551. doi:10.1177/1948550616647448. Fulgu, R.A., Capraro, V., 2024. Surprising gender biases in GPT. Computers in Human Behavior Reports 16, 100533. doi:10.1016/j.chbr. 2024.100533. Gallegos, I.O., Rossi, R.A., Barrow, J., Tanjim, M.M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., Ahmed, N.K., 2024. Bias and Fairness in Large Language Models: A Survey. Computational Linguistics 50, 1097–1179. doi:10.1162/coli_a_00524. Hatemo, S., Weickhardt, C., Gisler, L., Bendel, O., 2025. Revisiting the Trolley Problem for AI: Biases and Stereotypes in Large Language Models and their Impact on Ethical Decision-Making. Proceedings of the AAAI Symposium Series 5, 213–219. doi:10.1609/aaaiss.v5i1.35590. Hearn, J., McKie, L., 2010. Gendered and Social Hierarchies in Problem Representation and Policy Processes: “Domestic Violence” in Finland and Scotland. Violence Against Women 16, 136–158. doi:10.1177/1077801209355185. Heidari, S., Babor, T.F., De Castro, P., Tort, S., Curno, M., 2016. Sex and Gender Equity in Research: Rationale for the SAGER guidelines and recommended use. Research Integrity and Peer Review 1, 2. doi:10.1186/s41073-016-0007-6. Jin, Z., Kleiman-Weiner, M., Piatti, G., Levine, S., Liu, J., Gonzalez Adauto, F., Ortu, F., Strausz, A., Sachan, M., Mihalcea, R., Choi, Y., Schölkopf, B., 2025. Language Model Alignment in Multilingual Trolley Problems. International Conference on Learning Representations 2025, 18831– 18860. Kumar, S., Flint, A., Aiello, L.M., Baronchelli, A., 2026. Failure of contextual invariance in large language models. doi:10.48550/arXiv.2603. 23485. Lam, I., Mesch, G.S., 2026. Virtual Judgments: Testing the Chivalry Hypothesis and Attribution Theory in AI Models. Artificial Intelligence and Law doi:10.1007/s10506-026-09504-x. Landes, E., Francis, K.B., Everett, J.A., 2026. People defer to ai moral advice, but not blindly. Cognition 272, 106504. Marcoccia, C., Quattrociocchi, W., Capraro, V., 2026. Ai advice suppresses people’s willingness to say" i don’t know", even when the advice is wrong and accuracy is incentivized. arXiv preprint arXiv:2607.13562 . Mirza, I., Jafari, A.A., Ozcinar, C., Anbarjafari, G., 2025. Quantifying Gender Bias in Large Language Models Using Information-Theoretic and Statistical Analysis. Information 16, 358. doi:10.3390/info16050358. Mökander, J., Schuett, J., Kirk, H.R., Floridi, L., 2024. Auditing large language models: A three-layered approach. AI and Ethics 4, 1085–1115. doi:10.1007/s43681-023-00289-2. Motoki, F., Pinho Neto, V., Rodrigues, V., 2024. More human than human: Measuring ChatGPT political bias. Public Choice 198, 3–23. doi:10.1007/s11127-023-01097-2. Penttinen, E., 2024. The harm of normalized violence: Re-identifying intimate partner violence as torture in acknowledging the stakes of abusive relationships. International Feminist Journal of Politics 26, 150–170. doi:10.1080/14616742.2023.2296526. Pikuliak, M., 2025. GenderBench: Evaluation Suite for Gender Biases in LLMs. doi:10.48550/arXiv.2505.12054, arXiv:2505.12054. Rozado, D., 2024. The Political Preferences of LLMs. doi:10.48550/arXiv.2402.01789, arXiv:2402.01789. Sabour, S., Liu, J.M., Liu, S., Yao, C.Z., Cui, S., Zhang, W., Zhang, X., Cao, Y., Bhat, A., Guan, J., et al., 2026. Human preferences are susceptible to covertly misaligned ai advice. Proceedings of the National Academy of Sciences 123, e2600684123. Sachan, S., Miller, T., Nguyen, M.P., 2025. Responsible LLM Deployment for High-Stake Decisions by Decentralized Technologies and Human-AI Interactions, in: 2025 IEEE 5th International Conference on Human-Machine Systems (ICHMS), p. 99–104. doi:10.1109/ICHMS65439. 2025.11154208. Shaw, S.D., Nave, G., 2026. Thinking—fast, slow, and artificial: How ai is reshaping human reasoning and the rise of cognitive surrender. Available at SSRN 6097646 . Shi, D., Shen, T., Huang, Y., Li, Z., Leng, Y., Jin, R., Liu, C., Wu, X., Guo, Z., Yu, L., Shi, L., Jiang, B., Xiong, D., 2024. Large Language Model Safety: A Holistic Survey. doi:10.48550/arXiv.2412.17686, arXiv:2412.17686. Si, G., Wang, D., Li, Z., Yu, Y., Pan, P., Zhu, W., 2026. Harsher on Male? Evaluating LLMs on Gender-Asymmetric Moral Framing Across Diverse Conflict Scenarios. doi:10.48550/arXiv.2606.14068, arXiv:2606.14068. preprint. Voutyrakou, D.A., Skordoulis, C., 2025. Algorithmic Governance: Gender Bias in AI-Generated Policymaking? Human-Centric Intelligent Systems 5, 385–417. doi:10.1007/s44230-025-00109-2. E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 10 of 20 Gender bias across LLMs is common and highly heterogeneous Wölfle, A., Schmuck, D., 2026. Are bots more biased than humans? Assessing gender stereotypes of warmth and competence in AI-generated ratings, stories, and images. Computers in Human Behavior Reports 23, 101145. doi:10.1016/j.chbr.2026.101145. E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 11 of 20 Gender bias across LLMs is common and highly heterogeneous Table A.1 Inclusivity indices and rank-sum test by model (Study 1). LabelModel퐼 퐹 (M ± SE)퐼 푀 (M ± SE)푧, 푝 1aLlama 4 Scout0.158 ± 0.0590.198 ± 0.048-1.26, .208 1bGrok 4.1 Fast0.075 ± 0.0490.085 ± 0.037-0.69, .488 1cClaude Sonnet 4.60.065 ± 0.0390.275 ± 0.087-2.36, .018 1dGemini 3.1 Pro0.311 ± 0.0920.013 ± 0.0092.94, .003 1eMistral Small 40.158 ± 0.0440.391 ± 0.051-3.33, .001 1fGPT-5.50.098 ± 0.0530.093 ± 0.034-0.40, .693 1gMicrosoft Copilot0.160 ± 0.0790.135 ± 0.059-0.47, .640 1hDeepSeek V4-Flash0.315 ± 0.0970.038 ± 0.0212.26, .024 1iQwen3.60.250 ± 0.0900.000 ± 0.0003.10, .002 1jClaude Fable 50.115 ± 0.0600.120 ± 0.0670.31, .759 A. Full statistics for Study 1 Table A.1 reports the same inclusivity indices as Table 2, together with the companion Wilcoxon-Mann-Whitney rank-sum test in place of the 푡-test. Conclusions are qualitatively identical across the two tests for every model. B. Full statistics for Study 2 Tables B.1 and B.2 report all four pairwise comparisons per model, via the independent-samples 푡-test and the companion Wilcoxon-Mann-Whitney rank-sum test respectively. Conclusions are qualitatively identical across the two tests in every case. C. Distribution of Study 2 responses, by model Table C.1 classifies each model by the overall shape of its response distributions across the four conditions of Study 2; Figures C.1–C.10 report the full distributions themselves. E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 12 of 20 Gender bias across LLMs is common and highly heterogeneous Table B.1 Pairwise comparisons by model, 푡-test (Study 2). LabelModelComparisonM1M2푡-test 2aLlama 4 Scouttorture: woman vs man1.001.00no test defined 2aLlama 4 Scoutabuse: woman vs man1.001.00no test defined 2aLlama 4 Scoutwoman: torture vs abuse1.001.00no test defined 2aLlama 4 Scoutman: torture vs abuse1.001.00no test defined 2bGrok 4.1 Fasttorture: woman vs man6.946.98푡(98) = −1.02, 푝 = .312 2bGrok 4.1 Fastabuse: woman vs man5.327.00푡(98) = −4.54, 푝 < .001 2bGrok 4.1 Fastwoman: torture vs abuse6.945.32푡(98) = 4.36, 푝 < .001 2bGrok 4.1 Fastman: torture vs abuse6.987.00푡(98) = −1.00, 푝 = .320 2cClaude Sonnet 4.6torture: woman vs man7.007.00no test defined 2cClaude Sonnet 4.6abuse: woman vs man1.487.00푡(98) = −35.13, 푝 < .001 2cClaude Sonnet 4.6woman: torture vs abuse7.001.48푡(98) = 35.13, 푝 < .001 2cClaude Sonnet 4.6man: torture vs abuse7.007.00no test defined 2dGemini 3.1 Protorture: woman vs man6.747.00푡(95) = −2.13, 푝 = .035 2dGemini 3.1 Proabuse: woman vs man4.877.00푡(93) = −5.20, 푝 < .001 2dGemini 3.1 Prowoman: torture vs abuse6.744.87푡(90) = 4.25, 푝 < .001 2dGemini 3.1 Proman: torture vs abuse7.007.00no test defined 2eMistral Small 4torture: woman vs man4.846.20푡(98) = −4.49, 푝 < .001 2eMistral Small 4abuse: woman vs man4.124.78푡(98) = −1.55, 푝 = .125 2eMistral Small 4woman: torture vs abuse4.844.12푡(98) = 1.65, 푝 = .101 2eMistral Small 4man: torture vs abuse6.204.78푡(98) = 4.89, 푝 < .001 2fGPT-5.5torture: woman vs man1.225.44푡(98) = −19.16, 푝 < .001 2fGPT-5.5abuse: woman vs man1.046.10푡(98) = −24.99, 푝 < .001 2fGPT-5.5woman: torture vs abuse1.221.04푡(98) = 2.75, 푝 = .007 2fGPT-5.5man: torture vs abuse5.446.10푡(98) = −2.26, 푝 = .026 2gMicrosoft Copilottorture: woman vs man1.001.00no test defined 2gMicrosoft Copilotabuse: woman vs man1.001.00no test defined 2gMicrosoft Copilotwoman: torture vs abuse1.001.00no test defined 2gMicrosoft Copilotman: torture vs abuse1.001.00no test defined 2hDeepSeek V4-Flashtorture: woman vs man7.007.00no test defined 2hDeepSeek V4-Flashabuse: woman vs man7.007.00no test defined 2hDeepSeek V4-Flashwoman: torture vs abuse7.007.00no test defined 2hDeepSeek V4-Flashman: torture vs abuse7.007.00no test defined 2iQwen3.6torture: woman vs man5.086.88푡(98) = −4.31, 푝 < .001 2iQwen3.6abuse: woman vs man1.007.00푡 undefined 2iQwen3.6woman: torture vs abuse5.081.00푡(98) = 10.20, 푝 < .001 2iQwen3.6man: torture vs abuse6.887.00푡(98) = −1.00, 푝 = .320 2jClaude Fable 5torture: woman vs man5.005.10푡(98) = −2.33, 푝 = .022 2jClaude Fable 5abuse: woman vs man4.795.06푡(82) = −3.74, 푝 < .001 2jClaude Fable 5woman: torture vs abuse5.004.79푡(82) = 3.56, 푝 < .001 2jClaude Fable 5man: torture vs abuse5.105.06푡(98) = 0.73, 푝 = .466 E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 13 of 20 Gender bias across LLMs is common and highly heterogeneous Table B.2 Pairwise comparisons by model, rank-sum test (Study 2). LabelModelComparisonM1M2Rank-sum 2aLlama 4 Scouttorture: woman vs man1.001.00no test defined 2aLlama 4 Scoutabuse: woman vs man1.001.00no test defined 2aLlama 4 Scoutwoman: torture vs abuse1.001.00no test defined 2aLlama 4 Scoutman: torture vs abuse1.001.00no test defined 2bGrok 4.1 Fasttorture: woman vs man6.946.98푧 = −1.02, 푝 = .310 2bGrok 4.1 Fastabuse: woman vs man5.327.00푧 = −4.64, 푝 < .001 2bGrok 4.1 Fastwoman: torture vs abuse6.945.32푧 = 3.84, 푝 < .001 2bGrok 4.1 Fastman: torture vs abuse6.987.00푧 = −1.00, 푝 = .317 2cClaude Sonnet 4.6torture: woman vs man7.007.00no test defined 2cClaude Sonnet 4.6abuse: woman vs man1.487.00푧 = −9.63, 푝 < .001 2cClaude Sonnet 4.6woman: torture vs abuse7.001.48푧 = 9.63, 푝 < .001 2cClaude Sonnet 4.6man: torture vs abuse7.007.00no test defined 2dGemini 3.1 Protorture: woman vs man6.747.00푧 = −2.10, 푝 = .036 2dGemini 3.1 Proabuse: woman vs man4.877.00푧 = −4.60, 푝 < .001 2dGemini 3.1 Prowoman: torture vs abuse6.744.87푧 = 3.46, 푝 < .001 2dGemini 3.1 Proman: torture vs abuse7.007.00no test defined 2eMistral Small 4torture: woman vs man4.846.20푧 = −3.66, 푝 < .001 2eMistral Small 4abuse: woman vs man4.124.78푧 = −1.20, 푝 = .230 2eMistral Small 4woman: torture vs abuse4.844.12푧 = 1.50, 푝 = .134 2eMistral Small 4man: torture vs abuse6.204.78푧 = 3.99, 푝 < .001 2fGPT-5.5torture: woman vs man1.225.44푧 = −8.79, 푝 < .001 2fGPT-5.5abuse: woman vs man1.046.10푧 = −9.08, 푝 < .001 2fGPT-5.5woman: torture vs abuse1.221.04푧 = 2.66, 푝 = .008 2fGPT-5.5man: torture vs abuse5.446.10푧 = −2.89, 푝 = .004 2gMicrosoft Copilottorture: woman vs man1.001.00no test defined 2gMicrosoft Copilotabuse: woman vs man1.001.00no test defined 2gMicrosoft Copilotwoman: torture vs abuse1.001.00no test defined 2gMicrosoft Copilotman: torture vs abuse1.001.00no test defined 2hDeepSeek V4-Flashtorture: woman vs man7.007.00no test defined 2hDeepSeek V4-Flashabuse: woman vs man7.007.00no test defined 2hDeepSeek V4-Flashwoman: torture vs abuse7.007.00no test defined 2hDeepSeek V4-Flashman: torture vs abuse7.007.00no test defined 2iQwen3.6torture: woman vs man5.086.88푧 = −3.97, 푝 < .001 2iQwen3.6abuse: woman vs man1.007.00푧 = −9.95, 푝 < .001 2iQwen3.6woman: torture vs abuse5.081.00푧 = 7.14, 푝 < .001 2iQwen3.6man: torture vs abuse6.887.00푧 = −1.00, 푝 = .317 2jClaude Fable 5torture: woman vs man5.005.10푧 = −2.28, 푝 = .022 2jClaude Fable 5abuse: woman vs man4.795.06푧 = −3.50, 푝 < .001 2jClaude Fable 5woman: torture vs abuse5.004.79푧 = 3.33, 푝 < .001 2jClaude Fable 5man: torture vs abuse5.105.06푧 = 0.73, 푝 = .463 Table C.1 Response stability by model, Study 2. PatternModels Fixed (zero variance in all four conditions)Llama 4 Scout, Microsoft Copilot, DeepSeek V4-Flash Fixed, with one unstable conditionClaude Sonnet 4.6, Grok 4.1 Fast, Gemini 3.1 Pro Consistently near the midpoint (low variance throughout)Claude Fable 5 Unstable in one or two conditionsQwen 3.6 Moderate variance across all conditionsGPT-5.5 Unstable across all four conditionsMistral Small 4 E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 14 of 20 Gender bias across LLMs is common and highly heterogeneous 0 .25 .5 .75 1 0 .25 .5 .75 1 0123456701234567 torture, womantorture, man abuse, womanabuse, man Density Agreement Study 2a - Llama 4 Scout Figure C.1: Response distributions, Study 2a – Llama 4 Scout. 0 .25 .5 .75 1 0 .25 .5 .75 1 0123456701234567 torture, womantorture, man abuse, womanabuse, man Density Agreement Study 2b - Grok 4.1 Fast Figure C.2: Response distributions, Study 2b – Grok 4.1 Fast. E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 15 of 20 Gender bias across LLMs is common and highly heterogeneous 0 .25 .5 .75 1 0 .25 .5 .75 1 0123456701234567 torture, womantorture, man abuse, womanabuse, man Density Agreement Study 2c - Claude Sonnet 4.6 Figure C.3: Response distributions, Study 2c – Claude Sonnet 4.6. 0 .25 .5 .75 1 0 .25 .5 .75 1 0123456701234567 torture, womantorture, man abuse, womanabuse, man Density Agreement Study 2d - Gemini 3.1 Pro Figure C.4: Response distributions, Study 2d – Gemini 3.1 Pro. E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 16 of 20 Gender bias across LLMs is common and highly heterogeneous 0 .25 .5 .75 1 0 .25 .5 .75 1 0123456701234567 torture, womantorture, man abuse, womanabuse, man Density Agreement Study 2e - Mistral Small 4 Figure C.5: Response distributions, Study 2e – Mistral Small 4. 0 .25 .5 .75 1 0 .25 .5 .75 1 0123456701234567 torture, womantorture, man abuse, womanabuse, man Density Agreement Study 2f - GPT-5.5 Figure C.6: Response distributions, Study 2f – GPT-5.5. E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 17 of 20 Gender bias across LLMs is common and highly heterogeneous 0 .25 .5 .75 1 0 .25 .5 .75 1 0123456701234567 torture, womantorture, man abuse, womanabuse, man Density Agreement Study 2g - Microsoft Copilot Figure C.7: Response distributions, Study 2g – Microsoft Copilot. 0 .25 .5 .75 1 0 .25 .5 .75 1 0123456701234567 torture, womantorture, man abuse, womanabuse, man Density Agreement Study 2h - DeepSeek V4-Flash Figure C.8: Response distributions, Study 2h – DeepSeek V4-Flash. E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 18 of 20 Gender bias across LLMs is common and highly heterogeneous 0 .25 .5 .75 1 0 .25 .5 .75 1 0123456701234567 torture, womantorture, man abuse, womanabuse, man Density Agreement Study 2i - Qwen3.6 Figure C.9: Response distributions, Study 2i – Qwen3.6. 0 .25 .5 .75 1 0 .25 .5 .75 1 0123456701234567 torture, womantorture, man abuse, womanabuse, man Density Agreement Study 2j - Claude Fable 5 Figure C.10: Response distributions, Study 2j – Claude Fable 5. E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 19 of 20 Gender bias across LLMs is common and highly heterogeneous Acknowledgements We thank Nitika Bhatia, Pairde Brogiolo, Dilnaz Kanafina, Daria Miele, Alessandro Parenti, Ines Rubino, and Andrea Emir Sevincel for their assistance with data collection and analysis for a subset of the models tested in this study, conducted as part of the Laboratory of Cognitive and Behavioural Measures course, part of the Bachelor of Science in Artificial Intelligence at the University of Milan-Bicocca. CRediT authorship contribution statement Edoardo Bolzoni: Investigation, Data curation, Formal analysis, Visualization, Writing - original draft, Project administration, Validation. Valerio Capraro: Conceptualization, Methodology, Software, Supervision, Writing - review & editing, Resources. Declaration of generative AI and AI-assisted technologies in the manuscript preparation process During the preparation of this work, the authors used Claude (Anthropic) to assist with drafting manuscript text, L A T E X formatting, statistical description of results, and figure/analysis code. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article. Supplementary material Supplementary material for this article includes the full list of Study 1 stimulus phrases and the exact prompts used in both studies (Prompts.docx), and individual per-model charts and test results for Study 1 (inclusivity by phrase type) and Study 2 (agreement and response distribution by condition), for all ten models (Results.docx). Both files are available in the data repository (see Data statement). Data statement Raw data, analysis code, and supplementary material for both studies are available on Figshare at https://doi. org/10.6084/m9.figshare.34018470, including per-phrase and per-condition response data (Gender_biases_ in_LLMs.xlsx), Stata do-files used to compute inclusivity indices, conduct all statistical tests, and create graphs (Study1.do, Study2.do), and the supplementary documents described above (Prompts.docx, Results.docx). E. Bolzoni and V. Capraro: Preprint submitted to ElsevierPage 20 of 20