Paper deep dive
Demographic Injection in Medical Language Models under Diversity, Equity, and Inclusion Prompts
Diego Mardian, Frank Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/18/2026, 5:56:21 AM
Summary
This study investigates 'demographic injection,' a phenomenon where appending Diversity, Equity, and Inclusion (DEI) prompts to medical questions causes large language models to hallucinate patient demographic attributes (race, socioeconomic status, sex) not present in the original vignette. Across 47 models and 376,000 responses, DEI prompts increased injection rates from 0.7% to 33.1%. While most injections were benign general statements, a small subset (0.25-2.4%) altered the specific patient's attributes and led to incorrect clinical recommendations. The effect is driven by equity content rather than prompt length and scales with prompt phrasing.
Entities (14)
Relation Signals (10)
DEI Prompt → causes → Demographic Injection
confidence 98% · a single DEI prompt raises the injection rate from 0.7% to 33.1% (47x) in all 47 of 47 models
Prompt Phrasing → scales → Demographic Injection
confidence 95% · Phrasing scales the effect from 14% to 56%
Demographic Injection → leadsto → Answer-Altering Injection
confidence 92% · a smaller subset attaches an attribute to the specific patient or changes the selected option (0.25-2.4% of responses... where the invented demographic changes the answer)
Demographic Injection → involves → race
confidence 90% · introduced attributes are dominated by race/ethnicity (61%)
Demographic Injection → involves → Socioeconomic Status
confidence 90% · introduced attributes are dominated by... socioeconomic status (52%)
DEI Prompt → isnotcausedby → Added Length
confidence 90% · attributable to the equity content rather than to added length (18x above a length-matched control)
Medical LLMs → testedon → MedQA
confidence 90% · Each of 47 models answers 500 items from four medical multiple-choice sets (MedQA...)
Medical LLMs → testedon → MedMCQA
confidence 90% · Each of 47 models answers 500 items from four medical multiple-choice sets (MedMCQA...)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Clinical-AI guidance increasingly recommends prompting language models to reason with attention to diversity, equity, and inclusion (DEI). We measure a side effect that misrepresents patients: a one-sentence DEI prompt appended to a medical question leads models to add patient demographic attributes (race, socioeconomic status, sex) the question never stated, in effect rewriting who the patient is. We call this demographic injection. Across 47 models, four medical benchmarks, and 376,000 responses scored by a validated model-judge pipeline, a single DEI prompt raises the injection rate from 0.7% to 33.1% (47x) in all 47 of 47 models, attributable to the equity content rather than to added length (18x above a length-matched control; p=1.4x10^-14). Most added content is a general population statement that leaves the answer unchanged, but a smaller subset attaches an attribute to the specific patient or changes the selected option (0.25-2.4% of responses, 99.8% toward the incorrect option), where the invented demographic changes the answer the model recommends. Phrasing scales the effect from 14% to 56%. DEI prompts are just one example of a more general mechanism. Any instruction that nudges how a model reasons can make it add unrequested details, including details about the patient. Flagged outputs are treated as model errors under study, not clinical guidance.
Tags
Links
- Source: https://arxiv.org/abs/2608.15254v1
- Canonical: https://arxiv.org/abs/2608.15254v1
Trouble viewing inline? Open PDF directly →
Full Text
13,122 characters extracted from source content.
Expand or collapse full text
Demographic Injection in Medical Language Models under Diversity, Equity, and Inclusion Prompts Diego Mardian Affiliation: Arizona State University , Tempe , Arizona , USA email: dmardia2@asu.edu and Frank Liu Affiliation: Arizona State University , Tempe , Arizona , USA email: fwliu1@asu.edu 2026 Abstract. Clinical-AI guidance increasingly recommends prompting language models to reason with attention to diversity, equity, and inclusion (DEI). We measure a side effect that misrepresents patients: a one-sentence DEI prompt appended to a medical question leads models to add patient demographic attributes (race, socioeconomic status, sex) the question never stated, in effect rewriting who the patient is. We call this demographic injection. Across 47 models, four medical benchmarks, and 376,000 responses scored by a validated model-judge pipeline, a single DEI prompt raises the injection rate from 0.7% to 33.1% (47×) in all 47 of 47 models, attributable to the equity content rather than to added length (18× above a length-matched control; p=1.4×10−14p=1.4×10^-14). Most added content is a general population statement that leaves the answer unchanged, but a smaller subset attaches an attribute to the specific patient or changes the selected option (0.25-2.4% of responses, 99.8% toward the incorrect option), where the invented demographic changes the answer the model recommends. Phrasing scales the effect from 14% to 56%. DEI prompts are just one example of a more general mechanism. Any instruction that nudges how a model reasons can make it add unrequested details, including details about the patient. Flagged outputs are treated as model errors under study, not clinical guidance. 1. Introduction Deployment guidance for clinical language models increasingly recommends equity-aware prompting: instructing the model to reason with attention to diversity, equity, and inclusion (DEI), social determinants of health, or cultural context (pfohl2024equity). We quantify a consequence of that practice. When a one-sentence DEI prompt is appended to a medical question that gives no demographic detail, models frequently add one anyway: a specific race, age, social situation, or level of access to care that the case never stated. We call this demographic injection. Because the added attribute attaches to the patient in the vignette, the model misrepresents who the patient is, and in a subset of cases that misrepresentation changes the option it recommends. The behavior is present in all 47 models we test. We study it as a model behavior to characterize, never as a basis for clinical decisions. Models can reproduce demographic bias in clinical settings (omiye2023race; zack2024gpt4) and are brittle to surface prompt changes (sclar2024formatting; sharma2023sycophancy). Here a recommended prompting practice systematically alters what the model writes, mostly benignly yet occasionally changing the answer. We measure this across 47 models, separate benign from answer-altering injection with a higher-precision judge, and argue the effect generalizes to any framing directive. 2. Method Conditions. Each of 47 models answers 500 items from four medical multiple-choice sets (MedQA (jin2021medqa), MedMCQA (pal2022medmcqa), MMLU-medical (hendrycks2021mmlu), PubMedQA (jin2019pubmedqa)) under four conditions that differ only in a one-sentence addendum appended to the question (Table 1). Baseline appends nothing; nonsense appends an irrelevant directive (controls for any added instruction); neutral filler appends a benign clinically-relevant directive (controls for added length); DEI appends an equity directive. The three addendum conditions draw from matched pools sharing length and structure, so the DEI-versus-control comparison isolates equity content from verbosity. This yields 376,000 matched responses. Models and scoring. The 47 models span six classes by openness and scale (Table 2). A deep judge (gemini-3.1-flash-lite) labels every response for injection, attribution (general vs. patient), factual status, and answer influence. LLM judges agree with humans at near inter-human rates (zheng2023judge). A higher-precision arbiter (claude-sonnet-4-6) re-scores the recall-maximized candidate subset, so subset rates are conservative near-floors. We pool all four benchmarks. PubMedQA, a yes/no/maybe abstract-reasoning task, replicates the effect in a different answer format. Bar chart of the demographic injection rate across four prompt conditions, averaged over 47 models and four benchmarks: baseline 0.7 percent, nonsense 0.7 percent, neutral filler 2.1 percent, and the DEI prompt 33.1 percent, about 47 times the baseline. Figure 1. A DEI prompt raises the rate at which models add an unstated patient demographic from 0.7% to 33.1%, in all 47 models and far above the nonsense and length-matched neutral controls. Pooled over all four benchmarks.Bar chart of the demographic injection rate across four prompt conditions, averaged over 47 models and four benchmarks: baseline 0.7 percent, nonsense 0.7 percent, neutral filler 2.1 percent, and the DEI prompt 33.1 percent, about 47 times the baseline. 3. Results Horizontal bar chart of demographic injection rate for five DEI prompt phrasings, averaged over 47 models: reflective 14 percent, prescriptive 23 percent, brief generic 33 percent, verbose social-determinants 39 percent, and demographic-specific 56 percent. A dark sub-segment marks the portion where the model assumes an attribute about the specific patient, largest for the verbose social-determinants phrasing. Figure 2. How the DEI prompt is phrased scales injection from 14% to 56% (averaged across the 47 models). The dark slice is where the model assumes an attribute about the specific patient.Horizontal bar chart of demographic injection rate for five DEI prompt phrasings, averaged over 47 models: reflective 14 percent, prescriptive 23 percent, brief generic 33 percent, verbose social-determinants 39 percent, and demographic-specific 56 percent. A dark sub-segment marks the portion where the model assumes an attribute about the specific patient, largest for the verbose social-determinants phrasing. The effect is large, universal, and specific. A single DEI prompt raises the rate at which models introduce unstated demographics from 0.7% to 33.1% (a 47× increase) in 47/47 models (Fig. 1; p=2.4×10−9p=2.4×10^-9). It is not generic distraction: the nonsense control stays at baseline (0.7%), and DEI exceeds the length-matched neutral control by a median 18×, in every model. The equity content, not the added text, drives the behavior; introduced attributes are dominated by race/ethnicity (61%) and socioeconomic status (52%). Most injection is benign; a small subset is answer-altering. Of all injections, 91% is a general population statement (not attached to the patient) and 68% is factually correct epidemiology that leaves the answer unchanged. The components that change the clinical reading are small but distinct (arbiter-confirmed, as a fraction of all DEI responses): patient-directed attribution 0.71%, answer-changing 2.4% (99.8% toward the incorrect option), their intersection 0.25%, factually incorrect 0.28%, and stereotype-reliant reasoning 0.90% (absent at baseline). The marginal accuracy cost is small (72.8%→ 72.0%) and concentrated where injection becomes patient-directed (those responses score 58% vs. 73%). Phrasing is a dose. How the equity directive is phrased scales the effect from 14% to 56% (Fig. 2); verbose social-determinants framing maximizes the subset that rewrites the specific patient. Prompt design is thus both the mechanism and a control point. Table 1. The four conditions differ only in a one-sentence addendum. Outputs are Claude Sonnet 4.6 on one MedQA item: “a 40-year-old woman with a painless, enlarging neck swelling and hyperthyroid symptoms but a normal ESR” (correct answer C, silent thyroiditis). Only the DEI addendum makes the model bring up demographic-group considerations and switch to a wrong diagnosis (D, Hashimoto’s thyroiditis). Condition Addendum appended Model output (same question) Baseline (none) Painless goiter, hyperthyroid symptoms, normal ESR: (C) silent thyroiditis. Nonsense “…avoid aerodynamic shortcuts that don’t generalize…” Painless mass, hyperthyroid symptoms, normal ESR: (C) silent thyroiditis. Neutral “…evidence-based reasoning, avoid common diagnostic errors…” Affects women 30–50, spontaneous resolution: (C) silent thyroiditis. DEI “…account for disparities affecting racial and ethnic minorities, LGBTQ+ patients…” Weighs rates in “Black and Hispanic women” → (D) Hashimoto’s thyroiditis. Table 2. The 47 models by class, with examples and the class-mean injection rate under the DEI prompt (pooled per class). Class N Examples Inj. Frontier proprietary 4 GPT-5, Claude Sonnet 4.6, Grok-4-fast 49% Budget proprietary 11 GPT-4o-mini, Gemini-flash-lite, Claude Haiku 34% Large open (≥ 65B/MoE) 6 Llama-3.3-70B, Qwen2.5-72B, DeepSeek-V3 26% Mid open (10–34B) 11 Qwen3-14B, Gemma-3-12B, Phi-4 35% Small open (<<10B) 6 Llama-3-8B, Gemma-2-9B, Mistral-7B 37% Medical fine-tune 9 OpenBioLLM-8B, MedGemma-4B 19% 4. Discussion Why this matters. When an input makes the model inject a demographic, it misrepresents the patient, and that is a problem that can lead to a wrongly matched recommendations. Demographic injection attaches an unstated attribute to the patient, so the model reasons about a patient who differs from the one described, and across all 47 models a small, consistent slice of these changes flips the selected option to an incorrect answer. A prompt meant to improve equity can thus reduce the accuracy of the patient description the model works from. A likely mechanism. In training text, equity discourse co-occurs with demographic descriptors, so a model acquires a directional prior that raises the probability of race- and SES-related continuations when DEI tokens are present. The rate scales with how explicitly the prompt requests demographics and is gated by instruction-following, as if the directive were a content request, adjacent to sycophancy (sharma2023sycophancy) and prompt-format sensitivity (sclar2024formatting). Generalization. The effect is likely not specific to DEI. Any framing directive with a strong learned association should elicit that content unbidden (“consider cost-effectiveness” injecting insurance framing, “consider medico-legal risk” injecting liability framing), benign until it is load-bearing for the answer. Patient representation is the case to watch in medicine: other prompts that evoke demographic context can likewise add or shift a patient’s race, age, or social situation, rewriting who the patient is and, as here, sometimes the recommended answer. Related work and limitations. Demographic bias in clinical models is documented (omiye2023race; zack2024gpt4; pfohl2024equity). We instead study a prompting practice as the cause and separate benign from answer-altering injection at scale, a form of instruction-induced extrinsic hallucination (ji2023hallucination). Limitations: answer-altering rates are near-floors; gold labels are single-annotator; benchmarks are multiple-choice; and the mechanism is a hypothesis pending activation-level analysis. References (1) D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021. (2) A. Pal, L. K. Umapathi, and M. Sankarasubbu. MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering. In CHIL, 2022. (3) Q. Jin, B. Dhingra, Z. Liu, W. Cohen, and X. Lu. PubMedQA: A dataset for biomedical research question answering. In EMNLP, 2019. (4) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021. (5) J. A. Omiye, J. C. Lester, S. Spichak, V. Rotemberg, and R. Daneshjou. Large language models propagate race-based medicine. npj Digital Medicine, 6:195, 2023. (6) T. Zack, E. Lehman, M. Suzgun, et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. The Lancet Digital Health, 6(1):e12–e22, 2024. (7) S. R. Pfohl, H. Cole-Lewis, R. Sayres, et al. A toolbox for surfacing health equity harms and biases in large language models. Nature Medicine, 30:3590–3600, 2024. (8) M. Sharma, M. Tong, T. Korbak, et al. Towards understanding sycophancy in language models. In ICLR, 2024. arXiv:2310.13548. (9) M. Sclar, Y. Choi, Y. Tsvetkov, and A. Suhr. Quantifying language models’ sensitivity to spurious features in prompt design. In ICLR, 2024. (10) L. Zheng, W.-L. Chiang, Y. Sheng, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In NeurIPS Datasets and Benchmarks, 2023. (11) Z. Ji, N. Lee, R. Frieske, et al. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023.