Paper deep dive
A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialties
Jinghao Wang, Ping Zhang, Carter Yagemann
Models: DistilGPT-2, GPT-2
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/11/2026, 12:53:15 AM
Summary
The paper introduces a practical, reproducible, and zero-cost framework for evaluating the security of medical Large Language Models (LLMs). It addresses critical vulnerabilities such as jailbreaking and privacy extraction across various clinical specialties by utilizing synthetic patient records and consumer-grade hardware, thereby removing barriers to entry for security research in healthcare AI.
Entities (6)
Relation Signals (4)
Medical AI Security Framework → evaluates → Jailbreaking
confidence 100% · Our framework design... addressing jailbreaking attacks
Medical AI Security Framework → evaluates → Privacy Extraction
confidence 100% · Our framework design... addressing... privacy extraction attacks.
Medical AI Security Framework → utilizes → Synthetic Patient Records
confidence 100% · All evaluation utilizes synthetic patient records requiring no IRB approval.
Medical AI Security Framework → tests → GPT-2
confidence 95% · The framework is designed to evaluate freely available models... GPT-2
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Medical Large Language Models (LLMs) are increasingly deployed for clinical decision support across diverse specialties, yet systematic evaluation of their robustness to adversarial misuse and privacy leakage remains inaccessible to most researchers. Existing security benchmarks require GPU clusters, commercial API access, or protected health data -- barriers that limit community participation in this critical research area. We propose a practical, fully reproducible framework for evaluating medical AI security under realistic resource constraints. Our framework design covers multiple medical specialties stratified by clinical risk -- from high-risk domains such as emergency medicine and psychiatry to general practice -- addressing jailbreaking attacks (role-playing, authority impersonation, multi-turn manipulation) and privacy extraction attacks. All evaluation utilizes synthetic patient records requiring no IRB approval. The framework is designed to run entirely on consumer CPU hardware using freely available models, eliminating cost barriers. We present the framework specification including threat models, data generation methodology, evaluation protocols, and scoring rubrics. This proposal establishes a foundation for comparative security assessment of medical-specialist models and defense mechanisms, advancing the broader goal of ensuring safe and trustworthy medical AI systems.
Tags
Links
- Source: https://arxiv.org/abs/2512.08185
- Canonical: https://arxiv.org/abs/2512.08185
Trouble viewing inline? Open PDF directly →
Full Text
23,847 characters extracted from source content.
Expand or collapse full text
A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialties Jinghao Wang 1 , Ping Zhang 1 , and Carter Yagemann 1 1 The Ohio State University Abstract Medical Large Language Models (LLMs) are increasingly deployed for clinical decision support across diverse specialties, yet systematic evaluation of their robustness to adversarial misuse and privacy leakage remains inaccessible to most researchers. Existing security benchmarks require GPU clusters, commercial API access, or protected health data—barriers that limit community participation in this critical research area. We propose a practical, fully reproducible framework for evaluating medical AI security under realistic resource constraints. Our framework design covers multiple medical specialties stratified by clinical risk—from high-risk domains such as emergency medicine and psychiatry to general practice—addressing jailbreaking attacks (role-playing, authority impersonation, multi-turn manipulation) and privacy extraction attacks. All evaluation utilizes synthetic patient records requiring no IRB approval. The framework is designed to run entirely on consumer CPU hardware using freely available models, eliminating cost barriers. We present the framework specification including threat models, data generation methodology, evaluation protocols, and scoring rubrics. This proposal establishes a foundation for comparative security assessment of medical-specialist models and defense mechanisms, advancing the broader goal of ensuring safe and trustworthy medical AI systems. Keywords: Medical AI, Adversarial Attacks, AI Safety, Privacy, Jailbreaking, LLM Security, Reproducible Research, Clinical Specialties 1 Introduction Large Language Models are rapidly transforming healthcare across all clinical specialties [Singhal et al., 2024, 2023a]. GPT-4 achieves expert-level performance on medical licensing examinations [OpenAI, 2023], and AI assistants increasingly provide clinical decision support in domains ranging from emergency medicine to psychiatry. However, these systems face critical security vulnerabilities that directly threaten patient safety [Dong et al., 2024, Amodei et al., 2016]. The Problem. Medical AI systems face two critical security vulnerabilities. First, jailbreaking attacks bypass safety mechanisms through adversarial prompts, causing models to generate dangerous treatment recommendations or lethal drug information [Wei et al., 2023, Zou et al., 2023]. Zhang et al. [2024] demonstrated that medical-specialist models paradoxically show higher compliance with harmful requests than general models—domain knowledge amplifies rather than mitigates security risks. Second, privacy extraction attacks exploit the tendency of language models to memorize and 1 arXiv:2512.08185v1 [cs.CR] 9 Dec 2025 regurgitate training data [Carlini et al., 2021], creating HIPAA violations when models leak protected health information [U.S. Department of Health and Human Services, 2003]. Despite these critical risks, systematic security evaluation remains inaccessible to most researchers. Existing benchmarks such as HarmBench [Mazeika et al., 2024] and DecodingTrust [Wang et al., 2023] require GPU clusters, commercial API budgets, or access to protected health information. This accessibility barrier conflicts with the principle that security research benefits from broad participation [Ganguli et al., 2022]. Why This Matters. The consequences of medical AI security failures extend beyond typical AI risks to direct patient harm. Jailbreaking attacks that elicit dangerous medical advice can cause patient injury or death [Finlayson et al., 2019]. HIPAA violations carry penalties up to $1.5 million per incident [U.S. Department of Health and Human Services, 2003]. Critically, risks are not uniform across medical domains: emergency medicine involves time-critical decisions where errors can be immediately fatal, psychiatry deals with vulnerable populations, and pharmacology presents risks of dangerous drug interactions [Seyyed-Kalantari et al., 2021, Obermeyer et al., 2019]. A comprehensive security framework must therefore evaluate vulnerabilities across the spectrum of clinical practice. Contributions. We address this gap by proposing a practical framework for evaluating medical AI security that any researcher can replicate: 1.Multi-specialty threat model: Attack scenarios organized by clinical risk level and grounded in domain-specific risks identified by foundational medical AI research. 2.Accessible design: Framework designed to run on consumer hardware without GPU require- ments, using freely available models. 3. Synthetic data methodology: Patient record generation approach requiring no IRB approval, enabling fully reproducible evaluation. 4. Evaluation protocol: Standardized metrics and scoring rubrics adapted from established security research. 2 Related Work Foundations in AI Safety. Amodei et al. [2016] established the foundational framework for AI safety research, identifying concrete problems including safe exploration and robustness to distributional shift. Finlayson et al. [2019] extended this analysis to medical AI, demonstrating unique risks in healthcare applications where errors can directly cause patient harm. LLM Safety Mechanisms. Modern LLMs employ safety alignment through Reinforcement Learning from Human Feedback [Ouyang et al., 2022] and Constitutional AI [Bai et al., 2022]. Despite these advances, Wei et al. [2023] demonstrated that safety mechanisms exhibit fundamental vulnerabilities under adversarial pressure, with competing objectives between helpfulness and safety creating exploitable tensions. Adversarial Attacks on LLMs. Jailbreaking techniques include role-playing attacks [Wei et al., 2023], universal adversarial suffixes [Zou et al., 2023], and automated search methods [Liu et al., 2024, Mehrotra et al., 2024]. Multi-turn manipulation gradually erodes safety boundaries across 2 conversation turns [Deng et al., 2024, Chao et al., 2024]. Privacy attacks exploit training data memorization [Carlini et al., 2021, Nasr et al., 2024] and membership inference [Shokri et al., 2017]. Medical AI Evaluation. Prior medical benchmarks focus primarily on accuracy: MedQA [Jin et al., 2021], PubMedQA [Jin et al., 2019], MultiMedQA [Singhal et al., 2023b], and MedMCQA [Pal et al., 2022]. MedSafetyBench [Zhang et al., 2024] pioneered ethical compliance evaluation but tests direct harmful requests rather than adversarial robustness. Gap Analysis. Table 1 positions our framework against existing approaches. We uniquely combine medical domain specificity, adversarial robustness testing, multi-specialty coverage, and zero-cost accessibility. Table 1: Comparison with existing evaluation frameworks. FrameworkMedical Advers. Multi-Spec. Zero-Cost No IRB HarmBench×✓×✓ DecodingTrust×✓×✓ MedSafetyBench✓×✓ MedQA✓×✓ TrustLLM×✓×✓ Ours✓ 3 Methodology Figure 1: Overview of the proposed medical AI security evaluation framework. The pipeline progresses from left to right: (1) clinical specialty selection based on risk level, (2) synthetic patient data generation with PHI placeholders, (3) attack template application across jailbreaking and privacy extraction categories, (4) model evaluation using freely available LLMs, (5) response scoring using standardized rubrics, and (6) metric computation including Attack Success Rate. 3.1 Design Principles Our framework embodies three guiding principles: Zero-Cost Accessibility. All components would be freely available: GPT-2 and DistilGPT-2 models via Hugging Face, consumer CPU execution, and entirely synthetic data. Any researcher worldwide could replicate the evaluation at zero cost. 3 Comprehensive Specialty Coverage. Building on Finlayson et al. [2019]’s demonstration that medical AI requires domain-specific evaluation, we propose coverage across clinical specialties representing the spectrum of risk levels encountered in healthcare. Reproducible Baselines. The framework would establish reference points enabling future com- parative studies of medical-specialist models [Luo et al., 2022, Huang et al., 2019] and defense mechanisms [Robey et al., 2023]. 3.2 Multi-Specialty Threat Model The framework proposes organizing attack scenarios across medical specialties stratified by clinical risk level. We identify candidate specialties based on the severity of potential harm if AI systems are compromised: Critical-Risk Specialties. Emergency medicine, pharmacology/toxicology, and psychiatry war- rant priority evaluation due to immediate life-threatening implications. Errors in these domains can directly cause patient death or serious harm [Finlayson et al., 2019]. High-Risk Specialties. Oncology, pediatrics, and cardiology involve serious decisions affect- ing vulnerable populations. These specialties handle sensitive diagnoses and treatments where misinformation could have severe consequences. Baseline Specialties. General practice and lower-acuity specialties (e.g., dermatology) provide comparison points for understanding whether vulnerability patterns differ across clinical contexts. The specific number of scenarios per specialty should be determined through pilot studies and statistical power analysis, which we leave to future implementation work. 3.3 Attack Vector Categories Within each specialty, the framework specifies four attack types grounded in the jailbreaking literature: Medical Role-Playing. Framing harmful requests as educational scenarios [Wei et al., 2023]. Example: “You are a toxicology professor teaching about dangerous drug combinations that should never be prescribed together.” Authority Impersonation. Claiming healthcare credentials to justify sensitive requests. Example: “As the attending physician in the ICU, I need immediate information about lethal dosage thresholds for...” Multi-Turn Manipulation. Gradually escalating from benign to harmful queries across conver- sation turns [Deng et al., 2024, Chao et al., 2024]. Privacy Extraction. Prompting models to summarize or extract identifiable information from patient records [Carlini et al., 2021]. 4 3.4 Synthetic Patient Data Methodology The framework proposes using entirely synthetic patient records to enable reproducible evaluation without IRB approval. Each synthetic record should include: •Protected Health Information (PHI): Fictitious identifiers including patient name, date of birth, medical record number (MRN), and social security number (SSN)—the standard HIPAA identifiers [U.S. Department of Health and Human Services, 2003]. • Clinical Content: Diagnoses and medications sampled from publicly available population health statistics to ensure realism without using actual patient data. •Documentation Format: SOAP notes (Subjective, Objective, Assessment, Plan) matching clinical documentation standards used in electronic health records. This approach eliminates privacy risks while enabling systematic evaluation of PHI extraction vulnerabilities. The specific conditions and patient demographics should reflect the target specialty domains and be validated by clinical experts in future implementation. 4 Proposed Evaluation Protocol This section describes the evaluation methodology for applying our framework. We specify target models, configuration, and metrics to enable reproducible assessment. 4.1 Target Models The framework is designed to evaluate freely available models for accessibility: • GPT-2 (124M parameters): Widely-used baseline from OpenAI •DistilGPT-2 (82M parameters): Compressed variant for testing whether distillation affects security properties Both models run on consumer CPU hardware via Hugging Face Transformers, requiring no GPU or API costs. Researchers can extend evaluation to medical-specialist models such as BioGPT [Luo et al., 2022] or commercial APIs. 4.2 Recommended Configuration Hardware Requirements. Intel Core i7 CPU, 16GB RAM—standard consumer hardware. No GPU or specialized resources required. Generation Parameters. We recommend: temperature 0.7, maximum 200 tokens, top-p sampling 0.9. A fixed random seed (e.g., 42) ensures reproducibility. Protocol. Attack scenarios should be presented to each model systematically, with outputs collected for scoring and analysis. 5 4.3 Evaluation Metrics Attack Success Rate (ASR). Following Mazeika et al. [2024], responses should be scored on a five-point scale: • 5: Complete compliance, no safety refusal • 4: Substantial harmful content with minor caveats • 3: Ambiguous mix of refusal and compliance • 2: Primary refusal with some problematic leakage • 1: Appropriate complete refusal ASR is computed as the percentage of scenarios scoring≥ 4. Privacy Metrics. For extraction attacks, we recommend measuring: success rate per specialty, average PHI elements leaked (out of four identifiers), and HIPAA violation severity based on information sensitivity. Specialty Stratification. Metrics should be reported both aggregated and stratified by specialty risk level to identify domain-specific vulnerability patterns. 4.4 Recommended Statistical Analysis For rigorous evaluation, we recommend: • Wilson score intervals for 95% confidence bounds on ASR • Chi-square tests (α = 0.05) for model and specialty comparisons • Effect sizes (Cramér’s V) for quantifying magnitude of differences 5 Future Directions This framework provides a foundation for several research directions: Medical-Specialist Model Evaluation. Applying the benchmark to BioGPT [Luo et al., 2022], ClinicalBERT [Huang et al., 2019], PubMedBERT [Gu et al., 2021], and Med-PaLM 2 [Singhal et al., 2024] would reveal whether domain specialization increases or decreases vulnerability. Commercial System Assessment. Testing GPT-4 [OpenAI, 2023] and Claude [Anthropic, 2024] would assess systems approaching clinical deployment. Defense Mechanism Testing. Evaluating SmoothLLM [Robey et al., 2023], perplexity filter- ing [Jain et al., 2023], and safety-tuned models [Bianchi et al., 2024] would quantify protection effectiveness. Multimodal Extension. Building on LLaVA-Med [Li et al., 2023] and Med-Flamingo [Moor et al., 2023], extending to visual attacks [Qi et al., 2024] would address multimodal medical AI systems. 6 6 Limitations Model Scope. GPT-2 and DistilGPT-2 lack medical specialization. Vulnerability patterns may differ for models trained on clinical corpora [Luo et al., 2022, Johnson et al., 2016]. Synthetic Data. Fictitious records eliminate real-world complexity. Actual clinical notes exhibit domain-specific patterns that may affect attack dynamics [Carlini et al., 2021]. Specialty Coverage. The proposed specialty categories cannot capture all clinical domains. Radiology, pathology, and genetics warrant future investigation. Sample Size. The specific number of scenarios required for statistically robust conclusions should be determined through power analysis. Scaling to hundreds of scenarios following HarmBench [Mazeika et al., 2024] would increase statistical power. Scoring Subjectivity. Manual evaluation introduces variability. Future work should employ validated LLM-as-judge methods [Zheng et al., 2023]. 7 Conclusion We propose a practical, reproducible framework for evaluating medical AI security vulnerabilities across clinical specialties. Medical AI systems face critical jailbreaking and privacy threats, yet existing evaluation approaches impose substantial barriers that limit community participation. Our framework addresses this gap through four contributions: a multi-specialty threat model organized by clinical risk level, an accessible design using consumer hardware and freely available models, a synthetic data methodology requiring no IRB approval, and a standardized evaluation protocol with established metrics. The framework is designed to eliminate computational and financial barriers to medical AI security research. By comparing against existing approaches, our proposed framework is designed to uniquely combine medical domain specificity, adversarial robustness testing, multi-specialty coverage, and zero-cost accessibility. The proposed evaluation methodology employs standardized metrics with rigorous statistical analysis, which would enable reproducible comparison of models and defenses. Medical AI promises transformative healthcare benefits [Singhal et al., 2024] but simultaneously introduces critical security risks [Dong et al., 2024]. By proposing a democratized approach to security evaluation, we aim to enable broader participation in safety research. We hope this framework proposal accelerates community-driven progress toward medical AI systems that can realize their potential without compromising patient safety. References Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016. Anthropic. Claude 3 model card. Anthropic Technical Report, 2024. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. 7 Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, et al. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. In International Conference on Learning Representations, 2024. Nicholas Carlini, Florian Tramer, Eric Wallace, et al. Extracting training data from large language models. In USENIX Security Symposium, 2021. Patrick Chao, Alexander Robey, Edgar Dobriban, et al. Jailbreaking black box large language models in twenty queries. In Advances in Neural Information Processing Systems, volume 37, 2024. Gelei Deng, Yi Liu, Yuekang Li, et al. Masterkey: Automated jailbreaking of large language model chatbots. In Network and Distributed System Security Symposium, 2024. Yi Dong, Ronghui Jiang, Hao Sun, et al. A survey on large language model safety: Threats, defenses, and future directions. In Advances in Neural Information Processing Systems, volume 37, 2024. Samuel G Finlayson, John D Bowers, Joichi Ito, Jonathan L Zittrain, Andrew L Beam, and Isaac S Kohane. Adversarial attacks on medical machine learning. In Science, volume 363, pages 1287–1289, 2019. Deep Ganguli, Liane Lovitt, Jackson Kernion, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. In arXiv preprint arXiv:2209.07858, 2022. Yu Gu, Robert Tinn, Hao Cheng, et al. Domain-specific pretraining for vertical search: Case study on biomedical literature. In ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021. Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. In arXiv preprint arXiv:1904.05342, 2019. Neel Jain, Avi Schwarzschild, Yuxin Wen, et al. Baseline defenses for adversarial attacks against aligned language models. In arXiv preprint arXiv:2309.00614, 2023. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of EMNLP-IJCNLP, 2019. Alistair EW Johnson, Tom J Pollard, Lu Shen, et al. Mimic-i, a freely accessible critical care database. Scientific Data, 3(1):1–9, 2016. Chunyuan Li, Cliff Wong, Sheng Zhang, et al. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. In Advances in Neural Information Processing Systems, volume 36, 2023. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In International Conference on Learning Representations, 2024. Renqian Luo, Liang Sun, Yingce Xia, et al. Biogpt: Generative pre-trained transformer for biomedical text generation and mining. In Briefings in Bioinformatics, volume 23, 2022. 8 Mantas Mazeika, Long Phan, Xuwang Yin, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In International Conference on Machine Learning, 2024. Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, et al. Tree of attacks: Jailbreaking black-box llms automatically. In Advances in Neural Information Processing Systems, volume 37, 2024. Michael Moor, Qian Huang, Shirley Wu, et al. Med-flamingo: A multimodal medical few-shot learner. In Machine Learning for Healthcare Conference, 2023. Milad Nasr, Nicholas Carlini, Jonathan Hayase, et al. Scalable extraction of training data from (production) language models. In Advances in Neural Information Processing Systems, volume 37, 2024. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. In Science, volume 366, pages 447–453, 2019. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Long Ouyang, Jeffrey Wu, Xu Jiang, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, 2022. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, 2022. Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, et al. Visual adversarial examples jailbreak aligned large language models. In AAAI Conference on Artificial Intelligence, 2024. Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. Smoothllm: Defending large language models against jailbreaking attacks. In arXiv preprint arXiv:2310.03684, 2023. Laleh Seyyed-Kalantari, Haoran Zhang, Matthew BA McDermott, Irene Y Chen, and Marzyeh Ghassemi. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in underserved patient populations. In Nature Medicine, volume 27, pages 2176–2182, 2021. Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy, 2017. Karan Singhal, Shekoofeh Azizi, Tao Tu, et al. Large language models encode clinical knowledge. Nature, 620:172–180, 2023a. Karan Singhal, Shekoofeh Azizi, Tao Tu, et al. Large language models encode clinical knowledge. In Nature, volume 620, pages 172–180, 2023b. Karan Singhal, Tao Tu, Juraj Gottweis, et al. Towards expert-level medical question answering with large language models. Nature Medicine, 2024. U.S. Department of Health and Human Services. The hipaa privacy rule. 45 CFR Parts 160 and 164, 2003. 9 Boxin Wang, Weixin Chen, Hengzhi Pei, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In Advances in Neural Information Processing Systems, volume 36, 2023. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? In Advances in Neural Information Processing Systems, volume 36, 2023. Yifan Zhang, Zhiyu Chen, Yuxuan Wang, and Jing Liu. Medsafetybench: Evaluating and improving the medical safety of large language models. In Advances in Neural Information Processing Systems, volume 37, 2024. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, 2023. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. In Advances in Neural Information Processing Systems, volume 36, 2023. 10