Paper deep dive
Reliability Testing of Medical Model Performance under Distributed Deployment
Yifei Wang, Xiaohan Zhang, Youtao Ding, Tianlin Li, Xiaoyu Zhang, Yida Yang, Li Pan
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Distributed inference has become an indispensable part of deploying medical models under practical latency, memory, and throughput constraints. Although modern frameworks improve serving efficiency through tensor parallelism, mixed precision, kernel fusion, and multi-device communication, they are generally assumed to preserve the behavior observed during centralized HuggingFace evaluation. This assumption creates an evaluation-deployment mismatch: a model may pass offline evaluation but produce a different output after the execution stack changes. To address this mismatch, we propose a testing framework and an improved, distributed-execution-sensitive medical-model benchmark that evaluates the same checkpoint and input under a centralized HuggingFace reference and matched distributed deployments. Extensive experiments across language, vision, and multimodal medical models show that execution changes can produce measurable output disagreements. Across supported visual settings, the test success rate ranges from 0.21 to 0.43 for single-modality models and from 0.32 to 0.98 for multimodal models. The benchmark is aimed at extending medical-model evaluation from capability and security to evaluation-deployment consistency.
Tags
Links
- Source: https://arxiv.org/abs/2609.36525v1
- Canonical: https://arxiv.org/abs/2609.36525v1
Trouble viewing inline? Open PDF directly →
Full Text
50,400 characters extracted from source content.
Expand or collapse full text
Reliability Testing of Medical Model Performance under Distributed Deployment Yifei Wang Xiaohan Zhang Youtao Ding Tianlin Li Xiaoyu Zhang Yida Yang Li Pan Abstract Distributed inference has become an indispensable part of deploying medical models under practical latency, memory, and throughput constraints. Although modern frameworks improve serving efficiency through tensor parallelism, mixed precision, kernel fusion, and multi-device communication, they are generally assumed to preserve the behavior observed during centralized HuggingFace evaluation. This assumption creates an evaluation-deployment mismatch: a model may pass offline evaluation but produce a different output after the execution stack changes. To address this mismatch, we propose a testing framework and an improved, distributed-execution-sensitive medical-model benchmark that evaluates the same checkpoint and input under a centralized HuggingFace reference and matched distributed deployments. Extensive experiments across language, vision, and multimodal medical models show that execution changes can produce measurable output disagreements. Across supported visual settings, the test success rate ranges from 0.21 to 0.43 for single-modality models and from 0.32 to 0.98 for multimodal models. The benchmark is aimed at extending medical-model evaluation from capability and security to evaluation-deployment consistency. 1Shanghai Jiao Tong University, China 2Beihang University, China 3Nanyang Technological University, Singapore 4Tongji University, China 1 Introduction Medical foundation models are widely used for medical image classification, visual question answering and multimodal clinical decision support. Large language models (LLMs) (Touvron et al. 2023), vision models and vision-language models (VLMs) (Radford et al. 2021; Li et al. 2023b) underpin modern medical AI. Their evaluation and deployment environments, however, are often different. Offline evaluation commonly uses a centralized single-device HuggingFace configuration, whereas practical serving commonly uses distributed frameworks with tensor parallelism, mixed precision, fused kernels, and multi-device communication to satisfy latency and memory constraints (Aminabadi et al. 2022; Kwon et al. 2023; Pope et al. 2023). The deployment workflow therefore relies on a simple but largely untested assumption: a medical image that receives one prediction during evaluation should receive the same prediction after the execution system changes (Aminabadi et al. 2022; Pope et al. 2023; Kwon et al. 2023). Figure 1: The same abdominal CT image receives different predictions under HuggingFace and two-GPU DeepSpeed inference in our test. This assumption exposes an important gap in medical-model evaluation. Existing studies mainly measure factual correctness, reasoning ability, hallucination, and task accuracy on medical question-answering and reasoning benchmarks (Jin et al. 2021; Ben Abacha and Demner-Fushman 2019; Jin et al. 2019; Pal et al. 2022; Singhal et al. 2025; Zheng et al. 2023); they also examine robustness under harmful or deliberately difficult prompts (Wei et al. 2023; Liu et al. 2023). What they rarely measure is visual deployment consistency: whether a model validated on a medical image in one execution environment preserves the same confidence profile and final prediction in another. Figure 1 illustrates the consequence. The same abdominal CT image is classified as Abdominal CT Scan by the HuggingFace reference but as Chest X-Ray by a two-GPU DeepSpeed execution, although the checkpoint and image are unchanged. Thus, a small execution-level difference can become a confidence shift, and a confidence shift can become a different medical prediction. Deployment-induced inconsistency should therefore be tested as an independent reliability dimension rather than hidden inside aggregate accuracy. To address this evaluation-deployment mismatch, we present a deployment-centered differential testing framework and propose an improved, distributed-execution-sensitive medical-model benchmark. The benchmark follows a paired design: HuggingFace serves as the centralized evaluation reference, and each distributed execution serves as a matched deployment variant. In every comparison, the checkpoint, input semantics, and decision rule are fixed; only the execution stack changes. We instantiate this design with complementary visual and textual test-generation procedures and conduct extensive experiments across language, vision, and multimodal medical models. On the visual track, single-modality models exhibit test success rates from 0.21 to 0.43, whereas multimodal models show larger and more framework-dependent variation, ranging from 0.32 to 0.98 across the evaluated distributed configurations. These results show that deployment disagreement is not limited to language generation, but also affects medical vision and multimodal inference. The benchmark therefore turns an implicit deployment assumption into an explicit evaluation target. Based on its visual track, we further localize the layers that amplify execution differences and apply one targeted mitigation: selectively elevating these critical layers to higher precision while leaving the remaining deployment pipeline unchanged. Our main contributions are as follows: • We formulate distributed inference as a deployment-consistency testing problem for medical AI and show that centralized and distributed executions can disagree even when the checkpoint, input, and decision rule are identical. • We introduce an execution-aware differential testing framework with symmetric visual and textual test-generation procedures. Based on the visual tests, we further localize divergence-amplifying layers and mitigate disagreements by elevating only these critical layers to higher precision. • We propose an improved, distributed-execution-sensitive medical-model benchmark that makes evaluation-deployment consistency an explicit evaluation criterion by pairing centralized HuggingFace evaluation with matched distributed executions. 2 Related Work 2.1 Testing and Deployment Reliability Machine-learning testing has developed differential testing, metamorphic testing, coverage-guided analysis, fuzzing, and mutation testing to reveal failures in learned systems (Murphy et al. 2008; Dwarakanath et al. 2018; Pei et al. 2017; Chen et al. 2018; Odena et al. 2019; Guo et al. 2018; Ma et al. 2018). Domain-specific studies further examine structural coverage, autonomous-driving perception, and DNN debugging (Xie et al. 2019; Humbatova et al. 2020; Riccio et al. 2020). At the system level, CRADLE and Audee show that an identical DL program can behave differently across platforms (Pham et al. 2019; Guo et al. 2020; Guo et al. 2019; Wang et al. 2024); at the language-model level, recent work studies robustness under prompt perturbation and other difficult inputs (Wei et al. 2023; Liu et al. 2023; Wallace et al. 2019; Zou et al. 2023). Our work follows the differential-testing principle but centers the comparison on medical visual behavior: the model, image, and prediction rule remain fixed, while the execution stack changes from a centralized reference to a distributed deployment. 2.2 Medical Foundation Model Evaluation Medical foundation-model evaluation mainly measures capability and semantic safety, including medical image understanding, multimodal perception, question answering, reasoning quality, hallucination, and guideline consistency (Jin et al. 2021; Ben Abacha and Demner-Fushman 2019; Jin et al. 2019; Pal et al. 2022; Singhal et al. 2023; Singhal et al. 2025; Chen et al. 2023; Zheng et al. 2023; Rückert et al. 2024; Li et al. 2023a; Li et al. 2023b; Radford et al. 2021). Visual benchmarks evaluate whether a model extracts the correct evidence from an image, while language benchmarks such as MedQA, MedQuAD, PubMedQA, and MedMCQA evaluate whether it produces the correct textual decision. Representative biomedical models include BioBERT, ClinicalBERT, BioGPT, GatorTron, and specialized Qwen variants (Lee et al. 2020; Alsentzer et al. 2019; Luo et al. 2022; Yang et al. 2022; Devlin et al. 2019; Hui et al. 2024). These efforts answer whether a model is capable under a chosen evaluation environment; they do not fully answer whether the same medical image preserves its confidence and prediction after migration to a different execution environment. We complement them by treating visual deployment consistency as a separate and directly measurable reliability dimension. 2.3 Numerical Variability in Inference Systems Numerical variability is a known property of modern inference systems. Floating-point arithmetic is non-associative, and compiler choices, fused kernels, mixed precision, and parallel reductions can therefore produce different numerical results (Goldberg 1991; Higham 2002; Kahan 1996; Demmel et al. 2016; Ahrens et al. 2020; Micikevicius et al. 2017). At the same time, DeepSpeed, PagedAttention, FlashAttention, tensor/model parallelism, quantization, and related techniques improve throughput and memory efficiency precisely by changing execution order, precision, or operator implementation (Aminabadi et al. 2022; Kwon et al. 2023; Dao et al. 2022; Dao 2024; Shoeybi et al. 2019; Rajbhandari et al. 2020; Narayanan et al. 2019; Yu et al. 2022; Frantar et al. 2022; Dettmers et al. 2022). Recent work further reports inference-engine bugs, distributed-framework issues, hardware-level reliability problems, linear-algebra sensitivity, compiler vulnerabilities, and precision-induced disagreement risks (Liu et al. 2025; Yu et al. 2025; Möller et al. 2026; Möller et al. 2025; Chen et al. 2025; Wang et al. 2026b; Wang et al. 2026a). Our study connects these system-level causes to visual medical outputs: a numerical difference remains harmless when the visual prediction margin is large, but it becomes observable when it changes the leading class or answer associated with the image. 3 Problem Definition 3.1 Definition 1 (Medical Model Execution). A medical model fθf_θ is a sequence of neural operators ⟨L0,L1,…,Ln⟩ L_0,L_1,…,L_n parameterized by weights θ. Given a medical image x, or an image-text pair (x,q)(x,q), the model transforms the visual evidence layer by layer and produces a class-score vector or an answer-token distribution. We denote its execution under a deployment stack ϕφ by fθϕ(x)f_θ^φ(x), where ϕφ specifies the operator kernels, precision policy, tensor partitioning, communication collectives, attention implementation, and reduction order. A deterministic decision rule g(⋅)g(·) then converts the numerical output into an observable visual prediction: yϕ(x)=g(fθϕ(x)).y^φ(x)=g(f_θ^φ(x)). For image classification, g(⋅)g(·) is an argmax over class probabilities; for visual question answering, it is answer-option ranking or autoregressive token selection conditioned on the image. Medical language-only inference is treated as a complementary instance of the same execution model. 3.2 Definition 2 (Deployment-Induced Inconsistency). Consider a medical model fθf_θ, a visual input x, a reference execution stack ϕb _b, and a distributed or optimized execution stack ϕd _d. A deployment-induced inconsistency occurs when yϕb(x)≠yϕd(x),y _b(x)≠ y _d(x), while the checkpoint, image content, prompt template, and decision policy remain identical. Thus, the visual evidence is unchanged and the model is unchanged; only the execution stack differs. This definition is different from ordinary model error: it asks whether the same medical image receives a consistent prediction across deployments, not whether either prediction is clinically correct in isolation. 3.3 Definition 3 (Deployment-Sensitive Test Case). A deployment-sensitive test case is a tuple (fθ,x,ϕb,ϕd)(f_θ,x, _b, _d) for which the same medical image or image-text pair produces different predictions under the two execution stacks. For our primary visual and multimodal setting, we generate a bounded perturbation δ that preserves the image semantics and the reference prediction but changes the distributed prediction: yϕb(x+δ)=yϕb(x),yϕb(x+δ)≠yϕd(x+δ).y _b(x+δ)=y _b(x), y _b(x+δ)≠ y _d(x+δ). The first condition preserves the behavior validated under the reference execution; the second condition exposes a deployment-specific prediction change. For the complementary language setting, we analogously generate an invisible suffix s such that the visible prompt remains unchanged but the two executions select different answer options: yϕb(x⊕s)≠yϕd(x⊕s).y _b(x s)≠ y _d(x s). 3.4 Problem Condition. A numerical difference becomes an observable visual inconsistency only when it crosses a prediction boundary. Distributed execution can introduce small logit changes through non-associative floating-point computation, kernel fusion, mixed-precision casting, tensor-parallel reductions, communication collectives, or memory-efficient attention. When the margin between the leading visual classes or candidate answers is large, these changes remain internal; when the margin is small, the same changes can alter the argmax class or the image-conditioned answer. The key condition is therefore the interaction between execution-level numerical variation and a discrete visual decision rule. Following this observation, our primary test setting uses bounded image perturbations to expose single-card versus multi-card disagreement, while the language setting serves as a complementary test of the same condition. 4 Approach We develop two complementary methods, VDST and TDST, to test deployment-induced disagreements in visual and textual medical models, respectively. 4.1 Visual Deployment-Sensitive Testing We first introduce Visual Deployment-Sensitive Testing (VDST). VDST formulates visual deployment discrepancy as a paired testing problem in which the same model is executed in two environments: a single-card HuggingFace-style reference and a distributed multi-card deployment. The visual benchmark covers conventional vision models and medical VLMs under multiple framework and GPU configurations; the complete experimental matrix is specified in Section 5.1. Across all visual tasks, the checkpoint, prompt, and decoding rule are fixed, so VDST isolates the execution configuration. Algorithm 1 summarizes the VDST generation loop. Algorithm 1 VDST Test Generation 1: Input image x, ground-truth label/answer y, single-card model fsf_s, multi-card model fmf_m, perturbation budget ϵε, step size α, divergence weight λ, regularization weight β, maximum steps T 2: Test image x′x and success flag 3: Initialize δ∼(−ϵ/4,ϵ/4)δ (-ε/4,ε/4) 4: for t=1t=1 to T do 5: x′←clip(x+δ)x (x+δ) 6: Compute single-card prediction y^s←fs(x′) y_s← f_s(x ) 7: Compute multi-card prediction y^m←fm(x′) y_m← f_m(x ) 8: if y^s=y y_s=y and y^m≠y y_m≠ y then 9: return x′x , success 10: end if 11: Compute ℒsingle(x′,y)L_single(x ,y) on the single-card path 12: Compute ℒmulti(x′,y)L_multi(x ,y) on the multi-card path 13: ℒ←ℒsingle−λℒmulti+β‖δ‖22L _single- _multi+β\|δ\|_2^2 14: Update δ←δ−α∇δℒδ←δ-α _δL 15: Project δ back to the L∞L_∞ ball: δ←clip(δ,−ϵ,ϵ)δ (δ,-ε,ε) 16: end for 17: return clip(x+δ)clip(x+δ), failure For each input image, VDST searches for an additive perturbation δ within an L∞L_∞ budget, x′=clip(x+δ),‖δ‖∞≤ϵ,x =clip(x+δ), \|δ\|_∞≤ε, (1) where the implementation uses ϵ=0.03ε=0.03, Adam updates with step size α=0.01α=0.01, and a configured maximum of T optimization steps. The success condition is deliberately asymmetric: the single-card execution must remain correct, whereas the multi-card execution must produce a different prediction. Denoting the single-card predictor by fsf_s, the distributed predictor by fmf_m, and the gold label by y, a successful test satisfies fs(x′)=yandfm(x′)≠y.f_s(x )=y f_m(x )≠ y. (2) The reported success rate (SR) is the proportion of evaluated inputs for which this condition is reached. Thus, the objective is not to measure generic model error, but to reveal an error that appears only after the execution configuration changes. VDST uses the same paired objective for image classification and VQA-style models. For classification, VDST minimizes the single-card cross-entropy and maximizes the multi-card cross-entropy: ℒvis=ℒsingle(x′,y)−λℒmulti(x′,y)+β‖δ‖22,L_vis=L_single(x ,y)- _multi(x ,y)+β\|δ\|_2^2, (3) where λ=1.0λ=1.0 controls execution divergence and β=0.002β=0.002 regularizes the perturbation magnitude. For VLMs, ℒsingleL_single and ℒmultiL_multi are next-token negative log-likelihoods computed against the ground-truth short answer. In both cases, VDST follows three steps: preserve the reference prediction, increase the deployment-side loss, and check whether the two executions disagree. 4.2 Textual Deployment-Sensitive Testing We next introduce Textual Deployment-Sensitive Testing (TDST). TDST targets medical multiple-choice inference by keeping the visible question unchanged, appending an invisible suffix, and searching for a suffix that separates the distributed prediction from the HuggingFace reference. The candidate pool contains invisible Unicode characters, mainly variation selectors and formatting code points. TDST retains only characters with stable single-token encodings, so the visible input remains unchanged while the search space remains discrete and reproducible. We define invisibility operationally rather than typographically. A decoded token is retained only when every character belongs to a Unicode control, format, or separator category, or falls within a standard variation-selector range. Tokens containing null bytes or visible symbols are removed. We also require tokenizer stability: each retained character must map to exactly one token ID and must preserve that mapping under repetition. The admissible test-character pool is test=c∣ _test=\\,c ∀u∈c,cat(u)∈Cc,Cf,Zs,Zl,Zp∨u∈sel, ∀ u∈ c,\ cat(u)∈\Cc,Cf,Zs,Zl,Zp\ u _sel, (4) encode(c)=[t],encode(c∥c)=[t,t],NUL∉c. (c)=[t],\ encode(c\|c)=[t,t],\ NUL∉ c\,\. where cat(u)cat(u) denotes the Unicode category of code point u, selV_sel denotes the set of Unicode variation selectors, and t is a stable single-token ID. This definition satisfies two requirements at once: the suffix remains visually imperceptible to human readers, and it remains explicitly represented in the token sequence processed by the model. TDST can therefore test boundary-sensitive logits without changing the apparent medical question. Figure 2: Details of our proposed method TDST. Let the answer space consist of fixed choice tokens, such as A/B/C/D. TDST first identifies the two highest-probability answers under the reference model. It then reduces the margin between these two answers while preserving their joint dominance over the remaining choices. If p(1)p_(1) and p(2)p_(2) are the normalized probabilities of the top-two answers and p¯rest p_rest is the mean probability of the other choices, the objective is ℒtxt=λeq(p(1)−p(2))2−λdom(p(1)+p(2)2−p¯rest),L_txt= _eq (p_(1)-p_(2) )^2- _dom ( p_(1)+p_(2)2- p_rest ), (5) where the first term equalizes the top two and the second keeps them above the distractors. The resulting TDST input moves the prediction toward a narrow boundary, where a small execution-level difference is more likely to change the selected answer. TDST is gradient-guided but operates in discrete token space. At each iteration, it computes gradients for the current suffix tokens, restricts replacements to the invisible-token pool, selects high-impact candidates from top-k gradient directions, and reruns both reference and distributed executions. TDST stops when the two executions produce different answer letters. Figure 2 details our TDST, covering its optimization objective, invisible-token initialization, and tailored loss function. The visible medical question remains unchanged, while the tokenizer-preserved suffix steers predictions to the boundary where the two executions yield different answers. VDST works in image space, whereas TDST operates in token space; the two procedures are symmetric in purpose and both reveal deployment-induced disagreement. 5 Experiments We conduct extensive experiments and organize the evaluation around four research questions: • RQ1: Natural Discrepancy. Does the same medical model naturally produce different outputs across deployment frameworks? • RQ2: Language Model Testing. How effectively can TDST expose deployment disagreement in medical language models? • RQ3: Vision and Multimodal Testing. How effectively can VDST expose deployment disagreement in medical vision and multimodal models? • RQ4: Localization and Precision Elevation. Where are numerical differences amplified, and how effectively can critical-layer precision elevation reduce the disagreement? 5.1 Experimental Setting Benchmark protocol. Our benchmark is designed to measure the mismatch between offline evaluation and distributed deployment. For each test case, HuggingFace provides the centralized evaluation reference, while a distributed framework provides the deployment execution. The checkpoint, input, prompt template, and decision rule are held fixed. Consequently, the comparison changes the execution stack rather than the medical content, allowing the measured disagreement to be attributed to the evaluation-deployment transition. Language track. The language track uses TDST to evaluate four models: Llama-3.2-3B-Instruct, Qwen2.5-3B-Instruct, HuatuoGPT2-7B, and Qwen3-8B on MedMCQA and PubMedQA. We compare HuggingFace with DeepSpeed and vLLM, and all distributed runs use two GPUs. This yields a symmetric design across four models, two datasets, and two deployment frameworks under one fixed GPU configuration. Vision and multimodal track. The visual track uses VDST to evaluate five models under two-GPU and four-GPU execution. ResNet-18 and RCLIP are evaluated on ROCO, while BLIP2-OPT-2.7B, Qwen2.5-VL-7B-Instruct, and Qwen3-VL-8B-Instruct are evaluated on VQA-RAD. The deployment variants include DeepSpeed, FSDP, vLLM, and TensorRT-LLM. ResNet-18 and RCLIP are reported only for DeepSpeed and FSDP because they are not supported by vLLM and TensorRT-LLM in our setup. Table 1 summarizes the complete supported model-framework-GPU matrix. Metrics and mitigation protocol. We report Success Rate (SR), the proportion of test inputs for which the benchmark exposes a prediction disagreement, and Avg. Step, the average search steps among successful cases. Localization and mitigation are conducted on the visual track of the benchmark. 5.2 RQ1: Natural Discrepancy Across Deployment Frameworks Figure 3: Natural output discrepancy between the standard non-distributed configuration and the two-GPU DeepSpeed configuration on MedQA and MedQuAD. We first measure disagreement on unmodified medical questions, without constructed perturbations or optimization-based test generation. We evaluate HuatuoGPT2-7B on three medical QA corpora and compare a standard non-distributed execution with a two-GPU DeepSpeed execution. The checkpoint, question, prompt construction, and decoding settings remain unchanged; only the deployment framework changes. This paired setup makes the interpretation direct: an output difference is an execution disagreement, not a consequence of different model weights or different question semantics. As summarized in Figure 3, the natural disagreements are frequent and consistent across datasets. On the MedQA-derived US subset, 3,571 of 14,369 samples differ, giving a discrepancy rate of 24.85%. On the MedQA-derived Chinese subset, 8,838 of 34,253 samples differ, giving a similar rate of 25.80%. Together, the two MedQA subsets contain 12,409 disagreements among 48,622 samples, or 25.52% overall. The webMedQA split corresponding to MedQuAD shows a larger difference: 7,517 of 16,407 samples disagree, yielding 45.82%. Thus, the two MedQA subsets exhibit nearly symmetric discrepancy rates, whereas the MedQuAD setting is approximately 20 percentage points higher. In summary, deployment disagreement is not limited to specially generated boundary cases. It already appears at scale on ordinary medical questions, and it appears under a simple change from centralized to distributed execution. A disagreement does not by itself identify which output is clinically correct, but it does show that the validated behavior is not invariant to deployment. RQ1 therefore motivates systematic execution-aware testing before a medical model is released. Model DeepSpeed FSDP vLLM TensorRT-LLM SR Step SR Step SR Step SR Step 2-GPU Execution ResNet-18 0.27 35.67 0.22 41.74 - - - - RCLIP 0.41 100.63 0.38 86.06 - - - - BLIP2-OPT-2.7B 0.51 70.90 0.47 65.81 0.98 16.72 0.96 23.65 Qwen2.5-VL-7B-Instruct 0.33 72.48 0.42 31.40 0.93 22.57 0.47 156.16 Qwen3-VL-8B-Instruct 0.33 66.73 0.65 46.63 0.81 40.86 0.81 29.95 4-GPU Execution ResNet-18 0.24 37.88 0.21 44.77 - - - - RCLIP 0.43 96.81 0.42 93.79 - - - - BLIP2-OPT-2.7B 0.51 70.90 0.56 79.52 0.97 18.52 0.94 26.19 Qwen2.5-VL-7B-Instruct 0.32 66.09 0.45 35.73 0.92 17.70 0.48 170.14 Qwen3-VL-8B-Instruct 0.36 73.64 0.66 47.56 0.85 38.47 0.82 32.57 Table 1: Cross-framework evaluation under 2-GPU and 4-GPU execution. Each framework reports the Success Rate (SR) and Avg. Step. Note that ResNet and RCLIP are not supported by vLLM and TensorRT-LLM. 5.3 RQ2: Testing Medical Language Models Following the language-track setting in Section 5.1, we next evaluate whether TDST can expose the same inconsistency in medical language models. Figure 5 compares the four models on MedMCQA and PubMedQA under the two-GPU DeepSpeed and vLLM deployments using TDST. The success rate is the proportion of questions for which TDST finds a disagreement within the configured search budget, and Avg. Step is the mean number of steps required for successful cases. The DeepSpeed results show a clear separation between models. Llama-3.2-3B-Instruct reaches 53.33% SR on MedMCQA and 46.67% on PubMedQA, requiring 41.13 and 57.14 steps on average, respectively. In contrast, Qwen2.5-3B-Instruct, HuatuoGPT2-7B, and Qwen3-8B all reach 100% SR on both datasets. Among them, Qwen2.5-3B-Instruct is the easiest to test, requiring only 8.57 steps on MedMCQA and 10.70 steps on PubMedQA; HuatuoGPT2-7B and Qwen3-8B also remain below 15 average steps in all four settings. The vLLM results preserve the same model-level pattern but change the search difficulty. Llama-3.2-3B-Instruct reaches 43.33% SR on both datasets, with 40.08 and 51.23 average steps. Qwen2.5-3B-Instruct reaches 93.33% on MedMCQA and 100% on PubMedQA, while HuatuoGPT2-7B and Qwen3-8B reach 100% on both datasets. HuatuoGPT2-7B requires only 11.43-12.20 steps, whereas Qwen3-8B requires 21.30-25.60 steps under vLLM despite the same 100% SR. In summary, TDST succeeds broadly across both frameworks, but SR and search cost are jointly determined by the model, dataset, and runtime rather than by any one factor alone. Figure 4 gives a concrete multiple-choice example. The invisible suffix narrows the margin between the leading candidate answers, after which HuggingFace and the distributed execution select different options. Thus, RQ2 has an affirmative answer: deployment disagreement in medical LLMs can be systematically exposed by TDST, and the required search effort can be measured directly. Figure 4: A TDST-generated medical multiple-choice test that produces different answers under HuggingFace and distributed execution. Figure 5: Two-GPU divergence results between HuggingFace and the DeepSpeed and vLLM inference frameworks across four medical language models. Panel (a) reports the success rate of inducing output disagreement, while Panel (b) shows the average number of optimization steps required. Results are evaluated on MedMCQA and PubMedQA. 5.4 RQ3: Testing Medical Vision and Multimodal Models Following the visual-track setting in Section 5.1, we then evaluate whether VDST exposes deployment disagreement in medical vision and multimodal models. Table 1 reports the complete supported comparison under two-GPU and four-GPU execution, with HuggingFace as the centralized reference and DeepSpeed, FSDP, vLLM, and TensorRT-LLM as deployment variants. The results show both breadth and variation. For single-modality vision, we use ROCO with ResNet-18 and RCLIP; for multimodal inference, we use VQA-RAD with BLIP2-OPT-2.7B, Qwen2.5-VL-7B-Instruct, and Qwen3-VL-8B-Instruct. Across these settings, VDST exposes confidence shifts and prediction disagreements under distributed execution. Because the image, checkpoint, prompt, and evaluation protocol are fixed within each comparison, the measured difference is attributable to the serving stack rather than to random data variation. Three quantitative patterns are visible in Table 1. First, the single-modality models have comparatively moderate SRs: ResNet-18 ranges from 0.21 to 0.27 and RCLIP from 0.38 to 0.43 across DeepSpeed and FSDP. Second, the multimodal models are substantially more sensitive under several optimized runtimes. Under two-GPU execution, BLIP2-OPT-2.7B reaches 0.98 SR on vLLM and 0.96 on TensorRT-LLM, Qwen2.5-VL-7B-Instruct reaches 0.93 on vLLM, and Qwen3-VL-8B-Instruct reaches 0.81 on both vLLM and TensorRT-LLM. These high SRs are often accompanied by low search cost: BLIP2 requires 16.72 steps on vLLM and 23.65 on TensorRT-LLM. By contrast, Qwen2.5-VL reaches only 0.47 SR on TensorRT-LLM and requires 156.16 steps, showing that successful test generation can be both less frequent and more expensive in a difficult configuration. Third, increasing the execution scale from two GPUs to four GPUs does not remove the pattern. BLIP2 changes only from 0.98 to 0.97 on vLLM and from 0.96 to 0.94 on TensorRT-LLM; Qwen2.5-VL changes from 0.93 to 0.92 on vLLM; and Qwen3-VL changes from 0.81 to 0.85 on vLLM. The exact values shift, but the framework-specific sensitivity remains. RQ3 is therefore answered affirmatively: VDST exposes execution disagreement in both medical vision and multimodal models, while its magnitude and detection cost depend jointly on model architecture, serving framework, and GPU configuration. Model Success Rate (SR) ↓ Deep Speed FSDP vLLM TensorRT- LLM ResNet-18 0.00 0.00 - - RCLIP 0.01 0.00 - - BLIP2-OPT-2.7B 0.19 0.30 0.45 0.07 Qwen2.5-VL-7B 0.13 0.04 0.86 0.46 Qwen3-VL-8B 0.24 0.38 0.17 0.01 Table 2: Cross-framework mitigation performance under 2-GPU execution. We report the post-mitigation success rate (SR) after elevating the localized critical layers to higher precision; lower values indicate stronger mitigation. 5.5 RQ4: Localization and Precision Elevation RQ1-RQ3 show that deployment disagreement exists; RQ4 investigates where numerical differences are amplified and whether a targeted precision change can reduce them. All experiments in this section use the visual track of our benchmark. Layer-wise localization. We compare centralized and distributed executions with the same checkpoint, input, and decoding configuration. These executions may differ numerically because of floating-point non-associativity, mixed precision, kernel fusion, and framework-specific parallel reduction orders. During the forward pass that produces the first generated token, we record the output of every leaf module, since later tokens may already belong to diverged generation trajectories. For layer i, let OirefO_i^ref and OidistO_i^dist denote the reference and distributed activations. We measure their Mean Absolute Difference (MAD) as μi=1N∑j=1N|Oi,jref−Oi,jdist|, _i= 1N _j=1^N |O_i,j^ref-O_i,j^dist |, (6) where N is the number of tensor elements. To identify layers that amplify rather than merely inherit divergence, we define the Relative Divergence Lift (RL): RLi=μi−maxk<iμkmaxk<iμk+ε.RL_i= _i- _k<i _k _k<i _k+ . (7) Critical layers are selected using =i∣RLi>Percentile95(RL).C= \i _i>Percentile_95(RL) \. (8) The detected modules are mainly concentrated around numerically sensitive components such as self.rotary_emb and self.norm. Critical-layer precision elevation. We then execute only the detected critical modules in higher precision, such as float32, while leaving all other modules and deployment settings unchanged. This targeted strategy limits additional computation and memory compared with globally increasing model precision. Table 2 reports the post-mitigation SR under the same two-GPU settings as Table 1. Averaged over the 16 supported model-framework pairs, SR decreases from 0.56 to 0.21. The reduction is especially strong for ResNet-18 and RCLIP, whose SR approaches zero, and for several multimodal settings, such as Qwen3-VL on TensorRT-LLM. However, substantial residual disagreement remains in some configurations, particularly Qwen2.5-VL on vLLM. Overall, RQ4 shows that execution differences are disproportionately amplified by a small set of precision-sensitive modules. Selectively elevating their precision provides an effective minimal-change mitigation, although its effectiveness remains model- and framework-dependent. 6 Conclusion In this paper, we address the mismatch between centralized evaluation and distributed deployment for medical models by introducing a distributed sensitive medical-model benchmark. The benchmark treats evaluation and deployment as a paired system and measures whether model behavior remains consistent when only the execution stack changes. Extensive experiments across language, vision, and multimodal models show that distributed frameworks and GPU configurations can induce observable prediction disagreements, even when the checkpoint, input, and decoding policy are fixed. To further understand and mitigate these discrepancies, we localize the layers that amplify execution differences and selectively improve their numerical precision. This targeted strategy reduces disagreement while avoiding unnecessary changes to the entire model. Overall, our results demonstrate that deployment configurations are not behaviorally neutral and that evaluation-deployment consistency should be treated as an explicit criterion in medical-model assessment. References Ahrens et al. (2020) W. Ahrens, J. Demmel, and H. D. Nguyen Algorithms for efficient reproducible floating point summation. ACM Transactions on Mathematical Software (TOMS) 46 (3), p. 1–49. Cited by: §2.3. Alsentzer et al. (2019) E. Alsentzer, J. Murphy, W. Boag, W. Weng, D. Jindi, T. Naumann, and M. McDermott Publicly available clinical bert embeddings. In Proceedings of the 2nd clinical natural language processing workshop, p. 72–78. Cited by: §2.2. Aminabadi et al. (2022) R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang, J. Rasley, et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, p. 1–15. Cited by: §1, §2.3. Ben Abacha and Demner-Fushman (2019) A. Ben Abacha and D. Demner-Fushman A question-entailment approach to question answering. BMC bioinformatics 20 (1), p. 511. Cited by: §1, §2.2. Chen et al. (2023) J. Chen, X. Wang, K. Ji, A. Gao, F. Jiang, S. Chen, H. Zhang, D. Song, W. Xie, C. Kong, et al. Huatuogpt-i, one-stage training for medical adaption of llms. arXiv preprint arXiv:2311.09774. Cited by: §2.2. Chen et al. (2025) S. Chen, J. Peng, Y. He, J. Yang, and B. Ray Your compiler is backdooring your model: understanding and exploiting compilation inconsistency vulnerabilities in deep learning compilers. arXiv preprint arXiv:2509.11173. Cited by: §2.3. Chen et al. (2018) T. Y. Chen, F. Kuo, H. Liu, P. Poon, D. Towey, T. Tse, and Z. Q. Zhou Metamorphic testing: a review of challenges and opportunities. ACM Computing Surveys (CSUR) 51 (1), p. 1–27. Cited by: §2.1. Dao et al. (2022) T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré Flashattention: fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems 35, p. 16344–16359. Cited by: §2.3. Dao (2024) T. Dao Flashattention-2: faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, Vol. 2024, p. 35549–35562. Cited by: §2.3. Demmel et al. (2016) J. Demmel, P. Ahrens, and H. D. Nguyen Efficient reproducible floating point summation and blas. EECS Department, University of California, Berkeley, Tech. Rep. UCB/EECS-2016-121. Cited by: §2.3. Dettmers et al. (2022) T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems 35, p. 30318–30332. Cited by: §2.3. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), p. 4171–4186. Cited by: §2.2. Dwarakanath et al. (2018) A. Dwarakanath, M. Ahuja, S. Sikand, R. M. Rao, R. J. C. Bose, N. Dubash, and S. Podder Identifying implementation bugs in machine learning based image classifiers using metamorphic testing. In Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, p. 118–128. Cited by: §2.1. Frantar et al. (2022) E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh Gptq: accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. Cited by: §2.3. Goldberg (1991) D. Goldberg What every computer scientist should know about floating-point arithmetic. ACM computing surveys (CSUR) 23 (1), p. 5–48. Cited by: §2.3. Guo et al. (2018) J. Guo, Y. Jiang, Y. Zhao, Q. Chen, and J. Sun Dlfuzz: differential fuzzing testing of deep learning systems. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, p. 739–743. Cited by: §2.1. Guo et al. (2019) Q. Guo, S. Chen, X. Xie, L. Ma, Q. Hu, H. Liu, Y. Liu, J. Zhao, and X. Li An empirical study towards characterizing deep learning development and deployment across different frameworks and platforms. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), p. 810–822. Cited by: §2.1. Guo et al. (2020) Q. Guo, X. Xie, Y. Li, X. Zhang, Y. Liu, X. Li, and C. Shen Audee: automated testing for deep learning frameworks. In Proceedings of the 35th IEEE/ACM international conference on automated software engineering, p. 486–498. Cited by: §2.1. Higham (2002) N. J. Higham Accuracy and stability of numerical algorithms. SIAM. Cited by: §2.3. Hui et al. (2024) B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. Cited by: §2.2. Humbatova et al. (2020) N. Humbatova, G. Jahangirova, G. Bavota, V. Riccio, A. Stocco, and P. Tonella Taxonomy of real faults in deep learning systems. In Proceedings of the ACM/IEEE 42nd international conference on software engineering, p. 1110–1121. Cited by: §2.1. Jin et al. (2021) D. Jin, E. Pan, N. Oufattole, W. Weng, H. Fang, and P. Szolovits What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences 11 (14), p. 6421. Cited by: §1, §2.2. Jin et al. (2019) Q. Jin, B. Dhingra, Z. Liu, W. Cohen, and X. Lu Pubmedqa: a dataset for biomedical research question answering. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), p. 2567–2577. Cited by: §1, §2.2. Kahan (1996) W. Kahan IEEE standard 754 for binary floating-point arithmetic. Lecture Notes on the Status of IEEE 754 (94720-1776), p. 11. Cited by: §2.3. Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, p. 611–626. Cited by: §1, §2.3. Lee et al. (2020) J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36 (4), p. 1234–1240. Cited by: §2.2. Li et al. (2023a) C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao Llava-med: training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36, p. 28541–28564. Cited by: §2.2. Li et al. (2023b) J. Li, D. Li, S. Savarese, and S. Hoi Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, p. 19730–19742. Cited by: §1, §2.2. Liu et al. (2025) M. Liu, S. Zhong, W. Bi, Y. Zhang, Z. Chen, Z. Chen, X. Liu, and Y. Ma A first look at bugs in llm inference engines. ACM Transactions on Software Engineering and Methodology. Cited by: §2.3. Liu et al. (2023) Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, K. Wang, and Y. Liu Jailbreaking chatgpt via prompt engineering: an empirical study. arXiv preprint arXiv:2305.13860. Cited by: §1, §2.1. Luo et al. (2022) R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T. Liu BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings in bioinformatics 23 (6), p. bbac409. Cited by: §2.2. Ma et al. (2018) L. Ma, F. Zhang, J. Sun, M. Xue, B. Li, F. Juefei-Xu, C. Xie, L. Li, Y. Liu, J. Zhao, et al. Deepmutation: mutation testing of deep learning systems. In 2018 IEEE 29th international symposium on software reliability engineering (ISSRE), p. 100–111. Cited by: §2.1. Micikevicius et al. (2017) P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740. Cited by: §2.3. Möller et al. (2026) J. Möller, E. Imgrund, T. Eisenhofer, and K. Rieck Hardware-triggered backdoors. arXiv preprint arXiv:2601.21902. Cited by: §2.3. Möller et al. (2025) J. Möller, L. Pirch, F. Weissberg, S. Baunsgaard, T. Eisenhofer, and K. Rieck Adversarial inputs for linear algebra backends. In Forty-second International Conference on Machine Learning, Cited by: §2.3. Murphy et al. (2008) C. Murphy, G. E. Kaiser, and L. Hu Properties of machine learning applications for use in metamorphic testing. Cited by: §2.1. Narayanan et al. (2019) D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia PipeDream: generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM symposium on operating systems principles, p. 1–15. Cited by: §2.3. Odena et al. (2019) A. Odena, C. Olsson, D. Andersen, and I. Goodfellow Tensorfuzz: debugging neural networks with coverage-guided fuzzing. In International conference on machine learning, p. 4901–4911. Cited by: §2.1. Pal et al. (2022) A. Pal, L. K. Umapathi, and M. Sankarasubbu Medmcqa: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, p. 248–260. Cited by: §1, §2.2. Pei et al. (2017) K. Pei, Y. Cao, J. Yang, and S. Jana Deepxplore: automated whitebox testing of deep learning systems. In proceedings of the 26th Symposium on Operating Systems Principles, p. 1–18. Cited by: §2.1. Pham et al. (2019) H. V. Pham, T. Lutellier, W. Qi, and L. Tan CRADLE: cross-backend validation to detect and localize bugs in deep learning libraries. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), p. 1027–1038. Cited by: §2.1. Pope et al. (2023) R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean Efficiently scaling transformer inference. Proceedings of machine learning and systems 5, p. 606–624. Cited by: §1. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: §1, §2.2. Rajbhandari et al. (2020) S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He Zero: memory optimizations toward training trillion parameter models. In SC20: international conference for high performance computing, networking, storage and analysis, p. 1–16. Cited by: §2.3. Riccio et al. (2020) V. Riccio, G. Jahangirova, A. Stocco, N. Humbatova, M. Weiss, and P. Tonella Testing machine learning based systems: a systematic mapping. Empirical Software Engineering 25 (6), p. 5193–5254. Cited by: §2.1. Rückert et al. (2024) J. Rückert, L. Bloch, R. Brüngel, A. Idrissi-Yaghir, H. Schäfer, C. S. Schmidt, S. Koitka, O. Pelka, A. B. Abacha, A. G. Seco de Herrera, et al. Rocov2: radiology objects in context version 2, an updated multimodal image dataset. Scientific Data 11 (1), p. 688. Cited by: §2.2. Shoeybi et al. (2019) M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro Megatron-lm: training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053. Cited by: §2.3. Singhal et al. (2023) K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al. Large language models encode clinical knowledge. Nature 620 (7972), p. 172–180. Cited by: §2.2. Singhal et al. (2025) K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, M. Amin, L. Hou, K. Clark, S. R. Pfohl, H. Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature medicine 31 (3), p. 943–950. Cited by: §1, §2.2. Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §1. Wallace et al. (2019) E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh Universal adversarial triggers for attacking and analyzing nlp. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), p. 2153–2162. Cited by: §2.1. Wang et al. (2024) J. Wang, H. V. Pham, Q. Li, L. Tan, Y. Guo, A. Aziz, and E. Meijer D 3: differential testing of distributed deep learning with model generation. IEEE Transactions on Software Engineering 51 (1), p. 38–52. Cited by: §2.1. Wang et al. (2026a) Y. Wang, T. Li, X. Zhang, Y. Yang, X. Zhang, and L. Pan Trusted weights, treacherous optimizations? optimization-triggered backdoor attacks on llms. arXiv preprint arXiv:2605.20641. Cited by: §2.3. Wang et al. (2026b) Y. Wang, T. Li, X. Zhang, X. Zhang, W. Ma, M. Cheng, and L. Pan Hidden reliability risks in large language models: systematic identification of precision-induced output disagreements. arXiv preprint arXiv:2604.19790. Cited by: §2.3. Wei et al. (2023) A. Wei, N. Haghtalab, and J. Steinhardt Jailbroken: how does llm safety training fail?. Advances in neural information processing systems 36, p. 80079–80110. Cited by: §1, §2.1. Xie et al. (2019) X. Xie, L. Ma, H. Wang, Y. Li, Y. Liu, and X. Li Diffchaser: detecting disagreements for deep neural networks. Cited by: §2.1. Yang et al. (2022) X. Yang, A. Chen, N. PourNejatian, H. C. Shin, K. E. Smith, C. Parisien, C. Compas, C. Martin, A. B. Costa, M. G. Flores, et al. A large language model for electronic health records. NPJ digital medicine 5 (1), p. 194. Cited by: §2.2. Yu et al. (2022) G. Yu, J. S. Jeong, G. Kim, S. Kim, and B. Chun Orca: a distributed serving system for \transformer-based\ generative models. In 16th USENIX symposium on operating systems design and implementation (OSDI 22), p. 521–538. Cited by: §2.3. Yu et al. (2025) X. Yu, H. Chen, F. Niu, X. Hu, J. W. Keung, and X. Xia Towards understanding bugs in distributed training and inference frameworks for large language models. arXiv preprint arXiv:2506.10426. Cited by: §2.3. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36, p. 46595–46623. Cited by: §1, §2.2. Zou et al. (2023) A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. Cited by: §2.1.