Paper deep dive

To Reason or Not to: Selective Chain-of-Thought in Medical Question Answering

Zaifu Zhan, Min Zeng, Shuang Zhou, Yiran Song, Xiaoyi Chen, Yu Hou, Yifan Wu, Yang Ruan, Rui Zhang

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 36

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 7/20/2026, 3:04:45 PM

Summary

The paper introduces Selective Chain-of-Thought (Selective CoT), an inference-time strategy for medical question answering that dynamically decides whether to generate explicit reasoning rationales. Evaluated on Llama-3.1-8B and Qwen-2.5-7B across four biomedical benchmarks (HeadQA, MedQA-USMLE, MedMCQA, PubMedQA), Selective CoT reduces inference time by 13-45% and token usage by 8-47% with minimal accuracy loss, outperforming fixed-length CoT in efficiency while maintaining or improving accuracy.

Entities (9)

Selective Chain-of-Thought · method · 99%LLaMA-3.1-8B · model · 95%Qwen-2.5-7B · model · 95%HeadQA · dataset · 92%MedMCQA · dataset · 92%MedQA-USMLE · dataset · 92%PubMedQA · dataset · 92%Chain-of-Thought · method · 90%Nvidia A100 · hardware · 85%

Relation Signals (9)

Selective Chain-of-Thought → reduces → Inference Time

confidence 95% · Selective CoT reduced inference time by 13-45%

Selective Chain-of-Thought → reduces → Token usage

confidence 95% · Selective CoT reduced ... token usage by 8-47%

Selective Chain-of-Thought → appliedto → LLaMA-3.1-8B

confidence 90% · Two open-source LLMs (Llama-3.1-8B and Qwen-2.5-7B) were evaluated

Selective Chain-of-Thought → appliedto → Qwen-2.5-7B

confidence 90% · Two open-source LLMs (Llama-3.1-8B and Qwen-2.5-7B) were evaluated

Selective Chain-of-Thought → evaluatedon → MedMCQA

confidence 90% · evaluated on four biomedical QA benchmarks... MedMCQA

Selective Chain-of-Thought → evaluatedon → HeadQA

confidence 90% · evaluated on four biomedical QA benchmarks-HeadQA

Selective Chain-of-Thought → evaluatedon → MedQA-USMLE

confidence 90% · evaluated on four biomedical QA benchmarks... MedQA-USMLE

Selective Chain-of-Thought → evaluatedon → PubMedQA

confidence 90% · evaluated on four biomedical QA benchmarks... PubMedQA

Selective Chain-of-Thought → outperforms →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Objective: To improve the efficiency of medical question answering (MedQA) with large language models (LLMs) by avoiding unnecessary reasoning while maintaining accuracy. Methods: We propose Selective Chain-of-Thought (Selective CoT), an inference-time strategy that first predicts whether a question requires reasoning and generates a rationale only when needed. Two open-source LLMs (Llama-3.1-8B and Qwen-2.5-7B) were evaluated on four biomedical QA benchmarks-HeadQA, MedQA-USMLE, MedMCQA, and PubMedQA. Metrics included accuracy, total generated tokens, and inference time. Results: Selective CoT reduced inference time by 13-45% and token usage by 8-47% with minimal accuracy loss ($\leq$4\%). In some model-task pairs, it achieved both higher accuracy and greater efficiency than standard CoT. Compared with fixed-length CoT, Selective CoT reached similar or superior accuracy at substantially lower computational cost. Discussion: Selective CoT dynamically balances reasoning depth and efficiency by invoking explicit reasoning only when beneficial, reducing redundancy on recall-type questions while preserving interpretability. Conclusion: Selective CoT provides a simple, model-agnostic, and cost-effective approach for medical QA, aligning reasoning effort with question complexity to enhance real-world deployability of LLM-based clinical systems.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

34,951 characters extracted from source content.

Expand or collapse full text

Journal Title Here, 2022, p. 1–11 doi: DOI HERE Advance Access Publication Date: Day Month Year Paper PAPER To Reason or Not to: Selective Chain-of-Thought in Medical Question Answering Zaifu Zhan, MEng, 1,2 Min Zeng, PhD, 2 Shuang Zhou, PhD, 2 Yiran Song, PhD, 2 Xiaoyi Chen, PhD, 2 Yu Hou, PhD, 2 Yifan Wu, MS, 3 Yang Ruan, MS 4 and Rui Zhang, PhD 2,* 1 Department of Electrical and Computer Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States, 2 Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 420 Delaware St SE, 55455, Minneapolis, MN, United States, 3 Department of Computer Science and Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States and 4 Division of Biostatistics and Health Data Science, School of Public Health, University of Minnesota, 2221 University Ave SE, 55414, Minneapolis, MN, United States ∗ Corresponding author: Dr. Rui Zhang, PhD, Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Office: D528 Mayo building, 420 Delaware St SE, Minneapolis, MN 55455, ruizhang@umn.edu, Office Phone: 612-626-4209 Full paper word count: 2564 Abstract word count: 187 Abstract Objective: To improve the efficiency of medical question answering (MedQA) with large language models (LLMs) by avoiding unnecessary reasoning while maintaining accuracy. Methods: We propose Selective Chain-of-Thought (Selective CoT), an inference-time strategy that first predicts whether a question requires reasoning and generates a rationale only when needed. Two open-source LLMs (Llama-3.1-8B and Qwen-2.5-7B) were evaluated on four biomedical QA benchmarks—HeadQA, MedQA-USMLE, MedMCQA, and PubMedQA. Metrics included accuracy, total generated tokens, and inference time. Results: Selective CoT reduced inference time by 13–45% and token usage by 8–47% with minimal accuracy loss (≤4%). In some model–task pairs, it achieved both higher accuracy and greater efficiency than standard CoT. Compared with fixed-length CoT, Selective CoT reached similar or superior accuracy at substantially lower computational cost. Discussion: Selective CoT dynamically balances reasoning depth and efficiency by invoking explicit reasoning only when beneficial, reducing redundancy on recall-type questions while preserving interpretability. Conclusion: Selective CoT provides a simple, model-agnostic, and cost-effective approach for medical QA, aligning reasoning effort with question complexity to enhance real-world deployability of LLM-based clinical systems. Key words: Question answering, Large language models, Chain-of-thought, Reasoning, Efficiency Introduction Medical question answering (QA) [1], broadly including both clinical and biomedical QA, is an important challenge across medicine and the biomedical sciences [2], as it demands two competencies: precise factual recall of vast, long-tail biomedical knowledge [3] and structured reasoning [4] across clinical context, evidence, and distractors. Traditional NLP pipelines (e.g., keyword-matching search, manually crafted feature-based models, machine learning models, or rule-based © The Author 2022. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com 1 arXiv:2602.20130v1 [cs.CL] 23 Feb 2026 2Zaifu Zhan et al. systems) [5, 6, 7] have struggled to meet this dual requirement: they are brittle to paraphrase and domain shift, offer limited compositionality, and often fail to integrate dispersed cues into coherent, stepwise judgments. The advent of large language models (LLMs) [8, 9, 10] has materially changed this landscape along the two axes most stressed by MedQA: (1) knowledge coverage and access (via broad pretraining, domain adaptation, and retrieval-augmented generation that improve the memorization of rare entities and guidelines) [11, 12, 13, 14] and (2) reasoning capacity (via emergent abilities for multistep inference, uncertainty articulation, and resistance to distractors, further amplified by techniques such as chain-of- thought prompting, self-consistency, and tool use) [15, 16, 17, 18]. Chain-of-thought (CoT) prompting [19], in which a model generates intermediate reasoning steps before giving a final answer, has emerged as a strong default for such settings. CoT can improve logicality and interpretability when questions require multi-step reasoning [20, 21]. Numerous studies have leveraged CoT prompting to improve LLMs on the MedQA benchmark. For example, Jeon et al. [21] conduct a comprehensive evaluation of CoT across both clinical and non-clinical settings, benchmarking GPT- and Gemini-based models. Le et al. [22] tackle contextual medical QA by combining instruction tuning with CoT prompting. Going further, Wang et al. [23] integrate CoT with retrieval- augmented generation (RAG) to further enhance model performance. However, always generating long rationales carries nontrivial costs: more output tokens, higher latency, and increased compute expenditure, especially on knowledge- recall questions where explicit reasoning is unnecessary. This accuracy–efficiency trade-off is acute in realistic deployments (education platforms, decision support), where service-level objectives are tied to throughput and responsiveness as much as to accuracy. To avoid this waste, we introduce Selective Chain-of- Thought (Selective CoT), a simple and effective inference-time routing strategy that first decides whether a question requires explicit reasoning and, only when warranted, generates a rationale before answering. The premise is straightforward: a model capable of reasoning should also be able to recognize when reasoning is needed. In clinical and operational environments, where many queries are recall-oriented, this approach aims to preserve accuracy while reducing token consumption and end-to-end latency by avoiding gratuitous rationales.Conceptually,Selective CoT synthesizes the strengths of standard direct answering (speed) and full CoT (reasoning on hard items), shifting part of orchestration from external control to the model’s own self-selection mechanism. To test this idea, we evaluate Selective CoT across four complementary biomedical QA benchmarks across HeadQA, MedQA-USMLE, MedMCQA, and PubMedQA. Together, this suite blends exam-style and evidence-based QA and supports a comprehensive assessment of factual knowledge, cross- lingual robustness, and interpretive judgment. In addition, our experiments use two strong, accessible open-source LLMs, Llama-3.1-8B and Qwen-2.5-7B, and report a three-way evaluation: answer accuracy, total output tokens, and wall- clock inference time. We fix random seeds, cap generation length to accommodate rationales, and disclose environment details to support reproducibility. Our contributions are summarized as: • To the best of our knowledge, this is among the first studies to explicitly investigate selective reasoning control for improving the efficiency of large language models in medical question answering. • We introduced Selective CoT, a simple, model-agnostic strategy that decides per question whether to generate an explicit rationale, preserving accuracy while reducing tokens and latency relative to always-on CoT. • We provided a comprehensive evaluation. Across HeadQA, MedMCQA, MedQA-USMLE, and PubMedQA with the open-source models Llama-3.1-8B and Qwen-2.5-7B, Selective CoT typically reduces time by 13% to 45% and tokens by 8% to 47% with only small accuracy changes; in some model–task pairs, it is both faster and more accurate. 3 Fig. 1. Overview of Standard Prompt, Chain-of-thought, and Selective CoT prompting (ours) on a representative MedQA case. Left: an example clinical vignette (multiple-choice from MedQA-USMLE dataset). Middle: three inference paradigms—Standard Prompt (answer directly), CoT Prompt (produce step-by-step reasoning before the answer), and Selective CoT Prompt (first decide whether reasoning is necessary; if yes, generate CoT, otherwise answer directly). Right: a comparison illustrating that full CoT improves logicality and interpretability when reasoning is required, but wastes compute on recall-type questions; Selective CoT preserves accuracy while improving efficiency and cost-effectiveness by invoking reasoning only when needed. Methods Overview of methods Task and datasets Medical QA Task. Medical QA is a specialized form of question answering that focuses on understanding, retrieving, and reasoning over medical knowledge to provide accurate and reliable answers to health-related questions. The task requires models to integrate biomedical facts, clinical guidelines, and contextual understanding to answer questions ranging from basic biomedical science to complex clinical diagnosis and treatment scenarios. Unlike general-domain QA, medical QA emphasizes factual correctness, interpretability, and safety, as errors may have significant real-world implications. It serves as a critical benchmark for evaluating the reasoning and domain adaptation capabilities of language models in the healthcare domain, with potential applications in medical education, clinical decision support, and patient-facing information systems. HeadQA dataset [24]. HeadQA is a Spanish multiple-choice medical QA dataset built from official healthcare entrance and specialization exams in Spain. Items span several disciplines (e.g., medicine, nursing, psychology, chemistry, pharmacology, and biology) and reflect authentic exam difficulty. It is commonly used to assess cross-disciplinary medical reasoning and cross-lingual generalization in clinical QA models. MedQA–USMLE dataset [5]. MedQA-USMLE contains multiple-choice questions sourced from the United States Medical Licensing Examination and related professional board materials. The questions emphasize clinical reasoning, pathophysiology, and diagnosis across core rotations (internal medicine, surgery, pediatrics, OB/GYN, psychiatry, etc.). It serves as a benchmark for evaluating a model’s performance on real, high-stakes U.S. medical exam items. MedMCQA dataset [25]. MedMCQA is a large-scale, multi-disciplinary multiple-choice dataset derived from Indian medical entrance and postgraduate exams (e.g., AIIMS, NEET- PG). It covers over 20 subjects and thousands of topics, with carefully designed distractors that require both foundational knowledge and integrated clinical reasoning. The dataset is frequently used to test robustness in multi-subject medical understanding. PubMedQA dataset [26]. PubMedQA targets biomedical literature question answering, where each instance poses a 4Zaifu Zhan et al. research-style question answered as yes/no/maybe. Models must infer the answer from evidence in the linked abstract, encouraging careful reasoning over study summaries. It is suited for evaluating evidence aggregation and causal/interpretive judgment in biomedical QA. Taken together, our four datasets constitute a well-rounded benchmark suite. HeadQA contributes exam questions from Spanish and multi-disciplinary coverage; MedQA–USMLE provides high-stakes U.S. licensure items emphasizing clinical reasoning; MedMCQA adds large-scale, multi-subject questions from the Indian exam ecosystem; and PubMedQA complements exam-style QA with literature-grounded inference over biomedical abstracts. This mix spans styles, regions, and knowledge sources (exams vs. scientific literature), as well as task formats (multiple-choice vs. textual evidence). Consequently, it enables a comprehensive assessment of factual knowledge, cross-source robustness, and evidence-based clinical reasoning—offering both breadth and depth for reliable model evaluation. The dataset statistics and sources are summarized in Table 1. DatasetTestsetSource HeadQA244Spanish medical exams MedQA-USMLE1,273US Medical exam MedMCQA4,183Indian Medical exam PubMedQA500Biomedical literature Table 1. Datasets and their source. We report only test sets, as no training or fine-tuning was performed. Selective Chain-of-thought From the perspective of whether explicit reasoning is required, medical questions can be broadly grouped into (i) knowledge- recall items that can be answered directly from established medical facts (e.g., whether a drug is indicated for a disease), and (i) reasoning items that involve misleading cues, multi- step calculation, temporal ordering, or constraint composition, as shown in Figure 1. These categories align with two inference paradigms: standard prompting, which generates an answer directly, and chain-of-thought (CoT) prompting, which first produces intermediate reasoning and then the final answer. While full CoT often matches or exceeds the accuracy of standard prompting, it incurs substantial computational cost and latency, particularly on knowledge-recall questions where explicit reasoning is unnecessary. To balance accuracy and efficiency, we introduce Selective CoT, as shown in Figure 1, in which the model adaptively decides at inference time whether to invoke CoT. If the question is judged to require reasoning, CoT is enabled; otherwise, the system answers directly. The core premise is that a model capable of reasoning should also be able to recognize when reasoning is needed. In clinical and real-world settings—where many queries are knowledge-recall—Selective CoT preserves accuracy while reducing token consumption and end-to-end latency by avoiding gratuitous reasoning. The method is conceptually simple and synthesizes the strengths of standard and full-CoT prompting. Looking ahead, as large models’ reasoning capabilities continue to improve, the ability to decide whether to reason will become increasingly important, shifting part of the burden from external orchestration to the model’s own self-selection mechanism. Prompt In our experiments, we designed and utilized various prompting strategies to evaluate the reasoning capabilities of the models. As shown in Figure 1, the CoT prompt serves as the baseline, explicitly instructing the models to produce comprehensive and elaborate reasoning processes before providing a final answer. The selective CoT prompt introduces a mechanism for selectively generating reasoning, balancing between detail and efficiency. Additionally, we did experiments with prompts with explicitly specified reasoning lengths: 100 words, 200 words, and so on, where the weights control the level of detail in the reasoning process. These variations allow us to systematically analyze the trade-offs between reasoning depth, computational cost, and answer quality across different prompting settings. All prompts used in our experiments, including CoT and selective CoT, and fixed-length CoT variants, will be released with the code for reproducibility. 5 Models In our experiments, we utilized two open-source large language models: Llama-3.1-8B [27] and Qwen2.5-7B [28, 29]. Both models were selected for their accessibility and strong performance in reasoning tasks, making them suitable for evaluating our framework. The use of open-source models ensures that our experiments can be easily reproduced by others, fostering transparency and enabling further research in selective reasoning and medical question answering. Metrics We evaluated the performance of the models using several standard metrics to ensure a comprehensive assessment. Accuracy was used to measure the proportion of correct answers among all predictions. The total number of tokens (computed via tokenizer) and the total inference time were also recorded to analyze the computational efficiency of the models. These metrics provide a holistic view of both the correctness and efficiency of the reasoning and answering processes. Although we report both token counts and wall-clock time, noting that token usage is a more stable, model-proximal efficiency metric, while latency reflects end-to-end deployment cost. Experiments All experiments were conducted on NVIDIA A100 GPUs with 40 GB of memory to ensure sufficient computational resources for running large language models. We used a temperature of 0.7 for sampling to encourage diverse reasoning paths while maintaining coherence. The maximum generation length was set to 1,024 tokens to accommodate detailed reasoning outputs. For evaluation, we fixed random seeds to ensure reproducibility across runs. The experiments were implemented using Python, with key libraries including PyTorch [30] and Hugging Face Transformers [31], to facilitate efficient model inference and result analysis. Results Main results As shown in Table 2, across four biomedical benchmarks (HeadQA, MedMCQA, MedQA-USMLE, PubMedQA), we compare standard CoT with Selective CoT for Llama-3.1-8B and Qwen2.5-7B. Overall, Selective CoT consistently reduces inference cost—wall-clock time drops by∼13–45% and output tokens often shrink by∼8–47%—with only minor accuracy changes (typically within 0–4%). Notably, Qwen2.5-7B on HeadQA is “faster and more accurate,” improving accuracy by 8.70% while cutting tokens (−19.0%) and time (−17.6%). On MedMCQA, accuracy shifts are negligible (Llama−2.71%, Qwen +0.41%), yet time and token counts fall by∼23– 26%. On MedQA-USMLE, Selective CoT trades small accuracy declines (Llama−0.97%, Qwen−4.40%) for sizable efficiency gains (time−16–29%; tokens−8–14%). On PubMedQA, Qwen attains near-parity in accuracy (−0.55%) with the largest efficiency savings (tokens−46.5%, time−44.5%), whereas Llama sees a modest time reduction (∼13%) but a slight token increase. In sum, Selective CoT typically yields substantial speedups and token savings at minimal accuracy cost, and can even deliver simultaneous gains in both accuracy and efficiency for certain model–task pairs. Ablation Study: Reasoning Length vs. Selective CoT We compare Selective CoT with explicitly controlled fixed- length CoT. Specifically, for Llama-3.1-8B and Qwen-2.5-7B, we evaluate Selective CoT, CoT with approximately 300 words, and CoT with approximately 500 words (Fig. 2). In addition, on Qwen-2.5-7B we sweep the CoT length more finely from 100 to 600 words in 100-word increments and fit a quadratic curve to the “length–performance” relationship, marking Selective CoT as a red point for reference (Fig. 3). Across the four datasets, accuracy under Selective CoT is comparable to that of the 300-word and 500-word CoT baselines; in several cases (HeadQA, USMLE, and PubMedQA), Selective CoT is slightly better, while on MedMCQA it is close to the best fixed-length setting. Crucially, 6Zaifu Zhan et al. Table 2. Selective CoT vs. full CoT: accuracy–efficiency trade-offs. Accuracy, total generated tokens, and end-to-end inference time across datasets and models; “Drop(%)” denotes relative change vs. full CoT (positive means reduction). Acc# TokensTime (s) DatasetModelMethodScoreDrop (%)ValueDrop (%)ValueDrop (%) HeadQA Llama-3.1-8B CoT0.70901355042695.1428 Selective CoT 0.68034.0462136880-1.01552178.343819.1752 Qwen2.5-7B CoT0.6598933522574.1979 Selective CoT 0.7172-8.69577560019.01622120.552017.6228 MedMCQA Llama-3.1-8B CoT0.5826218442445140.2206 Selective CoT 0.56682.708221134193.250533378.016826.0570 Qwen2.5-7B CoT0.5252150120042031.0936 Selective CoT 0.5274-0.4096114850323.494332292.834423.1692 MedQA-USMLE Llama-3.1-8B CoT0.6465103470919077.2906 Selective CoT 0.64020.97219537267.826613487.332929.3016 Qwen2.5-7B CoT0.571163661518494.5302 Selective CoT 0.54604.401754741714.011315517.126616.0988 PubMedQA Llama-3.1-8B CoT0.74803570374610.8540 Selective CoT 0.72802.6738380567-6.59044004.796813.1441 Qwen2.5-7B CoT0.73202073095921.6214 Selective CoT 0.72800.546411090746.50163284.085644.5408 Fig. 2. Performance and efficiency comparison of Selective CoT versus fixed-length CoT across four biomedical QA datasets (HeadQA, MedMCQA, MedQA-USMLE, PubMedQA) and two open-source LLMs (Llama-3.1-8B, Qwen2.5-7B). The three rows report, respectively, Accuracy, #Tokens, and Inference Time. Selective CoT matches or slightly outperforms strong fixed-length CoT baselines while substantially reducing token usage and latency, yielding a superior compute–performance trade-off. Selective CoT achieves these results with a substantially smaller number of Tokens and shorter inference time (see the second and third rows of Fig. 2), yielding a markedly better compute–performance trade-off. The length sweep reveals a non-monotonic pattern: as the fixed CoT length increases from 100 to roughly 200–300 words, accuracy first rises and then declines (the dashed quadratic peak in Fig. 3). Selective CoT lies near this empirical optimum and, on HeadQA, USMLE, and PubMedQA, it sits on or above the fitted CoT curve—achieving equal or higher accuracy with lower cost; on MedMCQA it remains close to the curve’s peak. Overall, Selective CoT forms a superior frontier, maintaining similar (or higher) accuracy while significantly reducing #Tokens and time by triggering extended reasoning 7 Fig. 3. Ablation on reasoning length versus Selective CoT. For fixed-length CoT, we sweep reasoning lengths (e.g., 100–600 words) and fit a dashed quadratic curve to the length–accuracy relationship; Selective CoT is marked as a red point for reference. Accuracy under Selective CoT typically lies near the empirical optimum and often on or above the fitted curve for several datasets, achieving comparable or higher accuracy with fewer tokens and shorter time than long, uniform CoT. only when needed, rather than enforcing a uniform long reasoning length for all examples. Discussion Question answering in biomedicine underpins clinical education, decision support, and patient safety, where models must deliver both correctness and responsiveness under real-world compute and latency constraints. Medical QA items broadly split into recall-type versus reasoning-dependent questions: full CoT can benefit the latter but often wastes tokens and time on the former. Building on this observation, we propose Selective CoT, which decides per example whether explicit reasoning is necessary, aiming to preserve accuracy while reducing inference cost and improving practical deployability. Across four biomedical QA benchmarks (HeadQA, MedMCQA, MedQA-USMLE, PubMedQA) and two open-source LLMs (e.g., Llama-3.1-8B, Qwen2.5-7B), Selective CoT consistently reduced inference cost (wall-clock time and output tokens) with only minor accuracy changes relative to full CoT. On HeadQA and PubMedQA, accuracy matched or slightly exceeded fixed- length CoT (e.g., 300/500 tokens), while on MedMCQA and USMLE, accuracy was comparable to the best fixed-length baselines at substantially lower generation length and latency. Overall, Selective CoT yields a better compute–performance trade-off by generating reasoning traces only when they are likely to help. Further, we compared Selective CoT with fixed-length CoT across a sweep of reasoning lengths and fitted a second-order (quadratic) curve to the length–accuracy relationship. The fitted curve captures the typical rise toward an intermediate optimum followed by saturation or decline. Selective CoT typically lies on or above this empirical curve for most datasets, indicating that example-level on/off routing is more decisive than merely increasing the length of a uniform rationale. In practice, Selective CoT reaches near-peak accuracy without per-task length tuning while using fewer tokens and incurring shorter latency, yielding a more reliable and compute-efficient operating point. One limitation of this study is that the efficiency metric, wall-clock time, is sensitive to backend factors (hardware, system load, batching, runtime). Token usage is a more stable, model-proximal proxy, but end users ultimately care about latency. Accordingly, we report both tokens and time and disclose environment details; while time can be noisy, it still provides a coarse basis for comparison. 8Zaifu Zhan et al. Conclusion We introduced Selective CoT, which is an inference-time routing strategy that first decides whether a question requires explicit reasoning and,only when warranted, generates a rationale before answering. Across four biomedical QA benchmarks (HeadQA, MedMCQA, MedQA–USMLE, PubMedQA) and two open LLMs, Selective CoT preserved—or in some cases improved—accuracy while substantially reducing token usage and end-to-end latency compared to always- on CoT. By aligning the amount of reasoning with item difficulty, Selective CoT delivers interpretable rationales where they matter most and lightweight direct answers where they suffice, offering a practical, cost-efficient path for deployable clinical QA systems under real-world budget and responsiveness constraints. Data Availability The four QA datasets we used are all public: 1. HeadQA: https://aghie.github.io/head-qa/ 2. MedMCQA: https://medmcqa.github.io/ 3. MedQA(USMLE): https://github.com/jind11/MedQA 4. PubmedQA: https://pubmedqa.github.io/ Code Availability We will release the code upon the acceptance of the paper. Acknowledgements We would like to express our sincere gratitude to the reviewers for their valuable comments. Author Contributions Z.Z. conceptualized and led the study design, conducted the literature search and data collection, and was responsible for model development and experiments. Z.Z. and M.Z. drafted the initial manuscript. R.Z. provided supervision throughout the study. Z.Z., M.Z., S.Z., Y.S., X.C., Y.H., Y.W., Y.R., and R.Z. participated in research discussions, contributed feedback on methodology and results, reviewed and edited the manuscript, and approved the final version for submission. Competing Interests The authors declare no competing interests. Funding This work was supported by the National Institutes of Health’s National Center for Complementary and Integrative Health (grant numbers R01AT009457 and U01AT012871), the National Institute on Aging (grant number R01AG078154), the National Cancer Institute (grant number R01CA287413), the National Institute of Diabetes and Digestive and Kidney Diseases (grant number R01DK115629), and the National Institute on Minority Health and Health Disparities (grant number 1R21MD019134-01). References 1. Qiao Jin, Zheng Yuan, Guangzhi Xiong, Qianlan Yu, Huaiyuan Ying, Chuanqi Tan, Mosha Chen, Songfang Huang, Xiaozhong Liu, and Sheng Yu. Biomedical question answering: a survey of approaches and challenges. ACM Computing Surveys (CSUR), 55(2):1–36, 2022. 2. Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze.Benchmarking large language models on answering and explaining challenging medical questions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3563–3599, 2025. 3. Rui Yang, Haoran Liu, Edison Marrese-Taylor, Qingcheng Zeng, Yuhe Ke, Wanxin Li, Lechao Cheng, Qingyu Chen, James Caverlee, Yutaka Matsuo, and Irene Li. KG-rank: Enhancing large language models for medical QA with knowledge graphs and ranking techniques. In Dina Demner-Fushman, Sophia Ananiadou, Makoto Miwa, 9 Kirk Roberts, and Junichi Tsujii, editors, Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, pages 155–166, Bangkok, Thailand, August 2024. Association for Computational Linguistics. 4. Shuang Zhou, Wenya Xie, Jiaxi Li, Zaifu Zhan, Meijia Song, Han Yang, Cheyenna Espinoza, Lindsay Welton, Xinnie Mai, Yanwei Jin, et al. Automating expert-level medical reasoning evaluation of large language models.arXiv preprint arXiv:2507.07988, 2025. 5. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits.What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021. 6. Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang.Benchmarking retrieval-augmented generation for medicine.In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 6233– 6251, Bangkok, Thailand, August 2024. Association for Computational Linguistics. 7. Abdelrahman Abdallah, Mahmoud Kasem, Mohamed A Hamada, and Shaymaa Sdeek.Automated question- answer medical model based on deep learning technology. In Proceedings of the 6th International Conference on Engineering & MIS 2020, pages 1–8, 2020. 8. Shuang Zhou, Zidu Xu, Mian Zhang, Chunpu Xu, Yawen Guo, Zaifu Zhan, Yi Fang, Sirui Ding, Jiashuo Wang, Kaishuai Xu, et al. Large language models for disease diagnosis: A scoping review. npj Artificial Intelligence, 1(1):9, 2025. 9. Zaifu Zhan, Shuang Zhou, Huixue Zhou, Jiawen Deng, Yu Hou, Jeremy Yeung, and Rui Zhang. An evaluation of deepseek models in biomedical natural language processing. arXiv preprint arXiv:2503.00624, 2025. 10. Yu Hou, Zaifu Zhan, and Rui Zhang. Benchmarking gpt-5 for biomedical natural language processing. arXiv preprint arXiv:2509.04462, 2025. 11. Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Massimo Hong, Yushuai Wu, Mu Qiao, and Zaiqing Nie. Biomedgpt: An open multimodal large language model for biomedicine. IEEE Journal of Biomedical and Health Informatics, 2024. 12. Wei Lan, Zhentao Tang, Mingyang Liu, Qingfeng Chen, Wei Peng, Yiping Phoebe Chen, and Yi Pan. The large language models on biomedical data analysis: a survey.IEEE Journal of Biomedical and Health Informatics, 2025. 13. Soumen Pal, Manojit Bhattacharya, Sang-Soo Lee, and Chiranjib Chakraborty. A domain-specific next-generation large language model (llm) or chatgpt is required for biomedical engineering and research. Annals of biomedical engineering, 52(3):451–454, 2024. 14. Sina Shool, Sara Adimi, Reza Saboori Amleshi, Ehsan Bitaraf, Reza Golpira, and Mahmood Tara. A systematic review of large language model (llm) evaluations in clinical medicine.BMC Medical Informatics and Decision Making, 25(1):117, 2025. 15. Wenxuan Wang, Zizhan Ma, Meidan Ding, Shiyi Zheng, Shengyuan Liu, Jie Liu, Jiaming Ji, Wenting Chen, Xiang Li, Linlin Shen, et al. Medical reasoning in the era of llms: A systematic review of enhancement techniques and applications. arXiv preprint arXiv:2508.00669, 2025. 16. Mary M Lucas, Justin Yang, Jon K Pomeroy, and Christopher C Yang. Reasoning with large language models for medical question answering. Journal of the American Medical Informatics Association, 31(9):1964–1975, 2024. 17. Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S. Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. Mediq: Question-asking llms and a benchmark for reliable interactive clinical reasoning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 28858–28888. Curran Associates, Inc., 2024. 18. Minbyul Jeong, Jiwoong Sohn, Mujeen Sung, and Jaewoo Kang.Improving medical reasoning through retrieval 10Zaifu Zhan et al. and self-reflection with retrieval-augmented large language models.Bioinformatics, 40(Supplement1):i119–i129, 2024. 19. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models.In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc., 2022. 20. Valentin Li ́evin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther. Can large language models reason about medical questions? Patterns, 5(3), 2024. 21. Sohyeon Jeon and Hong-Gee Kim.A comparative evaluation of chain-of-thought-based prompt engineering techniques for medical question answering. Computers in Biology and Medicine, 196:110614, 2025. 22. Chenqian Le, Ziheng Gong, Chihang Wang, Haowei Ni, Panfeng Li, and Xupeng Chen. Instruction tuning and cot prompting for contextual medical qa with llms. arXiv preprint arXiv:2506.12182, 2025. 23. Ziyu Wang, Elahe Khatibi, and Amir M Rahmani. Medcot- rag: Causal chain-of-thought rag for medical question answering. arXiv preprint arXiv:2508.15849, 2025. 24. David Vilares and Carlos G ́omez-Rodr ́ıguez. HEAD-QA: A healthcare dataset for complex reasoning.In Anna Korhonen, David Traum, and Llu ́ıs M`arquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 960–966, Florence, Italy, July 2019. Association for Computational Linguistics. 25. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu.Medmcqa: A large-scale multi-subject multi-choicedatasetformedicaldomainquestion answering.In Conference on health, inference, and learning, pages 248–260. PMLR, 2022. 26. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu.Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146, 2019. 27. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407, 2024. 28. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. 29. Qwen Team.Qwen2.5: A party of foundation models, September 2024. 30. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019. 31. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the- art natural language processing. In Qun Liu and David 11 Schlangen, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October 2020. Association for Computational Linguistics.

Instant research discovery