Paper deep dive
Let LLMs Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering
Zaifu Zhan, Shuang Zhou, Rui Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 6/20/2026, 5:51:34 AM
Summary
The paper proposes a 'Multi-Agent Peer-Reviewed Reasoning' method to improve the accuracy, interpretability, and robustness of Large Language Models (LLMs) in Medical Question Answering (MedQA). Unlike standard majority voting which relies on answer agreement, this method involves multiple LLM agents generating Chain-of-Thought (CoT) reasoning and then acting as peer reviewers to score the logical soundness and factual correctness of each other's reasoning. Experiments using five models (Llama-3.1-8B, Qwen2.5-7B, Phi-4, DeepSeek-LLM-7B, and GPT-oss-20B) across three datasets (HeadQA, MedQA-USMLE, and PubMedQA) demonstrated that this approach consistently outperforms single-model CoT and CoT-based majority voting, achieving an average accuracy of 0.820.
Entities (10)
Relation Signals (3)
Llama-3.1-8b â evaluatedon â HeadQA
confidence 100% · Experiments were conducted with five state-of-the-art LLMs... on three benchmark datasets: HeadQA, MedQA-USMLE, and PubMedQA.
Multi-Agent Peer-Reviewed Reasoning â improves â MedQA
confidence 100% · The proposed multi-agent peer-reviewed reasoning method enables LLMs to act as both solvers and evaluators, yielding superior performance in MedQA.
GPT-OSS-20B â isstrongestsinglemodelin â MedQA
confidence 90% · The best single model (GPT-oss-20B) achieves an average accuracy of 0.777
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Objective: To enhance the accuracy, interpretability, and robustness of large language models (LLMs) in medical question answering (MedQA). Method: We designed a multi-agent peer-reviewed reasoning method in which multiple LLM agents independently generate chain-of-thought reasoning with candidate answers, then act as peer reviewers to evaluate each other's reasoning for factual correctness and logical soundness. The highest-rated reasoning chain is selected to produce the final answer. Experiments were conducted with five state-of-the-art LLMs (Llama-3.1-8B, Qwen2.5-7B, Phi-4, DeepSeek-LLM-7B, GPT-oss-20B) on three benchmark datasets: HeadQA, MedQA-USMLE, and PubMedQA. Performance was compared against single-model chain-of-thought reasoning and chain-of-thought-based majority voting. Results: Peer-reviewed reasoning consistently outperformed both baselines. The best model combination achieved an average accuracy of 0.820 across datasets, exceeding the strongest single model (0.777) and majority voting ensembles (up to 0.789). The method also scaled effectively with more participating models, while peer assessments reliably distinguished high- from low-quality reasoning chains. Conclusion: The proposed multi-agent peer-reviewed reasoning method enables LLMs to act as both solvers and evaluators, yielding superior performance in MedQA. By emphasizing reasoning quality rather than answer agreement alone, this approach improves accuracy, interpretability, and robustness, offering a promising direction for trustworthy biomedical AI systems.
Tags
Links
- Source: https://arxiv.org/abs/2606.15419v1
- Canonical: https://arxiv.org/abs/2606.15419v1
Trouble viewing inline? Open PDF directly â
Full Text
56,475 characters extracted from source content.
Expand or collapse full text
Journal of the American Medical Informatics Association, 2022, p. 1â16 doi: DOI HERE Advance Access Publication Date: Day Month Year Paper PAPER Let LLMs Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering Zaifu Zhan, MEng, 1 Shuang Zhou, PhD 2 and Rui Zhang, PhD 2,* 1 Department of Electrical and Computer Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States and 2 Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 420 Delaware St SE, 55455, Minneapolis, MN, United States â Corresponding author: Dr. Rui Zhang, PhD, Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Office: D528 Mayo building, 420 Delaware St SE, Minneapolis, MN 55455, zhan1386@umn.edu, Office Phone: 612-626-4209 Full paper word count: 3306 Abstract word count: 193 Abstract Objective: To enhance the accuracy, interpretability, and robustness of large language models (LLMs) in medical question answering (MedQA). Method: We designed a multi-agent peer-reviewed reasoning method in which multiple LLM agents independently generate chain- of-thought reasoning with candidate answers, then act as peer reviewers to evaluate each otherâs reasoning for factual correctness and logical soundness. The highest-rated reasoning chain is selected to produce the final answer. Experiments were conducted with five state-of-the-art LLMs (Llama-3.1-8B, Qwen2.5-7B, Phi-4, DeepSeek-LLM-7B, GPT-oss-20B) on three benchmark datasets: HeadQA, MedQA-USMLE, and PubMedQA. Performance was compared against single-model chain-of-thought reasoning and chain- of-thought-based majority voting. Results: Peer-reviewed reasoning consistently outperformed both baselines. The best model combination achieved an average accuracy of 0.820 across datasets, exceeding the strongest single model (0.777) and majority voting ensembles (up to 0.789). The method also scaled effectively with more participating models, while peer assessments reliably distinguished high- from low-quality reasoning chains. Conclusion: The proposed multi-agent peer-reviewed reasoning method enables LLMs to act as both solvers and evaluators, yielding superior performance in MedQA. By emphasizing reasoning quality rather than answer agreement alone, this approach improves accuracy, interpretability, and robustness, offering a promising direction for trustworthy biomedical AI systems. Key words: Question answering, Large language models, Chain-of-thought, Reasoning, Multi-agent Introduction Large language models (LLMs) have advanced biomedical natural language processing [1, 2, 3]. They have shown strong performance in tasks such as named entity recognition [4], relation extraction [5], summarization [6], disease classification [7], diagnosis [8], and question answering [9]. As model size has increased, LLMs have demonstrated emergent reasoning abilities [10, 11] that allow them to comprehend complex © The Author 2022. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com 1 arXiv:2606.15419v1 [cs.CL] 13 Jun 2026 2Zaifu Zhan et al. Fig. 1. Illustration of three reasoning paradigms. (a) Chain-of-thought (COT): each model independently generates a reasoning process and an answer. (b) COT-based group voting: multiple models produce reasoningâanswer pairs, and the final answer is decided by majority voting over the answers. (c) Multi-agent Peer-Reviewed Reasoning (ours): models not only generate reasoningâanswer pairs but also evaluate each otherâs reasoning; the final answer is selected based on peer review of reasoning quality. texts, understand human questions, and even address tasks that were not explicitly learned during training [12]. These advances have produced clear improvements in medical question answering (MedQA). The ability to answer questions accurately is particularly important in biomedicine because factual or logical errors may mislead clinical decision-making or research conclusions [13, 14]. Moreover, most biomedical information needs can be formulated as questionâanswering problems, which makes improvement in MedQA performance broadly valuable. Recent work has explored various approaches to improving LLMs on MedQA, with growing emphasis on leveraging their reasoning capabilities [15, 16, 10]. Researchers have become increasingly interested in whether models can mimic human- like reasoning [17]. One landmark development is chain-of- thought (CoT) prompting [18], which encourages the model to generate answers step by step and often leads to better performance [17, 19, 20]. Many studies have demonstrated that CoT improves LLM accuracy on QA tasks [21, 13]. Further progress has been achieved by combining CoT with techniques such as instruction fine-tuning [22], few-shot prompting [23], and retrieval-augmented generation [24]. These findings suggest that reasoning-based paradigms, especially CoT, are becoming a foundation for improving LLM performance on complex QA problems. For the application of CoT in MedQA, the quality of reasoning is crucial. Correct reasoning usually produces correct answers, while flawed reasoning often leads to wrong ones [24, 25, 26]. Most existing works [27, 28, 9] evaluate reasoning after the answer has been produced, either through human inspection or by comparing to human rationales, rather than incorporating reasoning evaluation into the decision process itself. Two research directions suggest possible solutions to this limitation.The first is LLM-as-a-judge[29],which has demonstrated that LLMs can evaluate output quality with a high level of agreement with human raters. This approach has been successfully applied to biomedical relation extraction[30], clinical trial recommendation [31], and patient record summarization [32]. For example, Zhou et al [9] adopted LLM as a judge and proposed LLM-w-Ref, a novel evaluation framework that leverages fine-grained rationales and LLM- as-a-Judge mechanisms to assess intermediate reasoning with expert-level fidelity while maintaining scalability. However, using LLMs as automatic judges has important limitations: their preferences do not always align with human experts, [33, 29] they can overvalue fluent but incorrect answers [34], and their scores are sensitive to prompt design and evaluation setup [35] 3 The second direction is multi-agent collaboration[36], which shows that groups of LLMs can achieve stronger reasoning abilities through interaction. For example, MDAgents[37] enables multiple LLMs to collaborate, achieving improvements on real-world medical knowledge and clinical diagnosis benchmarks. Similarly, Chen et al. [38] designed a multi- agent multi-disciplinary team framework to enhance medical consultation, thereby improving both diagnostic rationality and accuracy. Inspired by the advantages of LLM-as-a-judge and multi- agent collaboration, we propose a multi-agent peer-review reasoning method to improve MedQA, as illustrated in Figure 1. In this approach, multiple independent LLM agents generate chain-of-thought solutions to the same medical question. Each agent then reviews the reasoning generated by its peers and rates the logical soundness. The system aggregates these peer reviews to identify the most credible and coherent reasoning chain, which is subsequently used to generate the final answer. Unlike approaches that rely on majority voting over final answers, our method evaluates and selects the reasoning process itself. This enables the production of answers that are more accurate, interpretable, and robust in biomedical contexts. Our main contributions can be summarized as follows: 1. We propose a novel multi-agent peer-review reasoning method that leverages the evaluative and collaborative capacities of LLMs to enhance MedQA. 2. We conducted extensive experiments, demonstrating that our method is effective across different datasets, models, and agent configurations. Methods Overview of methods We propose a multi-agent peer-reviewed reasoning method to improve medical question answering. The method consists of two phases: (1) multiple LLM agents independently generate chain-of-thought reasoning with candidate answers, and (2) agents act as peer reviewers, scoring each otherâs reasoning for factual correctness and logical soundness. The highest-rated reasoning chain is selected, and its answer is adopted as the final output. We evaluated this method on five state-of-the-art LLMs (Llama-3.1-8B, Qwen2.5-7B, Phi- 4, DeepSeek-LLM-7B, and GPT-oss-20B) across three benchmark datasets(HeadQA, MedQA-USMLE, and PubMedQA), and compared it against two baselines: single-model chain-of-thought reasoning and CoT-based majority voting. This design allows us to assess both the effectiveness and robustness of our approach. MedQA task and datasets To evaluate our proposed multi-agent peer-review reasoning method, we adopt the MedQA task, which serves as a widely recognized benchmark for assessing the reasoning ability of large language models in the biomedical domain. In this task, models are required to answer domain-specific questions, often in multiple-choice format, while demonstrating coherent reasoning chains that reflect both factual knowledge and logical consistency. We consider three representative datasets: HeadQA, MedQA(USMLE), and PubMedQA. HeadQA [39] is constructed from official Spanish healthcare professional exams, covering diverse domains such as medicine, nursing, psychology, and pharmacy. The questions are concise but knowledge-intensive, demanding specialized expertise for accurate answers. MedQA(USMLE) [40], on the other hand, is derived from the United States Medical Licensing Examination and is widely regarded as one of the most challenging benchmarks in biomedical NLP. It consists of high-quality, expert-level multiple-choice questions designed to evaluate not only factual recall but also complex clinical reasoning and diagnostic decision-making. Complementary to these exam-style datasets, PubMedQA [41] is built from biomedical research articles in PubMed, where the task is to answer yes/no/maybe research questions based on scientific abstracts. Unlike HeadQA and MedQA, which test professional exam knowledge, PubMedQA emphasizes 4Zaifu Zhan et al. DatasetTestsetSource MedQA (USMLE)1,273Medical exam (USMLE) HeadQA244Spanish medical exams PubMedQA500Biomedical literature Table 1. Statistics and characteristics of medical QA datasets. evidence-based reasoning and the ability to interpret biomedical literature. Together, these datasets provide a comprehensive and complementary evaluation setting: HeadQA highlights the breadth of domain knowledge, MedQA-USMLE represents the depth of expert-level reasoning, and PubMedQA emphasizes literature-driven inference. This diversity enables us to rigorously assess both the robustness and generalization capacity of our method across different forms of biomedical reasoning. All the dataset statistics are summarized in Table 1. Multi-Agent Peer-Review Reasoning Method Inspired by the peer-review process of academic papersâwhere peer-reviewed work is deemed worthy of publicationâwe adopt a similar idea in our method. Reasoning and answers that undergo peer review are regarded as high-quality responses. Thus, we proposed the multi-agent peer-review reasoning method. By comparison, figure 1 presents three paradigms: (a) Chain-of-Thought, where a single model reasons step- by-step before answering; (b) Majority Vote, where multiple models output answers and the most frequent answer is selected; and (c) Multi-agent peer-review method (this work). While majority voting may improve robustness, agreement does not guarantee correctnessâmodels can share the same biases or training artifactsâand it discards valuable reasoning traces. Our method preserves and evaluates reasoning: multiple models first generate CoT traces, then act as judges to score one anotherâs reasoning. The highest-rated (peer-reviewed) reasoning and its associated answer are selected as the final output. By combining explicit reasoning generation with cross- model peer review, our method evaluates both performance and the reliability of the underlying reasoning, providing a more informative and trustworthy approach to medical question answering than CoT or majority vote alone. Phase 1: Chain-of-Thought Generation We elicit explicit, structured reasoning from multiple LLMs before revealing final answers. For multiple-choice medical QA (e.g., MedQA, HeadQA), prompts require option-by-option analysis with stepwise justification [? 42]. For literature- grounded QA (e.g., PubMedQA), prompts instruct models to examine the abstract, extract supporting evidence, and form a logically consistent conclusion [24]. This phase (i) makes decision paths transparent, (i) typically improves answer quality via systematic analysis, and (i) produces rich intermediate traces that are amenable to evaluation. The CoT prompt could be found in supplementary material 1. Phase 2: Peer-Review of Reasoning LLMs then evaluate the responses produced by other models using a 6-point rubric (0â5) that jointly assesses answer correctness and reasoning quality. Scores of 4â5 denote correct answers with sound medical reasoning; 3 denotes correct answers with minor flaws; 1â2 capture substantial reasoning issues or incorrect answers; 0 reflects fundamental misunderstanding. Importantly, judges do not access ground- truth labels; they rely on domain knowledge and internal reasoning, yielding an assessment of understanding rather than pattern matching. For each candidate response, the judge receives the original question, the full reasoning trace, and the final answer, and returns a numeric score plus a concise justification. Cross-model evaluations form a score matrix; we aggregate scores (e.g., by mean or median) to select the top peer-reviewed reasoning and adopt its answer as final. The judgement prompt could be found in supplementary material 1. Scoring rubric: The 0â5 scoring rubric is not intended to replicate formal peer-review scoring standards, but rather serves as an interpretable and lightweight proxy for reasoning quality assessment. Alternative prompting inspirationsâsuch as checklist-based evaluation, pairwise preference judgments, or 5 binary accept/reject schemesâcould plausibly influence both model judgments and downstream performance. We leave a systematic comparison of such evaluation strategies to future work. Tie handling: During score aggregation, we observed that ties in the peer-reviewed scores occurred in approximately 15% of the cases. In such situations, we adopted a simple and deterministic tie-handling strategy: when multiple reasoning chains received identical aggregated scores, the system selected the response that appeared first in the initial generation order. This behavior follows the stability property of Pythonâs sorting function, which preserves the original relative order of elements with equal keys. We applied the same tie-handling strategy consistently to the CoT-based majority voting baseline, where ties in vote counts were resolved by selecting the first encountered answer. Experiments We designed two baselines to evaluate the effectiveness of our method: (1) Chain-of-Thought (CoT) reasoning, where individual models generate reasoning and answers independently, and (2) Majority Voting, where the final answer is determined by aggregating the outputs of multiple models. These baselines provide a comparison for understanding the contributions of reasoning generation and peer review evaluation. We conducted experiments using five state-of-the-art language models: Llama-3.1-8B [43], Qwen2.5-7B [44, 45], Phi-4 [46], DeepSeek-LLM-7B [47], and GPT-oss-20B [48]. These models were selected to introduce diversity along several concrete axes. First, they span a wide range of parameter scales (7Bâ20B), covering both smaller and larger LLMs in our evaluation pool. Second, they come from distinct model families (Llama, Qwen, Phi, DeepSeek, and GPT-oss), which reflect different development lineages and training paradigms. Third, this diversity is empirically reflected in heterogeneous baseline performance: the best single model (GPT-oss-20B) achieves an average accuracy of 0.777, whereas DeepSeek- LLM-7B yields 0.437, indicating substantial variation in their reasoning and answering behaviors even before peer review. Together, these differences provide a more challenging and informative setting to test whether cross-model peer evaluation can leverage complementary strengths rather than relying on redundant agents. To evaluate model performance, we used standard accuracy metrics for final answer correctness, following prior works [39, 41, 40]. Additionally, we introduced a peer review scoring system based on a 6-point rubric (0-5) to assess the quality of reasoning. This dual-metric approach captures both the correctness of answers and the logical consistency of reasoning processes. We report 95% confidence intervals for accuracy using bootstrap resampling (1000 iterations) over the test set. For each iteration, accuracy was recomputed on the resampled data, and the 2.5th and 97.5th percentiles were used to define the confidence interval. All experiments were conducted on NVIDIA A100 GPUs with 40 GB of memory. We used a temperature of 0.1 for less randomness, a maximum generation length of 1,024 tokens, and fixed random seeds for reproducibility. The experiments were implemented using Python, with key libraries including PyTorch [49] and Hugging Face Transformers [50]. In experiments, we decoupled the experiments in three stages: first generating all LLM responses for every question in every dataset, then generating all LLM peer reviews in a separate batch, and finally aggregating the outputs to compute the final performance metrics. Results As shown in Table 2, we evaluated the performance of the proposed peer-reviewed reasoning method across three representative biomedical QA datasets: HeadQA, MedQA- USMLE, and PubMedQA. Results were compared against two baselines: chain-of-thought reasoning and CoT-based majority voting. In each method of experiments, we swept through all possible model combinations, ranging from pairwise 6Zaifu Zhan et al. Table 2. Performance comparison of individual LLMs, COT-based group voting, and peer-reviewed reasoning on three medical QA datasets (HeadQA, MedQA, PubMedQA). 95% confidence intervals and the average accuracy are reported. Throughout the CoT-based majority voting and peer-review reasoning settings, we anonymize model identities by replacing names with numeric indices. The correspondence between indices and models is detailed in the Chain-of-Thought part. MethodModelHeadQA(95%CI)MedQA(95%CI)PubMedQA(95%CI) Avg. Chain-of-thought (1) Deepseek-llm-7b0.4303 (0.3689, 0.4918) 0.3103 (0.2852, 0.3362)0.57 (0.526, 0.614)0.437 (2) Qwen2.5-7B0.709 (0.6516, 0.7664)0.5695 (0.542, 0.597)0.67 (0.628, 0.71)0.650 (3) Llama-3.1-8B0.7213 (0.6639, 0.7746) 0.6575 (0.6316, 0.6834)0.754 (0.716, 0.792)0.711 (4) Phi-40.8197 (0.7705, 0.8648)0.7313 (0.707, 0.7557)0.736 (0.696, 0.774)0.762 (5) GPT-oss-20B0.8361 (0.7869, 0.8811) 0.7903 (0.7675, 0.8123)0.706 (0.666, 0.746)0.777 COT-based group voting (1) + (2) + (3) + (5)0.7787 (0.7254, 0.8279) 0.6897 (0.6638, 0.7148)0.698 (0.658, 0.738)0.722 (1) + (3) + (4)0.7787 (0.7254, 0.8279) 0.6874 (0.6614, 0.7125)0.718 (0.678, 0.758)0.728 (3) + (4)0.7951 (0.7418, 0.8443) 0.6999 (0.6748, 0.7251)0.702 (0.662, 0.742)0.732 (1) + (2) + (4) + (5)0.7951 (0.7418, 0.8443)0.7266 (0.7023, 0.751)0.69 (0.65, 0.73)0.737 (2) + (3) + (4)0.7992 (0.7459, 0.8484) 0.7015 (0.6764, 0.7266)0.728 (0.688, 0.766)0.743 (1) + (3) + (5)0.7869 (0.7336, 0.8361) 0.7258 (0.7015, 0.7502)0.72 (0.68, 0.758)0.744 (3) + (5)0.8033 (0.7541, 0.8525) 0.7628 (0.7392, 0.7855)0.672 (0.63, 0.712)0.746 (1) + (2) + (3) + (4) + (5) 0.8115 (0.7623, 0.8607) 0.7188 (0.6936, 0.7431)0.732 (0.692, 0.77)0.754 (2) + (3) + (5)0.8115 (0.7623, 0.8607) 0.7353 (0.7109, 0.7596)0.73 (0.69, 0.768)0.759 (1) + (3) + (4) + (5)0.8115 (0.7623, 0.8607)0.7526 (0.729, 0.7761)0.72 (0.68, 0.758)0.761 (1) + (4) + (5)0.8156 (0.7664, 0.8648) 0.7612 (0.7376, 0.7848)0.714 (0.674, 0.754)0.764 (2) + (3) + (4) + (5)0.8197 (0.7705, 0.8648) 0.7502 (0.7258, 0.7738)0.726 (0.686, 0.764)0.765 (4) + (5)0.8402 (0.791, 0.8852)0.7777 (0.7549, 0.8005)0.682 (0.64, 0.722)0.767 (2) + (4) + (5)0.8279 (0.7787, 0.873)0.7777 (0.7549, 0.8005)0.726 (0.686, 0.764)0.777 (3) + (4) + (5)0.8443 (0.7992, 0.8893)0.7832 (0.7604, 0.806)0.736 (0.696, 0.774)0.788 Peer-reviewd reasoning (2) + (5)0.832 (0.7828, 0.877)0.7997 (0.7777, 0.8217)0.716 (0.676, 0.756)0.783 (1) + (5)0.8361 (0.7869, 0.8811) 0.8044 (0.7824, 0.8256)0.73 (0.69, 0.768)0.790 (1) + (2) + (5)0.8361 (0.7869, 0.8811) 0.8044 (0.7824, 0.8256)0.734 (0.694, 0.772)0.791 (2) + (3) + (5)0.8361 (0.7869, 0.8811) 0.8052 (0.7832, 0.8264)0.744 (0.706, 0.782)0.795 (1) + (2) + (3) + (5)0.8361 (0.7869, 0.8811) 0.8099 (0.7879, 0.8311)0.752 (0.714, 0.79)0.799 (3) + (5)0.8566 (0.8115, 0.8975)0.8162 (0.795, 0.8374)0.744 (0.706, 0.782)0.806 (2) + (4) + (5)0.8566 (0.8115, 0.8975)0.8068 (0.7848, 0.828)0.754 (0.716, 0.792)0.806 (1) + (3) + (5)0.8484 (0.8033, 0.8934) 0.8201 (0.7989, 0.8413)0.754 (0.716, 0.792)0.807 (1) + (2) + (4) + (5)0.8566 (0.8115, 0.8975)0.8138 (0.7918, 0.835)0.756 (0.718, 0.794)0.809 (4) + (5)0.8566 (0.8115, 0.8975) 0.8225 (0.8013, 0.8429)0.752 (0.714, 0.79)0.810 (2) + (3) + (4) + (5)0.8525 (0.8074, 0.8934)0.813 (0.791, 0.8342)0.772 (0.734, 0.808)0.812 (1) + (4) + (5)0.8607 (0.8156, 0.9016)0.8233 (0.802, 0.8437)0.754 (0.716, 0.792)0.813 (1) + (2) + (3) + (4) + (5) 0.8525 (0.8074, 0.8934)0.817 (0.7958, 0.8382)0.774 (0.736, 0.81)0.814 (3) + (4) + (5)0.8484 (0.8033, 0.8934) 0.8264 (0.8052, 0.8468)0.774 (0.736, 0.81)0.816 (1) + (3) + (4) + (5)0.8566 (0.8115, 0.8975)0.8272 (0.806, 0.8476)0.776 (0.738, 0.812)0.820 combinations (10 groups) to triplets (10 groups), quadruplets (5 groups), and quintuplets (1 group). We then selected and sorted the combination with the highest average accuracy for presentation. Performance of Individual Models. Individual LLMs displayed heterogeneous performance across datasets, as shown in Table 2. The best single model, GPT- oss-20B, reached an average accuracy of 0.777, followed by Phi-4 (0.762) and Llama-3.1-8B (0.710). In contrast, DeepSeek- LLM-7B yielded a substantially lower average accuracy of 0.437. These results indicate the variability in baseline performance among different model architectures/sizes and training paradigms. Effect of Majority Voting. CoT-based majority voting led to consistent improvements over single-model reasoning. For instance, combining models (3 + 4 + 5) achieved an average accuracy of 0.789, outperforming all individual models. When more models were aggregated, performance generally improved, with accuracies ranging from 0.722 to 0.789 depending on the combination. In contrast to the performance of an individual model, the aggregated results from multiple models exhibit reduced variance. These results demonstrate that simple ensemble strategies enhance 7 Fig. 2. Comparison of COT-based majority voting (blue) and Peer- reviewed reasoning (pink) across different numbers of models (2â5). Panels (aâc) present results on three different 3 datasets. The y-axis shows accuracy with error bars indicating standard deviations. Overall, peer- reviewed reasoning consistently outperforms COT-based majority voting, and performance generally improves as the number of models increases. When the number of models is 5, only one set of experimental data is available, so there is no standard deviation. robustness, though gains remained moderate relative to the best single models. Peer-Reviewed Reasoning results Peer-reviewed reasoning consistently outperformed both individual models and majority voting. Across all datasets, accuracy improvements were observed regardless of the specific model combinations. For example, using models (Llama, Phi, GPT-oss) achieved an average accuracy of 0.816, while the combination (Deepseek, Llama, Phi, GPT-oss) yielded the strongest results with an accuracy of 0.820. Although DeepSeek exhibits relatively weaker performance as a standalone solver, its inclusion contributes additional diversity in the peer-review stage, which improves the selection of high-quality reasoning chains. On individual datasets, peer-reviewed reasoning reached up to 0.861 on HeadQA, 0.827 on MedQA-USMLE, and 0.776 on PubMedQA, surpassing the corresponding results from CoT-based majority voting or individual LLM. Scaling with Model Numbers. Figure 2 illustrates the effect of increasing the number of models on performance. Peer-reviewed reasoning consistently exceeded majority voting across two-, three-, four-, and five-model settings. Accuracy gains were most pronounced in settings with larger model pools, highlighting the scalability of the proposed approach. Reasoning Quality Assessment. Beyond accuracy,the peer-review process successfully distinguished reasoning quality across models. As shown in Figure 3, pairwise rating distributions reveal that LLM judges provided differentiated assessments of their peersâ reasoning. High-performing models such as GPT-oss-20B (mean:4.1, std:1.2) and Phi-4 (mean:4.4, std:0.9) generally received higher ratings, while weaker models like DeepSeek- LLM-7B (mean:3.2, std:1.6) were consistently rated lower. These patterns confirm that the peer-review mechanism is effective in identifying logically sound reasoning chains, which in turn contribute to improved final answers. Additionally, to make the relationship between reasoning quality assessment and performance explicit, we computed correlations between the mean peer-review score of each model and its single-model accuracy. We observed a strong positive Pearson correlation (r = 0.91, p = 0.034), indicating that models receiving higher peer-evaluated reasoning scores tend to achieve higher answer accuracy. 8Zaifu Zhan et al. Fig. 3. Pairwise rating score distributions across models and datasets. Each subplot corresponds to one judge model (column) evaluating another model (row). Model1â model2 means model1 judges model2âs reasoning. Within each subplot, violin plots illustrate the distribution of rating scores on three datasets (HeadQA, MedQA, and PubmedQA), distinguished by different fill colors. The black horizontal line inside each violin indicates the median score. Taken together, these findings demonstrate that peer- reviewed reasoning achieves superior performance compared to both chain-of-thought and majority voting baselines. By incorporating reasoning evaluation into the answer selection process, our method yields more accurate, robust, and high- quality outputs across diverse biomedical QA benchmarks. Ablation study: Judge Prompt Compared with majority voting, our method introduces an additional mutual judging stage. A key component of this stage is the design of the judge prompt. As shown in Table 3, we compared three different judge prompts on the MedQA-USMLE dataset across 15 model combinations. For each combination, we evaluate three judge prompts (Prompt1â3) and report the mean accuracy as well as the standard deviation across the three prompts. It is important to note that these results are obtained by evaluating each configuration on the entire test set, rather than through repeated sampling. Therefore, the reported standard deviation reflects the performance variation 9 ModelPrompt1 Prompt2 Prompt3AvgStd (3)+(4)+(5)0.8260.8350.8260.829 0.005 (1)+(3)+(4)+(5)0.8270.8340.8200.827 0.007 (3)+(5)0.8160.8280.8210.822 0.006 (1)+(4)+(5)0.8230.8220.8180.821 0.003 (1)+(3)+(5)0.8200.8280.8150.821 0.006 (4)+(5)0.8220.8190.8170.820 0.003 (1)+(2)+(3)+(4)+(5)0.8170.8140.8130.815 0.002 (2)+(3)+(4)+(5)0.8130.8130.8120.813 0.000 (1)+(2)+(4)+(5)0.8140.8100.8080.810 0.003 (1)+(2)+(3)+(5)0.8100.8110.8040.809 0.004 (2)+(3)+(5)0.8050.8090.8080.807 0.002 (2)+(4)+(5)0.8070.8070.8080.807 0.000 (1)+(2)+(5)0.8040.8080.8040.806 0.002 (1)+(5)0.8040.8040.8030.804 0.001 (2)+(5)0.8000.7990.7990.799 0.000 Table 3. Accuracy of peer-reviewed reasoning under different judge prompt designs on the MedQA-USMLE dataset. We report results for 15 model combinations over whole testset. For each combination, Prompt 1â3 are evaluated, and Avg/Std are computed across the three prompts. The relatively small std indicates that, although different prompts are used, the final performance varies only slightly. This suggests that our framework is robust to prompt variations. across different prompt formulations, rather than uncertainty from sampling. We observe that the variation across prompts is small, with the largest standard deviation being 0.007. This suggests that the performance of our framework is relatively consistent under different judge prompt designs, indicating a degree of robustness to prompt variations. The detailed designs of the different judge prompts can be found in Supplementary Material 1. Ablation study: biased peer-review Our method also includes cases where a model evaluates its own outputs. Although the model is not explicitly informed that the evaluated response is generated by itself, there remains a possibility that it may recognize its own answer and assign a higher score. To address this concern, we conduct additional experiments in which the judge models are different from the answer-generating model. For example, responses generated by the DeepSeek model are evaluated by the other four models as judges. As shown in Table 4, the biased peer-review is competitive to our method with slightly improvement. Run time analysis Finally, we report the average runtime of our experiments across the three datasets, which consists of two phases: Phase Fig. 4. Runtime analysis of the proposed method across three datasets (MedQA-USMLE, HeadQA, and PubMedQA). For each model, average inference time is reported separately for Phase 1 (chain-of-thought generation) and Phase 2 (peer-review judgment). Overall, peer-review judgment is slightly faster than CoT generation, while the two-stage pipeline incurs higher total runtime than single-pass CoT prompting. 1, CoT generation, and Phase 2, peer-review judgment (as shown in the Fig. 4). Overall, Phase 2 is slightly faster 10Zaifu Zhan et al. ModelHeadQA(95%CI)MedQA(95%CI)PubMedQA(95%CI) Avg.Unbiased (1)+(2)+(4)0.7582 (0.7049, 0.8115) 0.6174 (0.5907, 0.6441)0.712 (0.672, 0.752)0.6960.728 (2)+(3)+(4)0.7418 (0.6844, 0.7951) 0.6112 (0.5844, 0.6379)0.764 (0.726, 0.8)0.7060.737 (1)+(2)+(3)+(4)0.7418 (0.6844, 0.7951) 0.6284 (0.6017, 0.6551)0.774 (0.736, 0.81)0.7150.742 (1)+(3)+(4)0.7582 (0.7049, 0.8115) 0.6952 (0.6701, 0.7203)0.802 (0.766, 0.836)0.7520.761 (1)+(2)+(5)0.8443 (0.7992, 0.8893) 0.7989 (0.7769, 0.8209)0.728 (0.688, 0.766)0.790 0.791 (2)+(3)+(5)0.832 (0.7828, 0.877)0.802 (0.78, 0.824)0.746 (0.708, 0.784)0.7930.795 (1)+(2)+(3)+(5)0.8361 (0.7869, 0.8811)0.806 (0.784, 0.8272)0.75 (0.712, 0.788)0.7970.799 (2)+(4)+(5)0.8484 (0.8033, 0.8934)0.8068 (0.7848, 0.828)0.75 (0.712, 0.788)0.8020.806 (1)+(2)+(4)+(5)0.8525 (0.8074, 0.8934) 0.8091 (0.7871, 0.8303)0.75 (0.712, 0.788)0.8040.809 (2)+(3)+(4)+(5)0.8443 (0.7992, 0.8893) 0.8091 (0.7871, 0.8303)0.77 (0.732, 0.806)0.8080.812 (1)+(4)+(5)0.8484 (0.8033, 0.8934) 0.8225 (0.8013, 0.8429)0.756 (0.718, 0.794)0.809 0.813 (1)+(3)+(5)0.8484 (0.8033, 0.8934) 0.8209 (0.7997, 0.8413)0.762 (0.724, 0.798)0.8100.807 (1)+(2)+(3)+(4)+(5) 0.8484 (0.8033, 0.8934)0.813 (0.791, 0.8342)0.77 (0.732, 0.806)0.8100.814 (3)+(4)+(5)0.8525 (0.8074, 0.8934) 0.8264 (0.8052, 0.8468)0.766 (0.728, 0.802)0.8150.816 (1)+(3)+(4)+(5)0.8566 (0.8115, 0.8975)0.828 (0.8068, 0.8484)0.778 (0.742, 0.814)0.8210.820 Table 4. Performance of biased peer-review settings across three medical QA datasets. In this setting, the answer-generating model is excluded from the pool of judge models. Results are shown for the top 15 model combinations ranked by average accuracy under the biased peer-review setting; combinations may differ from those in Table 2. Accuracy with 95% confidence intervals is reported along with the average accuracy. The last column âUnbiasedâ results are from the main experiments (Table 2) for comparison than Phase 1, which is expected: the first phase requires generating a longer reasoning trajectory, whereas the second phase mainly focuses on scoring the peer-review outputs. Since our method follows a two-stage pipeline, it inevitably incurs higher computational cost and longer runtime than standard one-shot CoT prompting. Discussion MedQA is the fundamental task in biomedical natural language processing, and our proposed multi-agent method is designed for better question-answering ability from LLMs. Our experiments provide strong evidence that the proposed peer- reviewed reasoning method offers substantial improvements for MedQA. By systematically comparing against chain- of-thought reasoning and majority voting, we observed consistent gains in accuracy across HeadQA, MedQA-USMLE, and PubMedQA. Beyond accuracy, the peer review process successfully distinguished high- and low-quality reasoning chains, providing a novel way to enhance both robustness and interpretability in biomedical contexts. The observed improvements can be attributed to the methodâs focus on reasoning quality rather than final answers alone. Majority voting aggregates outputs from multiple models, but agreement does not guarantee correctness, as models trained on similar data often converge on the same flawed reasoning or biased predictions. Peer-reviewed reasoning addresses this limitation by incorporating cross- model evaluation of the reasoning process itself. Through structured peer assessments, logically sound reasoning chains are preferentially selected, even when they may not initially be in the majority. For example, while DeepSeek achieves lower accuracy as an individual solver, it introduces heterogeneity in the evaluation stage. As a judge, it penalizes certain flawed or shallow reasoning patterns that stronger models may occasionally overlook due to shared inductive biases. This diversity in judgment helps filter out low-quality reasoning chains, leading to improved overall performance despite DeepSeekâs weaker individual accuracy. This mechanism mirrors the human peer-review process, in which arguments are evaluated not only by outcomes but also by their underlying validity. Furthermore, our results highlight that peer-reviewed reasoning scales effectively with the number of participating models. As shown in Figure 2, performance improvements became more pronounced as additional agents were introduced. This trend suggests that differences in model outputs and reasoning tendencies across agents provide complementary strengths, which can be systematically leveraged through peer evaluation. In practice, such scalability offers a pathway to 11 stronger ensemble-style methods that avoid simple majority bias. In biomedical applications, interpretability is not merely desirable but essential. Models deployed in clinical decision support or evidence synthesis must provide reasoning that clinicians and researchers can inspect. Traditional CoT reasoning improves transparency by revealing intermediate steps, but it does not inherently filter flawed reasoning. Peer-reviewed reasoning directly addresses this gap: the evaluative stage explicitly rewards coherent, medically sound reasoning chains. As demonstrated in Figure 3, LLM judges reliably differentiated strong from weak rationales, with high- performing models such as GPT-oss-20B and Phi-4 consistently receiving higher ratings. This evaluative dimension produces outputs that are both more accurate and more trustworthy for downstream use. In addition, robustness is a critical requirement for biomedical AI. Peer review strengthens robustness by ensuring that final answers are supported by the most credible reasoning available in the agent pool. This mitigates risks of overfitting to spurious patterns or propagating systematic errors shared across models. By integrating both diversity (multiple agents) and selectivity (peer review), our method reduces variance in outcomes and enhances confidence in results. Our method draws inspiration from two recent research directions: LLM-as-a-judge and multi-agent collaboration. Prior studies have shown that LLMs can achieve high agreement with human raters when evaluating biomedical relation extraction, summarization, and trial design outputs. Similarly, multi-agent approaches have demonstrated gains in medical consultation and diagnostic reasoning by encouraging collaboration among models. However, these directions have largely evolved separately. To our knowledge, this is the first work that explicitly integrates LLM-as-a-judge into a multi- agent reasoning paradigm for MedQA. By uniting evaluative and collaborative capacities, we show that LLMs can act as both solvers and reviewers, producing synergistic improvements that exceed either paradigm alone. While our Peer Review method demonstrates significant performance gains over standard baselines,a critical consideration for practical deployment is the computational overhead. We acknowledge that the improved accuracy comes with increased resource consumption and wall-clock time. Our method involves three main stages: initial generation, peer review, and aggregation. The bottleneck lies in the peer review stage, which requires each of theN agents to evaluate the N reasonings produced, resulting inN 2 additional generation calls. Quantitatively, compared to the cost of a CoT generation (1 computational unit), the Majority Vote baseline withN = 5 agents requiresN = 5 units. Our approach, however, necessitatesN +N 2 = 5 + 25 = 30 computational units per question. This makes our method 5 times more resource- intensive than the majority vote baseline. Furthermore, due to the sequential dependencyâthe review stage must wait for all initial generationsâthe wall-clock time is also increased, approximately doubling that of the Majority Vote method, even when utilizing high parallelism for the review sub-tasks. Despite these promising findings, several limitations should be acknowledged. First, the number of models and agent configurations was limited. While we observed consistent improvements up to five agents, scaling could reveal new challenges in coordination, computational cost, and aggregation strategy. Second, due to the computation limitation, we didnât further explore the setting with more LLMs. With more models, the performance gain is not certain. Third, we didnât consider the data leakage issues. The models we used might have been trained on these datasets. Future research could focus on several extensions. One direction is to investigate integration with retrieval-augmented generation or external knowledge bases, which may further improve reasoning quality. In addition, expanding the method to multimodal inputsâsuch as combining text with imaging or laboratory dataâwould increase applicability to real- world clinical decision support. Third, we acknowledge that tie handling represents a design choice that may influence final answer selection. While the simple deterministic 12Zaifu Zhan et al. strategy adopted here ensures reproducibility and consistency across methods, alternative tie-breaking mechanismsâsuch as secondary reasoning quality metrics, confidence estimation, or additional judging roundsâmay further improve robustness. We leave systematic exploration of such strategies to future work. Overall, our findings underscore the potential of peer- reviewed reasoning as a new paradigm for MedQA. By combining collaborative reasoning with evaluative judgment, our method enhances accuracy, interpretability, and robustness in ways that conventional methods cannot. As biomedical AI systems continue to move toward clinical deployment, approaches that emphasize both performance and trustworthiness will be essential.Peer-reviewed reasoning represents a promising step in this direction. Conclusion In this work, we proposed a multi-agent peer-reviewed reasoning method for medical question answering. By enabling large language models to act as both solvers and reviewers, the method consistently improved accuracy over chain-of-thought reasoning and majority voting across three benchmark datasets. Beyond higher accuracy, peer-reviewed reasoning enhanced interpretability and robustness by explicitly selecting reasoning chains of higher logical quality. Data Availability 0 The three datasets we used are all public: 1. HeadQA: https://aghie.github.io/head-qa/ 2. MedQA(USMLE): https://github.com/jind11/MedQA 3. PubMedQA: https://pubmedqa.github.io/ Code Availability The code is available at https://github.com/Learner4everrr/Multi- agent-peer-reviewed-framework-for-MedQA Acknowledgements We would like to express our sincere gratitude to the reviewers for their valuable comments. Author Contributions Z.Z.: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Validation, Visualization, Writing â original draft, Writing â review & editing. S.Z.: Validation, Writing â review & editing. R.Z.: Conceptualization, Project administration, Supervision, Writing â review & editing. Competing Interests The authors declare no competing interests. Funding This work was supported by the National Institutes of Healthâs National Center for Complementary and Integrative Health (grant numbers R01AT009457 and U01AT012871), the National Institute on Aging (grant number R01AG078154), the National Cancer Institute (grant number R01CA287413), the National Institute of Diabetes and Digestive and Kidney Diseases (grant number R01DK115629), and the National Institute on Minority Health and Health Disparities (grant number 1R21MD019134-01). References 1. Qingyu Chen, Yan Hu, Xueqing Peng, Qianqian Xie, Qiao Jin, Aidan Gilson, Maxwell B Singer, Xuguang Ai, Po-Ting Lai, Zhizheng Wang, et al. Benchmarking large language models for biomedical natural language processing applications and recommendations.Nature communications, 16(1):3280, 2025. 2. Shuang Zhou, Zidu Xu, Mian Zhang, Chunpu Xu, Yawen Guo, Zaifu Zhan, Yi Fang, Sirui Ding, Jiashuo Wang, Kaishuai Xu, et al. Large language models for disease 13 diagnosis: A scoping review. npj Artificial Intelligence, 1(1):9, 2025. 3. Zaifu Zhan, Shuang Zhou, Huixue Zhou, Jiawen Deng, Yu Hou, Jeremy Yeung, and Rui Zhang. An evaluation of deepseek models in biomedical natural language processing. arXiv preprint arXiv:2503.00624, 2025. 4. Mingchen Li, Zaifu Zhan, Han Yang, Yongkang Xiao, Huixue Zhou, Jiatan Huang, and Rui Zhang. Benchmarking retrieval-augmented large language models in biomedical nlp: Application, robustness, and self-awareness. Science Advances, 11(47):eadr1443, 2025. 5. Zaifu Zhan, Shuang Zhou, Mingchen Li, and Rui Zhang. Ramie:retrieval-augmentedmulti-taskinformation extraction with large language models on dietary supplements.Journal of the American Medical Informatics Association, 32(3):545â554, 2025. 6. Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin F Rousseau, et al.Evaluating large language models on medical evidence summarization. NPJ digital medicine, 6(1):158, 2023. 7. Zaifu Zhan, Shuang Zhou, Xiaoshan Zhou, Yongkang Xiao, Jun Wang, Jiawen Deng, He Zhu, Yu Hou, Yiran Song, Mingquan Lin, and Rui Zhang. Retrieval-augmented in- context learning for multimodal large language models in disease classification. Journal of Biomedical Informatics, 178:105017, 2026. 8. Shuang Zhou, Jiashuo Wang, Zidu Xu, Song Wang, David Brauer, Lindsay Welton, Jacob Cogan, Yuen-Hei Chung, Lei Tian, Zaifu Zhan, et al.Uncertainty-aware large language models for explainable disease diagnosis.npj Digital Medicine, 8(1):690, 2025. 9. Shuang Zhou, Wenya Xie, Jiaxi Li, Zaifu Zhan, Meijia Song, Han Yang, Cheyenna Espinoza, Lindsay Welton, Xinnie Mai, Yanwei Jin, et al. Automating expert-level medical reasoning evaluation of large language models. npj Digital Medicine, 2025. 10. Mary M Lucas, Justin Yang, Jon K Pomeroy, and Christopher C Yang. Reasoning with large language models for medical question answering. Journal of the American Medical Informatics Association, 31(9):1964â1975, 2024. 11. Hang Yang, Hao Chen, Hui Guo, Yineng Chen, Ching- Sheng Lin, Shu Hu, Jinrong Hu, Xi Wu, and Xin Wang. Llm-medqa: Enhancing medical question answering through case studies in large language models.In 2025 International Joint Conference on Neural Networks (IJCNN), pages 1â8. IEEE, 2025. 12. Dylan Zhang, Justin Wang, and Francois Charton. Instruction diversity drives generalization to unseen tasks. arXiv preprint arXiv:2402.10891, 2024. 13. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al.Toward expert-level medical question answering with large language models. Nature Medicine, 31(3):943â950, 2025. 14. Qiao Jin, Zheng Yuan, Guangzhi Xiong, Qianlan Yu, Huaiyuan Ying, Chuanqi Tan, Mosha Chen, Songfang Huang, Xiaozhong Liu, and Sheng Yu. Biomedical question answering: a survey of approaches and challenges. ACM Computing Surveys (CSUR), 55(2):1â36, 2022. 15. Congyun Jin, Ming Zhang, Weixiao Ma, Yujiao Li, Yingbo Wang, Yabo Jia, Yuliang Du, Tao Sun, Haowen Wang, Cong Fan, et al. Rjua-meddqa: A multimodal benchmark for medical document question answering and clinical reasoning.In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5218â5229, 2024. 16. Yuxin Zuo, Shang Qu, Yifei Li, Zhang-Ren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou.MedxpertQA: Benchmarking expert-level medical reasoning and understanding.In Forty-second International Conference on Machine Learning, 2025. 17. Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, and Benyou Wang. Towards medical complex reasoning with LLMs through 14Zaifu Zhan et al. medical verifiable problems.In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar,editors,Findings of the Association for Computational Linguistics: ACL 2025, pages 14552â 14573, Vienna, Austria, July 2025. Association for Computational Linguistics. 18. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824â24837, 2022. 19. Yanming Liu, Xinyue Peng, Tianyu Du, Jianwei Yin, Weihao Liu, and Xuhong Zhang. ERA-CoT: Improving chain-of-thought through entity relationship analysis. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8780â8794, Bangkok, Thailand, August 2024. Association for Computational Linguistics. 20. Mingyu Jin, Haochen Xue, Zhenting Wang, Boming Kang, Ruosong Ye, Kaixiong Zhou, Mengnan Du, and Yongfeng Zhang.ProLLM: Protein chain-of-thoughts enhanced LLM for protein-protein interaction prediction. In First Conference on Language Modeling, 2024. 21. Sohyeon Jeon and Hong-Gee Kim.A comparative evaluation of chain-of-thought-based prompt engineering techniques for medical question answering. Computers in Biology and Medicine, 196:110614, 2025. 22. Chenqian Le, Ziheng Gong, Chihang Wang, Haowei Ni, Panfeng Li, and Xupeng Chen. Instruction tuning and cot prompting for contextual medical qa with llms. arXiv preprint arXiv:2506.12182, 2025. 23. Saeel Sandeep Nachane, Ojas Gramopadhye, Prateek Chanda, Ganesh Ramakrishnan, Kshitij Sharad Jadhav, Yatin Nandwani, Dinesh Raghu, and Sachindra Joshi. Few shot chain-of-thought driven reasoning to prompt LLMs for open-ended medical question answering.In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 542â573, Miami, Florida, USA, November 2024. Association for Computational Linguistics. 24. Ziyu Wang, Elahe Khatibi, and Amir M Rahmani. Medcot- rag: Causal chain-of-thought rag for medical question answering. In 2025 IEEE 21st International Conference on Body Sensor Networks (BSN), pages 1â4. IEEE, 2025. 25. Valentin Li Ìevin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther. Can large language models reason about medical questions? Patterns, 5(3), 2024. 26. Jiageng Wu,Kevin Xie,Bowen Gu,Nils Kr Ìuger, Kueiyu Joshua Lin, and Jie Yang. Why chain of thought fails in clinical text understanding.arXiv preprint arXiv:2509.21933, 2025. 27. Kevin Wu, Eric Wu, Rahul Thapa, Kevin Wei, Angela Zhang, Arvind Suresh, Jacqueline J Tao, Min Woo Sun, Alejandro Lozano, and James Zou.Medcasereasoning: Evaluating and learning diagnostic reasoning from clinical case reports. arXiv preprint arXiv:2505.11733, 2025. 28. Birger Mo Ìell, Fredrik Sand Aronsson, and Sanian Akbar. Medical reasoning in llms: an in-depth analysis of deepseek r1. Frontiers in Artificial Intelligence, 8:1616145, 2025. 29. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems, 36:46595â46623, 2023. 30. Md Tahmid Rahman Laskar,Israt Jahan,Elham Dolatabadi, Chun Peng, Enamul Hoque, and Jimmy Huang. Improving automatic evaluation of large language models (LLMs) in biomedical relation extraction via LLMs-as-the-judge.In Wanxiang Che, Joyce Nabende, Ekaterina Shutova,and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25483â25497, Vienna, Austria, July 2025. Association for Computational Linguistics. 15 31. Corey Curran, Nafis Neehal, Keerthiram Murugesan, and Kristin P Bennett. Examining trustworthiness of llm-as-a- judge systems in a clinical trial design benchmark. In 2024 IEEE International Conference on Big Data (BigData), pages 4627â4631. IEEE, 2024. 32. Emma Croxford, Yanjun Gao, Elliot First, Nicholas Pellegrino, Miranda Schnier, John Caskey, Madeline Oguss, Graham Wills, Guanhua Chen, Dmitriy Dligach, et al. Automating evaluation of ai text generation in healthcare with a large language model (llm)-as-a-judge. medRxiv, pages 2025â04, 2025. 33. Jiashuo Wang, Haozhao Wang, Shichao Sun, and Wenjie Li. Aligning language models with human preferences via a bayesian approach.Advances in Neural Information Processing Systems, 36:49113â49132, 2023. 34. Tom Kocmi and Christian Federmann.Large language models are state-of-the-art evaluators of translation quality. In Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassenhove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escart Ìın, Mikel Forcada, Maja Popovic, Carolina Scarton, and Helena Moniz, editors, Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193â203, Tampere, Finland, June 2023. European Association for Machine Translation. 35. Negar Arabzadeh and Charles LA Clarke.A human- ai comparative analysis of prompt sensitivity in llm- based relevance judgment.In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2784â2788, 2025. 36. Wenxuan Wang, Zizhan Ma, Zheng Wang, Chenghan Wu, Jiaming Ji, Wenting Chen, Xiang Li, and Yixuan Yuan. A survey of LLM-based agents in medicine: How far are we from baymax? In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 10345â10359, Vienna, Austria, July 2025. Association for Computational Linguistics. 37. Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik S Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, and Hae W Park. Mdagents: An adaptive collaboration of llms for medical decision- making.Advances in Neural Information Processing Systems, 37:79410â79452, 2024. 38. Kai Chen, Xinfeng Li, Tianpei Yang, Hewei Wang, Wei Dong, and Yang Gao. Mdteamgpt: A self-evolving llm- based multi-agent framework for multi-disciplinary team medical consultation. arXiv preprint arXiv:2503.13856, 2025. 39. David Vilares and Carlos G Ìomez-Rodr Ìıguez. HEAD-QA: A healthcare dataset for complex reasoning.In Anna Korhonen, David Traum, and Llu Ìıs M`arquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 960â966, Florence, Italy, July 2019. Association for Computational Linguistics. 40. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits.What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021. 41. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pages 2567â2577, Hong Kong, China, November 2019. Association for Computational Linguistics. 42. Guangfu Guo, Kai Zhang, Bryan Hoo, Yujun Cai, Xiaoqian Lu, Nanyun Peng, and Yiwei Wang. Structured outputs enable general-purpose llms to be medical experts. arXiv preprint arXiv:2503.03194, 2025. 16Zaifu Zhan et al. 43. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pages arXivâ2407, 2024. 44. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. 45. Qwen Team.Qwen2.5: A party of foundation models, September 2024. 46. Marah Abdin, Jyoti Aneja, Harkirat Behl, S Ìebastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al.Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024. 47. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open- source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. 48. OpenAI. gpt-oss-120b & gpt-oss-20b model card, 2025. 49. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019. 50. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the- art natural language processing. In Qun Liu and David Schlangen, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38â45, Online, October 2020. Association for Computational Linguistics.