Paper deep dive
Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models
Isha Puri, Mehul Damani, Idan Shenfeld, Marzyeh Ghassemi, Jacob Andreas, Yoon Kim
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/27/2026, 1:17:07 AM
Summary
The paper introduces 'Multi-Answer Reinforcement Learning' (Multi-Answer RL), a training framework for Large Language Models (LMs) that enables them to generate a distribution of plausible answers in a single forward pass. By modifying the RL objective to optimize for set-level correctness and calibration, the approach improves diversity, coverage, and efficiency across medical diagnosis, ambiguous question answering, and coding tasks, outperforming traditional single-answer RL baselines.
Entities (6)
Relation Signals (3)
Multi-Answer Reinforcement Learning → evaluatedon → DDXPlus
confidence 100% · We empirically validate our approach on ambiguous and multi-label reasoning tasks (a general question answering dataset with incomplete information, a medical diagnostic dataset, and a coding dataset).
Multi-Answer Reinforcement Learning → evaluatedon → MBPP
confidence 100% · We empirically validate our approach on ambiguous and multi-label reasoning tasks... and a coding dataset (MBPP).
Multi-Answer Reinforcement Learning → improves → Coverage
confidence 95% · We find that multi-answer RL scales better than baselines, producing substantial improvements in coverage, answer diversity, and token efficiency.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Given a question, a language model (LM) implicitly encodes a distribution over possible answers. In practice, post-training procedures for LMs often collapse this distribution onto a single dominant mode. While this is generally not a problem for benchmark-style evaluations that assume one correct answer, many real-world tasks inherently involve multiple valid answers or irreducible uncertainty. Examples include medical diagnosis, ambiguous question answering, and settings with incomplete information. In these cases, we would like LMs to generate multiple plausible hypotheses, ideally with confidence estimates for each one, and without computationally intensive repeated sampling to generate non-modal answers. This paper describes a multi-answer reinforcement learning approach for training LMs to perform distributional reasoning over multiple answers during inference. We modify the RL objective to enable models to explicitly generate multiple candidate answers in a single forward pass, internalizing aspects of inference-time search into the model's generative process. Across question-answering, medical diagnostic, and coding benchmarks, we observe improved diversity, coverage, and set-level calibration scores compared to single answer trained baselines. Models trained with our approach require fewer tokens to generate multiple answers than competing approaches. On coding tasks, they are also substantially more accurate. These results position multi-answer RL as a principled and compute-efficient alternative to inference-time scaling procedures such as best-of-k. Code and more information can be found at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.24844v1
- Canonical: https://arxiv.org/abs/2603.24844v1
Trouble viewing inline? Open PDF directly →
Full Text
79,150 characters extracted from source content.
Expand or collapse full text
Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models Isha Puri ∗ Mehul DamaniIdan ShenfeldMarzyeh Ghassemi Jacob AndreasYoon Kim Massachusetts Institute of Technology Abstract Given a question, a language model (LM) implicitly encodes a distribution over possible answers. In practice, post-training procedures for LMs often collapse this distribution onto a single dominant mode. While this is generally not a problem for benchmark-style evaluations that assume one correct answer, many real-world tasks inherently involve multiple valid answers or irreducible uncertainty. Exam- ples include medical diagnosis, ambiguous question answering, and settings with incomplete information. In these cases, we would like LMs to generate multiple plausible hypotheses, ideally with confidence estimates for each one, and without (computationally intensive) repeated sampling to generate non-modal answers. This paper describes a multi-answer reinforcement learning approach for training LMs to perform distributional reasoning over multiple answers during inference. We modify the RL objective to enable models to explicitly generate multiple candi- date answers in a single forward pass, internalizing aspects of inference-time search into the model’s generative process. Across question-answering, medical diagnostic, and coding benchmarks, we observe improved diversity, coverage, and set-level calibration scores compared to single answer-trained baselines. Models trained with our approach require fewer tokens to generate multiple answers than competing approaches. On coding tasks, they are also substantially more accurate. These results position multi-answer RL as a principled and compute-efficient alternative to (externalized) inference-time scaling procedures like best-of-k. Code and more information can be found at multi-answer-rl.github.io. <ans> <ans1> <ans2> <ans3> “Given preliminary symptoms, list plausible diagnoses.” Standard RL (3) × samples from the distribution, collapses to the mode Multi Answer RL returns the whole distribution, encourages diversity Multi-Answer RL recovers answers that RLVR-Single never reaches. Tuberculosis RLVR Multi RLVR Single Tuberculosis Tuberculosis Tuberculosis, 50% Pneumonia, 25% Bronchitis, 25% Figure 1: While standard RL trains LMs to consistently output the most likely answer to a question, Multi-Answer Reinforcement Learning trains models to output distributions of diverse answers. 1 Introduction Modern large language models (LMs) are typically (post-)trained via reinforcement learning to reason in natural language in order to produce a single best answer per query, implicitly ∗ Correspondence to ishapuri@mit.edu. arXiv:2603.24844v1 [cs.LG] 25 Mar 2026 Reaching Beyond the Mode: RL for Distributional Reasoning in LMs incentivizing the most likely correct answer (Guo et al., 2025). This objective is funda- mentally mismatched to many real-world settings in which multiple distinct answers may be simultaneously correct, or where uncertainty arises due to incomplete or ambiguous information. As a motivating example, consider a clinical setting in which a patient comes in complaining of right lower quadrant abdominal pain and fever. A clinician in this setting might suspect a diagnosis of acute appendicitis or a right-sided kidney stone, but be unsure as to which due to incomplete information (i.e., epistemic uncertainty over a single true diagnosis). To check, they might order a complete blood count and a urinalysis, both of which would be clinically appropriate tests to order (i.e., multiple correct answers). For such applications, models should ideally be able to go beyond single answers and give a set of answers. However, in challenging tasks like math, coding, or medical diagnosis, LMs must typically generate a long natural-language reasoning chain before producing even a single correct answer. Training LMs to reason their way toward a single correct answer (e.g. using reinforcement learning) can often suppress alternative plausible hypotheses, leading models to repeatedly generate the same dominant answer even when other correct possibilities exist. Indeed, entropy collapse is a commonly documented failure mode of models trained via RL with binary rewards (Lin et al., 2025; Yu et al., 2025; Jin et al., 2025; Wu & Choi, 2025). Beyond requiring a set of answers, many applications of interest—especially in high-stakes settings—would further benefit from uncertainty estimates associated with each answer (Kapoor et al., 2024). In other words, we ideally want the response to be a explicit distribution over plausible responses, including their estimated probabilities. Inference-time techniques have been proposed to address some of these limitations: tech- niques for sampling multiple answers through sampling in parallel (Puri et al., 2025; Beirami et al., 2025) or sequentially (Shinn et al., 2023; Xie et al., 2023) can produce an answer set for a given query; for uncertainty, models could be prompted or trained to verbalize uncertainty estimates for each answer (Xiong et al., 2024; Lin et al., 2022; Yang et al., 2025; Damani et al., 2025), or a separate model could be trained on top of model representations to output uncer- tainty scores (Liu et al., 2023; Azaria & Mitchell, 2023). However, these post-hoc methods do not change the underlying training objective, and this train-test mismatch could lead to poor performance in downstream decision-making scenarios where exploring and producing calibrated uncertainty estimates across multiple possibilities is crucial. This problem is particularly acute for models trained with modern post-training methods (as opposed to, say, base models), as while RLVR has made real strides in teaching models to reason, that training optimizes for a single high-reward completion, reasoning trace included. Recover- ing alternatives through repeated sampling then becomes both computationally expensive and behaviorally misaligned: the model has been trained to commit to one answer, not to hedge across several. How can we move beyond single-answer RL objectives and instead train models to represent and generate sets of plausible answers directly? If diversity of responses and calibrated uncertainty estimates are desirable properties of language model outputs, then they should be explicitly optimized for during training rather than heuristically recovered at inference time. To this end, we propose Multi-Answer RL, an RL approach which explicitly optimizes language models to generate the distribution P over answers directly. Concretely, multi-answer RL trains models to reason jointly over multiple plausible hy- potheses within a single chain of thought and to verbalize structured sets of candidate answers in a single generation. Our approach supports both single-answer and multi- answer ground-truth settings and encourages explicit hypothesis exploration rather than repeated resampling. By introducing a reward function inspired by proper scoring rules that incentivizes calibrated answer distributions, we further show how this set generation approach can be extended to produce verbalized confidence scores for each answer in the set, thus making it into the full distribution P. We empirically validate our approach on ambiguous and multi-label reasoning tasks (a general question answering dataset with incomplete information, a medical diagnostic dataset, and a coding dataset). We find that multi-answer RL scales better than baselines, producing substantial improvements in coverage, answer diversity, and token efficiency. For example, on coding tasks, our approach boosts top-1 accuracy by over 50% while cutting Reaching Beyond the Mode: RL for Distributional Reasoning in LMs token usage by more than half. We further show that our set-level calibration reward enables models to produce better-calibrated uncertainty scores at the set level. 2 Background We start by describing the standard reinforcement learning setting for language models. A policy parameterized as an LMπ θ maps a promptx ∈ Xto a distribution over textual outputsy ∈ Y. Given a datasetD = (x i ,y ∗ i )of prompt–answer pairs and a reward function R : Y× Y → R, training aims to maximize expected reward: max θ E (x,y ∗ )∼D, y∼π θ (·|x) R(y, y ∗ ) .(1) This approach trains the LM to output samples which have high reward on average. In practice, this can result in policies that place much of its mass on single outputs. Reinforcement learning with verifiable rewards (RLVR). RLVR focuses on the class of reward functions where rewards are deterministically verifiable from model outputs. A standard choice is the binary correctness reward, R correct (y, y ∗ ) = 1 y≡y ∗ ,(2) where1 y≡y ∗ ∈ 0, 1indicates whether the outputymatches the ground-truth answery ∗ (e.g. up to formatting differences). Reinforcement learning with calibration rewards (RLCR). A key limitation of RLVR is that models are incentivized to guess when uncertain. Building on the theory of proper scoring rules (Gneiting & Raftery, 2007), RLCR prompts models to produce reasoning chains that output both an answeryand an associated confidence estimateq ∈ [0, 1](Damani et al., 2025). Models are then trained using a reward that jointly incentivizes correctness and calibration: R RLCR (y, q, y ∗ ) = 1 y≡y ∗ − S q, 1 y≡y ∗ ,(3) whereSis a proper scoring rule. Damani et al. prove that for a particular class of proper scoring rules, RLCR incentivizes predictions are both accurate (in the sense that the highest- probability answer is generated) and calibrated. In practice, Damani et al. use the Brier score, which yields S(q, 1 y≡y ∗ ) = (q− 1 y≡y ∗ ) 2 . 3 Multi-Answer Reinforcement Learning 3.1 Multi-Answer RLVR We consider a generalized setting in which a promptxis associated not with a single ground- truth answer, but with a set of valid answersY ∗ (x) = y ∗ 1 ,y ∗ 2 ,. . .,y ∗ N , whereN ≥1 may vary across instances. Our goal is to train a model to recover the full set—or a high-coverage subset—of these valid answers within a single generation. We prompt the policyπ θ to produce a structured output consisting of a set ofKdistinct candidate answers, A =a 1 , a 2 , . . . , a K , within a single chain of thought. We then train the model using a set-level reward that checks how many of the generated answers belong to the ground-truth set: R multi RLVR (A,Y ∗ ) = K ∑ i=1 1 [ a i ∈Y ∗ ] ,(4) This objective can be viewed as a natural generalization of RLVR from single-answer correctness to set-level correctness and subsumes several familiar training objectives: Reaching Beyond the Mode: RL for Distributional Reasoning in LMs 1. N =1,K =1 (Standard RLVR). When there is a single ground-truth answer and the model produces a single output, the reward reduces exactly to the binary correctness signal used in vanilla RLVR. 2. N =1,K>1 (Pass@K). When there is a single correct answer but the model produces multiple candidates, the reward is equivalent to a pass@K objective. 3. N>1,K ≤ N(Partial set recovery). When multiple correct answers exist but the model is constrained to produce fewer candidates than the size of the ground-truth set, the objective encourages maximal coverage of distinct valid answers. 4. N>1,K ≥ N(Full set recovery). When allowed to generate at least as many candidates as valid answers, the optimal policy recovers the entire ground-truth set. Since we want to output distinct answers from the model, we use an additional format reward to enforce uniqueness among the generated answers. Full details are in Appendix B. 3.2 Multi-Answer RLCR Building on Multi-Answer RLVR, we introduce Multi-Answer RLCR, which additionally trains models to produce calibrated confidence estimates per answer. In a single chain- of-thought, the model is asked to output both a set ofKdistinct candidate answersA = a 1 ,a 2 ,. . .,a K and a corresponding set of confidence valuesQ = q 1 ,q 2 ,. . .,q K where each q i ∈ [0, 1] represents the model’s reported confidence that the answer a i is correct. As in RLCR, training jointly optimizes answer correctness and confidence calibration by combining a correctness-based reward with a proper scoring rule. To measure calibration in the multi-answer setting, we use the Multi-Brier score, defined as the average squared error between reported confidences and answer-level correctness: R multi Brier (A, Q,Y ∗ ) = 1 K K ∑ i=1 q i − 1 [ a i ∈Y ∗ ] 2 . Intuitively, this objective trains each confidenceq i to approximate the probability that the specific answera i is correct. The full objective, which jointly incentivizes coverage of valid answers and calibrated confidence estimates at the answer level, is given by R multi RLCR (A, Q,Y ∗ ) = R multi RLVR (A,Y ∗ ) − R multi Brier (A, Q,Y ∗ ). This recovers RLCR (Damani et al., 2025) as a special case when N = K = 1. Model output as a distribution.The model’s output(A,Q)can be interpreted as defining a distribution over plausible answers. WhenN =1, our format reward enforces that the Kconfidence scores sum to at most one, yielding a discrete probability distribution over the answer set. WhenN>1, confidence scores do not necessarily need to sum to one, and the output naturally corresponds to a multivariate Bernoulli distribution over correctness events. This approach is also closely related to conformal prediction approaches (Angelopoulos & Bates, 2023). In our approach however models learn to output theKmost likely answers fromp(y| x)rather than guaranteeing that these answers will cover some fixed fraction of its probability mass. 4 Experiments 4.1 Experimental Setup Training Details. We use GRPO as the base RL algorithm with some modifications (see Appendix B). Specifically, we use the Qwen3-8B base models with a max response length of 1536. To enable easier verification and more structured outputs, we augment our reward with a simple format reward that encourages models to enclose CoTs within the right tags and keep answers produced in Multi-Answer sets unique. Reaching Beyond the Mode: RL for Distributional Reasoning in LMs Condensed RLCR-Multi Example Output <think>Tuberculosis is a consideration, [...] then there’s pneumonia, which can cause similar symptoms. [...] Another possibility is bronchitis, but again, hemoptysis is more common in other conditions. [...] Given the hemoptysis and fever, TB and pneumonia are higher. Since the patient is underweight, TB might be more likely as undernutrition can impair immune response. [...] Bronchitis is possible but less so. [...] ranking TB first, pneumonia second, bronchitis third[...] Assigning confidences: TB maybe 0.4, pneumonia 0.30, bronchitis 0.30.</think> <a1>Tuberculosis</a1><conf1>0.40</conf1> <a2>Pneumonia</a2><conf2>0.30</conf2> <a3>Bronchitis</a3><conf3>0.30</conf3> Datasets. We evaluate our approach on three datasets (DDXPLUS, HOTPOTQA- MODIFIED, MBPP) that require set-valued reasoning, but differ in structure: DDXPlus admits multiple simultaneously correct answers given incomplete medical information; MBPP consists of well-specified, unambiguous tasks that admit multiple correct imple- mentations via distinct algorithmic approaches; and HotPotQA-Modified has a single gold answer but significant ambiguity due to incomplete or underspecified context. 1. DDXPlus (Tchango et al., 2022) is a large-scale medical diagnostic dataset in which each example consists of basic patient demographics along with a brief description of symptoms and antecedents. The target output is a differential diagnosis—a set of medical conditions that are plausible given the available information. Because multiple diagnoses may be simultaneously correct (N ≥1), performance is naturally evaluated using coverage over the set of gold answers rather than top-1 accuracy. A representative example is shown in Figure 1. We train on 25,000 examples, evaluate correctness using exact string match, and prompt models to generate K = 3 diagnoses in a single output. 2. HotPotQA-Modified is a modified version of the HotPotQA distractor dataset (Yang et al., 2018). Each example contains a multi-hop question with 10 paragraphs (2 relevant, 8 distractors), where we remove 1 or both relevant paragraphs to vary information complete- ness. Although each question retains a single ground-truth answer (N =1), incomplete information introduces ambiguity, making it beneficial for models to reason over and output multiple plausible candidates. This setting corresponds to theN =1,K>1 regime, where the Multi-Answer RL objective corresponds to a pass@K reward. 3. Coding: MBPP (Austin et al., 2021) is a benchmark of crowd-sourced programming tasks, each with a natural language description and unit tests, and is well-specified and unambiguous. While multiple implementations may be correct, they correspond to distinct algorithmic approaches for solving the same underlying task, rather than different valid answers. We measure the number of unique answers using AST-based uniqueness (i.e., answers are considered distinct if their abstract syntax trees differ). We use MBPP to represent the low-ambiguity, multi-solution regime, enabling evaluation of Multi-Answer RL across the full spectrum—from inherently multi-answer (DDXPlus), to ambiguous single-answer (HotPotQA-Modified), to structured code generation tasks. Methods. We evaluate the following methods: 1.Base: The base pre-trained model (Qwen3-8B). In our experiments, we prompt the base model with both single and multi versions of RLVR and RLCR prompts. 2.RLVR Single: Initialized from the base model and trained using standard RLVR to output a single answer. During evaluation, the model is also prompted to output a verbalized confidence. 3.RLCR Single: Initialized from the base model and trained using RLCR (Preliminar- ies) to output a single answer and corresponding confidence. 4. RLVR Single Prompted with RLVR Multi Prompt: We prompt the trained RLVR Single model to produce multiple answers using RLVR Multi prompt. Reaching Beyond the Mode: RL for Distributional Reasoning in LMs 5.RLCR Single Prompted with RLCR Multi Prompt: We prompt the trained RLCR Single model to produce multiple answers and confidences using the RLCR Multi system prompt. 6.RLVR Multi (ours): Trained withR RLVRMulti .Answers are extracted from <answeri> tags. 7.RLCR Multi (ours): Trained withR RLCR Multi . Answers and confidences are ex- tracted from <answeri> and <confidencei> tags. We compare Single-Answer and Multi-Answer methods by constructing sets ofKcandidate answers in both cases. For Single models, we sampleKindependent responses and treat them as a set, while Multi models naturally produce a single set ofKanswers in a single generation. This comparison allows us to isolate the effect of multi-answer training from inference-time sampling. Correctness Metrics.We evaluate correctness, diversity, and efficiency using the following metrics: 1.Coverage (Avg. # Correct per Set) (↑): Measures the average number of correct answers produced per example. Formally, for a set of K answers, Coverage = 1 K K ∑ i=1 1a i is correct.(5) 2.Pass@1 (↑): Measures the accuracy of a single selected answer from the generated set. For multi-answer methods, we use the first answer in the set, which we empirically find the model treats as most likely (implicitly in the RLVR case, explicitly in the RLCR case). For single-answer RLVR with independent samples, we select an answer uniformly at random from the set. 3.Avg. Token Count (↓): The average, over all questions, of the total token count across K generated answers. 4.Uniqueness (↑): The number of distinct answers within the generated set, measur- ing output diversity. Calibration Metrics.All methods are prompted to output a confidence scoreq i ∈ [0, 1]for each generated answer a i . We evaluate calibration using the following metrics: 1. Brier Score (↓). Measures the squared error between predicted confidence and binary correctness: Brier = 1 K K ∑ i=1 q i − 1a i ∈Y ∗ 2 .(6) We report Brier scores for the top-ranked answer (used for Pass@1) and for all answers pooled across the set. 2. Expected Calibration Error (ECE) (↓). Measures the discrepancy between predicted confidence and empirical accuracy by binning confidence scores: ECE = M ∑ m=1 |B m | L | acc(B m )− conf(B m ) | ,(7) whereMis the number of bins,B m denotes the samples in binm, andLis the total number of samples. We useM =10, and report ECE both for the top-ranked answer and the entire set of answers. 3.Set ECE (↓, diagnostic). Measures calibration at the level of answer sets. For each example, we define set-level correctnessy set = 1∃i : a i ∈Y ∗ . For datasets where questions only admit one correct answer (and thus set level probabilities must sum to 1, we define set-level confidenceq set = ∑ K i=1 q i . When there are multiple correct answers and probabilities can sum to>1, set-level confidence is defined asq set =1− ∏ K i=1 (1− q i ). Set ECE is computed by binningq set and comparing empirical set accuracy to predicted set confidence, much like standard ECE. Reaching Beyond the Mode: RL for Distributional Reasoning in LMs (a) DDXPlus: Medical Differential Diagnoses MethodAvg # Correct DiversityEfficiencyTop-1 Acc. (↑)(↑)(↓)(↑) RLVR (Single Loss + Single Prompt)0.620.6211910.50 RLVR (Single Loss + Multi Prompt)0.670.996950.42 Zero-Shot (Multi Prompt)0.310.9311310.24 Multi-Answer RLVR (Multi Loss + Multi Prompt) (Ours) 0.791.006770.42 RLCR (Single Loss + Single Prompt)0.650.5013780.55 RLCR (Single Loss + Multi Prompt)0.491.07030.32 Zero-Shot (Multi Prompt)0.410.8912690.16 Multi-Answer RLCR (Multi Loss + Multi Prompt) (Ours) 0.771.005100.43 (b) HotPotQA-Modified MethodAvg # Correct DiversityEfficiencyTop-1 Acc. (↑)(↑)(↓)(↑) RLVR (Single Loss + Single Prompt)0.170.5914660.19 RLVR (Single Loss + Multi Prompt)0.231.005510.17 Zero-Shot (Multi Prompt)0.200.7512650.17 Multi-Answer RLVR (Multi Loss + Multi Prompt) (Ours) 0.271.005440.19 RLCR (Single Loss + Single Prompt)0.220.6217820.17 RLCR (Single Loss + Multi Prompt)0.230.986700.17 Zero-Shot (Multi Prompt)0.200.7312980.15 Multi-Answer RLCR (Multi Loss + Multi Prompt) (Ours) 0.271.006220.19 (c) Coding: MBPP MethodAvg # Correct DiversityEfficiencyTop-1 Acc. (↑)(↑)(↓)(↑) RLVR (Single Loss + Single Prompt)0.982.09511.730.29 RLVR (Single Loss + Multi Prompt)0.951.93695.130.35 Zero-Shot (Multi Prompt) [RLVR]0.911.94664.780.34 Multi-Answer RLVR (Multi Loss + Multi Prompt) (Ours) 1.352.98235.490.49 RLCR (Single Loss + Single Prompt)0.892.13518.070.27 RLCR (Single Loss + Multi Prompt)0.921.90724.890.35 Zero-Shot (Multi Prompt) [RLCR]0.972.02677.930.37 Multi-Answer RLCR (Multi Loss + Multi Prompt) (Ours) 1.382.94250.940.48 Table 1: Correctness, diversity, and efficiency for Multi-Answer RL. Multi-Answer RL (RLVR / RLCR) substantially improves coverage, diversity, and efficiency over single-answer baselines in DDXPlus (a), HotPotQA-Modified (b), and MBPP (Coding) (c). All results use k = 3. Comparability Across Settings. Answer sets produced by single-answer and multi-answer methods differ in a fundamental way. Single-answer methods construct sets via repeated sampling, which often results in duplicate answers and variable effective set sizes. In contrast, multi-answer methods explicitly generate a set of distinct candidates. Because of this mismatch, pooled and set-level calibration metrics are not directly comparable across Reaching Beyond the Mode: RL for Distributional Reasoning in LMs RLVR-Single vs. RLVR-Multi: MBPP Output Diversity Example Prompt:Write a python function to check whether the given two numbers have same number of digits. RLVR-Multi (3 samples): [Candidate 1] def same_number_of_digits(num1, num2): len1 = len(str(num1)) len2 = len(str(num2)) return len1 == len2 [Candidate 2] def same_number_of_digits(num1, num2): digits1 = 0 digits2 = 0 while num1 > 0: num1 //= 10 digits1 += 1 while num2 > 0: num2 //= 10 digits2 += 1 return digits1 == digits2 [Candidate 3] def same_number_of_digits(num1, num2): str_num1 = str(num1) str_num2 = str(num2) return len(str_num1) == len(str_num2) RLVR-Single (3 samples): [Candidate 1] def same_number_of_digits(num1, num2): return len(str(num1)) == len(str(num2)) [Candidate 2] def same_number_of_digits(a, b): return len(str(a)) == len(str(b)) [Candidate 3] def same_number_of_digits(num1, num2): return len((str(num1)) == len(str(num2))) RLVR-Single repeatedly produces identical programs, while RLVR-Multi generates function- ally correct but structurally diverse implementations. single and multi settings, as their values depend on set size. Accordingly, we use pooled and set-level calibration metrics only to compare RLVR Multi and RLCR Multi, where the answer set construction mechanism is shared. In contrast, Top-1 Individual ECE and Top-1 Brier Score evaluate calibration of the highest-ranked answer, and are directly comparable. 51015202530 k (total answers) 0.6 0.7 0.8 0.9 1.0 1.1 1.2 Avg Coverage DDXPlus: Average Coverage over k RLVR Multi RLVR Single 36912151821242730 k (total answers) 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 Avg Coverage MBPP: Average Coverage over k RLVR Multi RLVR Single Figure 2: On DDXPLUS (left) and MBPP (right), we generate 30 answers from RLVR - 30 individual samples from RLVR Single, and 10 sets of 3 from RLVR Multi. Despite equal total generations, RLVR-Multi produces significantly more unique correct answers than RLVR-Single, indicating that RLVR-Single’s mode-seeking behavior limits diversity. 4.2 Main Results In this section, we compare Single-Answer to Multi-Answer RL, as well as RLVR to RLCR, along both correctness and calibration on DDXPLUS, HOTPOTQA-MODIFIED, and MBPP. Correctness.Table 1 reports correctness results across all settings. Across both RLVR and RLCR, Multi-Answer models substantially outperform their Single-Answer counterparts Reaching Beyond the Mode: RL for Distributional Reasoning in LMs on set-level correctness metrics including coverage. Moreover, simply prompting single- answer models to produce multiple answers performs substantially worse than trained multi-answer models, showing that generating answer sets requires explicit multi-answer training. On DDXPLUS, where multiple diagnoses may be simultaneously correct, coverage is the primary notion of correctness. Multi-Answer models recover a significantly larger fraction of the gold differential diagnoses per example, demonstrating that explicitly optimizing for set-valued outputs enables models to capture relevant alternatives that might be suppressed by single-answer RL. A similar trend is observed on MBPP, where Multi-Answer models produce a more diverse set of correct implementations, capturing distinct algorithmic approaches that are often missed under single-answer training. On HOTPOTQA-MODIFIED, each question admits a single correct answer but missing information induces significant uncertainty. In this setting, Multi-Answer RL yields large gains in pass@k, reflecting improved coverage of plausible answers. This improvement arises despite using just a single generation: by reasoning over multiple plausible hypotheses within a single generation, Multi-Answer models are more likely to include the correct answer among their candidates. In contrast, Single-Answer models exhibit answer collapse, repeatedly regenerating the same dominant answer, limiting coverage and under performing on pass@k. (a) DDXPlus: Medical Method Set ECE Top-1 ECE Top-k ECE Top-1 Brier Top-k Brier (↓)(↓)(↓)(↓)(↓) RLVR Single–0.34–0.35– RLCR Single–0.02–0.24– RLVR Multi0.130.160.100.270.19 RLCR Multi0.020.010.040.240.18 (b) HotPotQA-Hard: Trivia Method Set ECE Top-1 ECE Top-k ECE Top-1 Brier Top-k Brier (↓)(↓)(↓)(↓)(↓) RLVR Single–0.48–0.38– RLCR Single–0.13–0.16– RLVR Multi0.470.380.220.300.16 RLCR Multi0.440.310.210.220.12 (c) MBPP: Coding Method Set ECE Top-1 ECE Top-k ECE Top-1 Brier Top-k Brier (↓)(↓)(↓)(↓)(↓) RLVR Single–0.56–0.55– RLCR Single–0.53–0.53– RLVR Multi0.440.540.510.510.48 RLCR Multi0.260.370.330.380.34 Table 2: Calibration metrics for Multi-Answer RL (k =3). RLCR Multi substantially improves set- and answer-level calibration over RLVR Multi on DDXPLUS and MBPP. On HOTPOTQA-HARD, RLCR Multi improves set-level calibration over RLVR Multi but underperforms RLCR Single at the top-1 answer level, consistent with a learned prior encouraging confidences to sum to one in extremely difficult single-label settings. Calibration. Table 2 reports calibration results across RLVR/RLCR Single/Multi. Across all three datasets, RLCR consistently improves calibration relative to RLVR in both Single- and Multi-Answer settings. Reaching Beyond the Mode: RL for Distributional Reasoning in LMs Figure 3: Calibration curves on DDXPlus. RLCR-Multi is significantly better calibrated than RLVR- Multi, though it diverges at higher confidences. RLVR-Multi remains systematically overconfident. The size of each dot corresponds to how many examples are found in that bucket. On DDXPLUS and MBPP, RLCR-Multi achieves markedly better set-level calibration than RLVR-Multi across all metrics, demonstrating RLCR’s ability to assign meaningful confi- dences to multiple plausible answers. Calibration curves (Figure 3) provide a complemen- tary view for DDXPLUS. RLCR-Multi is better calibrated than RLVR-Multi overall, though it still deviates from perfect calibration, especially at higher confidence levels. In contrast, RLVR-Multi exhibits systematic overconfidence across the range. RLCR-Multi closely tracks the identity line across confidence bins, while RLVR-Multi exhibits systematic overconfidence, particularly at higher confidence levels—effects that are less visible in aggregate metrics such as ECE. Finally, of the calibration metrics considered, only Top-1 ECE and Top-1 Brier permit direct comparison between Single- and Multi-Answer RL. These metrics show that Multi-Answer training matches Single-Answer RL in calibration, indicating that explicitly optimizing for set-valued outputs does not degrade calibration of the top answer. On HOTPOTQA-MODIFIED, the quantitative trends observed on DDXPLUS largely per- sist, with RLCR improving calibration relative to RLVR. Surprisingly, however, while it outperforms Multi-Answer RLVR, Multi-Answer RLCR underperforms Single-Answer RLCR, indicating degraded calibration. On inspection, we find that this behavior stems from the model having a strong generative prior to sum predicted confidences across the answer set to 1. In a challenging single–gold-answer setting such as HOTPOTQA-MODIFIED (pass@K<30%), well-calibrated behavior would instead assign total confidence strictly less than one for most questions. While, in principle, exploration during RL training could overcome this prior, we find that this does not occur in practice. We leave the development of improved exploration strategies for future work. 4.3 Analysis Does multi-answer RL improve output diversity? Many tasks admit multiple plausible answers, making it important for models to explore a broader set of hypotheses rather than collapsing to a single dominant mode. While Table 1 showed that multi-answer training improves coverage, indicating that models generate a wider range of plausible answers, we analyze diversity more directly here. To analyze diversity more directly, we compare RLVR- Single and RLVR-Multi under a matched sampling budget on both the medical domain (DDXPLUS) and coding (MBPP). We sample 30 generations from RLVR-Single, yielding 30 answers, and 10 generations from RLVR-Multi, yielding 10× K =30 answers total. We then compute the number of unique answers per set and plot the distribution in Figure 4. We Reaching Beyond the Mode: RL for Distributional Reasoning in LMs 024681012141618 Unique diagnoses (out of 30) 0.00 0.05 0.10 0.15 0.20 Density Diversity of Diagnoses per Question RLVR (Single) RLVR (Multi) Figure 4: Distribution of the number of unique diagnoses per question across 5,000 test examples of DDXPLUS. RLVR-Multi produces more distinct diagnoses than RLVR-Single, explaining coverage gains under multi-answer training. also plot average coverage—the mean number of correct answers per set—as a function of K in Figure 2 for both DDXPLUS and MBPP. Across RLVR-Multi exhibits greater output diversity, producing nearly twice as many unique answers on average (8 vs. 4). Moreover, its higher average coverage indicates that this increased diversity corresponds to a larger number of correct answers, rather than spurious variation. Taken together, these results demonstrate that multi-answer training increases diversity without compromising correctness, addressing a key failure mode of single-answer RL. Is multi-answer RL more compute efficient than single-answer RL?The standard approach to obtaining answer sets from single-answer models is inference-time scaling, most commonly by repeated sampling. Figure 5 illustrates a representative instance, showing three sampled generations from an RLVR-Single model. Although each generation produces a different final answer, the corresponding chains exhibit substantial overlap, repeatedly reproducing the same reasoning scaffolding and intermediate phrases. Quantitatively, this manifests as highn-gram overlap across samples (see Figure 5), indicating that inference- time sampling often incurs significant computational redundancy. To compare this redundancy against multi-answer training, we measure the number of tokens for a set ofKanswers under both approaches. For RLVR-Single, we sample the model Ktimes independently and sum the resulting token lengths. For RLVR-Multi, we generateK answers within a single response and measure its total token length. In the medical domain, the average token length of an RLVR-Multi response is only 56% of the total token length required by an RLVR-Single model to produce the same number of answers (Figure 7). These results show that training models to directly generate answer sets can eliminate a significant amount of redundant computation incurred by inference-time sampling. (A) RLVR-Single: Response 1 <think> Okay , let's tackle this medical case . The patient is a 35 - year - old female with several symptoms and antecedents . The key symptoms include worsening symptoms over two weeks , chest pain at rest , pain in multiple areas like lower and upper chest , breasts , epigastric region , and radiation to the biceps , thyroid cartilage , thoracic spine , under the jaw , and left shoulder . The pain is intense ( 4 / 10 ) and poorly localized ( score 2 ) . She also reports fatigue and non - restful sleep , with symptoms aggravated by exertion and relieved by rest . Looking at the antecedents : she's overweight , smokes , has high blood pressure , a history of heart attack or angina , exercises regularly , and has a family history of cardiovascular disease before age 50 . The combination of chest pain at rest , exertional angina , and a history of angina or heart attack points towards coronary artery disease . The risk factors like smoking , hypertension , obesity , and family history further support this . The radiation of pain to the left shoulder and jaw is classic for myocardial infarction . Despite regular exercise , her other risk factors are significant . The worsening symptoms and increased fatigue could indicate unstable angina or a new cardiac event . Given her history and current symptoms , the most likely diagnosis is unstable angina or acute coronary syndrome . But since the question asks for the most likely , and considering her existing history of angina , unstable angina is the probable answer .</think> <answer> Unstable angina </answer> (B) RLVR-Single: Response 2 Significant token-by- token overlap between responses which Multi- Answer RL avoids <think> Okay , let's tackle this medical case . The patient is a 35 - year - old woman with several symptoms and antecedents . The key symptoms include worsening symptoms over two weeks , chest pain at rest , pain in multiple areas like lower and upper chest , both breasts , epigastric region , and radiation to the biceps , thyroid cartilage , thoracic spine , under the jaw , and left shoulder . The pain is intense ( 4 / 10 ) and poorly located ( 2 / 10 ) . She also reports fatigue and non - restful sleep , with symptoms exacerbated by exertion and relieved by rest . Looking at the antecedents : she's overweight , smokes , has high blood pressure , a history of heart attack or angina , exercises regularly , and has a family history of cardiovascular disease before age 50 . The combination of chest pain at rest , exertional angina , and a history of angina or heart attack points towards coronary artery disease . The presence of multiple risk factors like obesity , smoking , hypertension , and family history further supports this . The radiation of pain to different areas is common in cardiac issues . Despite regular exercise , her other risk factors likely contribute to CAD . The worsening symptoms and increased fatigue could indicate unstable angina or a progression of CAD . Other possibilities like pericarditis or pulmonary issues are less likely given the history and symptoms . The most probable diagnosis is Coronary Artery Disease ( CAD ) with possible unstable angina .</think> <answer> Coronary Artery Disease </answer> Even when end answers are different Figure 5: Significant subsequence overlap between independently sampled RLVR-Single responses, even those that yield different answers, indicating that independent sampling largely re-instantiates the same reasoning tokens. Multi-Answer RL mitigates this effect by optimizing multiple generations jointly, which reduces repeated token sequences and yields lower within-question overlap. Reaching Beyond the Mode: RL for Distributional Reasoning in LMs Summary.Together, these results show that training models to explicitly optimize multi- answer structure results in outputs that are more accurate, calibrated, and diverse, while simultaneously being more efficient to generate. 4.4 Does multi-answer training remain stable with increasing K? To understand the performance of multi-answer training asKincreases, we train Multi- Answer RLVR withK = 2, 3, 4, 5. Because each medical example has multiple correct diagnoses, increasingkdirectly expands the model’s capacity to recover valid alternatives. As shown in Figure 6, the average number of correct answers per set increases monotonically withk. Since outputs are constrained to be unique, these gains reflect the model surfacing additional correct hypotheses rather than repeating dominant answers. Training remains stable across all values of K. Figure 6: As k increases, Multi-Answer RLVR stably recovers more unique correct diagnoses per set. 5 Related Work RLVR and Diversity. Recent work has documented systematic trade-offs between RL training and output diver- sity (Kirk et al., 2024; Shypula et al., 2025; Yang & Holtzman, 2025). (Wu & Choi, 2025) and (Yue et al., 2025) showed that improvements in pass@1 under RLVR are accompanied by a contraction of the model’s support, narrowing its reasoning space and leading to degraded pass@k performance. Similarly, (West & Potts, 2025) find that base models often outperform aligned models on creative tasks. Several works have proposed methods to preserve diver- sity during RL training. (Chen et al., 2025) and (Walder & Karkhanis, 2025) directly optimize pass@k-style objectives by rewarding responses based on their contribution to set-level success. (Li et al., 2025) augment standard quality rewards with a learned diversity reward. (Song et al., 2025) use exploration bonuses conditioned on final outcomes. A complementary line of work focuses on explicitly controlling policy entropy during training (Cui et al., 2025; Cheng et al., 2025). Most of these approaches operate in the single-answer setting, focusing on encouraging diversity rather than training models to explicitly generate answer sets within a single generation. Calibration. Reliable deployment of language models requires accurate uncertainty esti- mation (Kalai et al., 2025). Prior work has explored a range of approaches for estimating model confidence, including intent probing (Kadavath et al., 2022; Gupta et al., 2024; Azaria & Mitchell, 2023), sampling-based uncertainty estimation (Kuhn et al., 2023; Kang et al., 2025), and post-hoc verbalization (Xiong et al., 2024; Lin et al., 2022). Among these, post-hoc verbalization, where models output a confidence score after generating a response, has recently gained popularity. However, multiple works have shown that models are systemat- ically overconfident when verbalizing their confidence (Xiong et al., 2024; Mei et al., 2025; Kirichenko et al., 2025). To address this, recent work has focused on directly training models to produce calibrated confidence estimates. (Lin et al., 2022) fine-tune GPT-3 to predict confidence conditioned on a question and generated answer. A smaller line of work uses RL Reaching Beyond the Mode: RL for Distributional Reasoning in LMs with proper scoring rules (Brier, 1950; Gneiting & Raftery, 2007) as training rewards to align predicted confidence with empirical correctness. Within this paradigm, (Stangel et al., 2025) and (Xu et al., 2024) optimize purely for calibration, whereas (Damani et al., 2025) employ a joint objective that incentivizes both correctness and calibration. Generating Answer Sets. There have been several recent efforts aimed at eliciting answer sets from LMs. (Wang et al., 2024) prompt models to output probability distributions over a fixed set of labels. (Zhang et al., 2025) introduce verbalized sampling, a prompting strategy that asks models to explicitly verbalize a distribution over multiple responses. Similarly, (Troshin et al., 2025) show that prompting models to enumerate or iteratively sample answers can increase output diversity. Although similar in intuition, these approaches are training-free and operate purely at inference time. Most closely related to our work is concurrent research by (Wang et al., 2025), which extends RLCR (Damani et al., 2025) to train models to output verbalized probability distributions over answer sets. Although aligned in motivation, their primary focus is on calibration of answer distributions, whereas we study the broader benefits of multi-an swer training, including diversity (pass@K) and computational efficiency. 6 Conclusion We propose Multi-Answer Reinforcement Learning for language models, which trains reasoning LMs to generate sets of plausible answers rather than a single most probable re- sponse. Across medical and QA benchmarks, this objective improves coverage and diversity, recovering correct alternatives missed by standard RL while using fewer total tokens. More broadly, our work extends a line of approaches that use language to serialize structured reasoning, moving from single answers to explicit representations of output distributions. Limitations remain: single-answer objectives still achieve higher top-1 accuracy, our experi- ments are confined to QA, and serial generation limits parallelism despite improved token efficiency. Nevertheless, these results suggest a path toward models that can explore their full internal distribution without sacrificing performance or calibration. Reaching Beyond the Mode: RL for Distributional Reasoning in LMs Acknowledgments This research was supported by the MIT-IBM Watson AI Lab and the DARPA AIQ program through the DARPA CMO contract number HR00112520025. IP is supported by the NSF Graduate Research Fellowship. JA is additionally supported by a Sloan Research Fellowship. MG is supported in part by National Science Foundation (NSF) 22-586 Faculty Early Career Development Award (#2339381) and the AI2050 Program at Schmidt Sciences. YK was supported in part by the National Science Foundation under CAREER Award No. 2441872. References Anastasios N Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494–591, 2023. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732. Amos Azaria and Tom Mitchell. The internal state of an llm knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 967–976, 2023. Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander Nicholas D’Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=u3U8qzFV7w. Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1–3, 1950. Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@ k training for adaptively balancing exploration and exploitation of large reasoning models. arXiv preprint arXiv:2508.10751, 2025. Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506.14758, 2025. Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025. Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas. Beyond binary rewards: Training lms to reason about their uncertainty, 2025. URL https://arxiv.org/abs/2507.16806. Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359–378, 2007. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Kr- ishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=KgaBScZ4VI. Renren Jin, Pengzhi Gao, Yuqi Ren, Zhuowen Han, Tongxuan Zhang, Wuwei Huang, Wei Liu, Jian Luan, and Deyi Xiong. Revisiting entropy in reinforcement learning for large reasoning models. arXiv preprint arXiv:2511.05993, 2025. Reaching Beyond the Mode: RL for Distributional Reasoning in LMs Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. CoRR, abs/2207.05221, 2022. URL https://doi.org/10.48550/arXiv.2207.05221. Adam Tauman Kalai, Ofir Nachum, Santosh S Vempala, and Edwin Zhang. Why language models hallucinate. arXiv preprint arXiv:2509.04664, 2025. Zhewei Kang, Xuandong Zhao, and Dawn Song. Scalable best-of-n selection for large language models via self-certainty. In 2nd AI for Math Workshop @ ICML 2025, 2025. URL https://openreview.net/forum?id=nddwJseiiy. Sanyam Kapoor, Nate Gruver, Manley Roberts, Katie Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew G Wilson. Large language models must be taught to know what they don’t know. Advances in Neural Information Processing Systems, 37:85932–85972, 2024. Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J. Bell. Abstention- bench: Reasoning LLMs fail on unanswerable questions. In ICML 2025 Workshop on Reliable and Responsible Foundation Models, 2025. URLhttps://openreview.net/forum? id=kYbojsAOBj. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of RLHF on LLM generalisation and diversity. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=PXD3FAVHJT. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic in- variances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023. URLhttps://openreview.net/ forum?id=VD-AYtP0dve. Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lanchantin, and Tianlu Wang. Jointly reinforcing diversity and quality in language model generations. arXiv preprint arXiv:2509.02534, 2025. Jiacheng Lin, Zhenbang Wu, and Jimeng Sun. Training llms for ehr-based reasoning tasks via reinforcement learning. arXiv preprint arXiv:2505.24105, 2025. Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URLhttps: //openreview.net/forum?id=8s8K2UZGTZ. Kevin Liu, Stephen Casper, Dylan Hadfield-Menell, and Jacob Andreas. Cognitive dis- sonance: Why do language model outputs disagree with internal representations of truthfulness? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 4791–4797, 2023. Zhiting Mei, Christina Zhang, Tenny Yin, Justin Lidard, Ola Shorinwa, and Anirudha Majumdar. Reasoning about uncertainty: Do reasoning models know when they don’t know? arXiv preprint arXiv:2506.18183, 2025. Isha Puri, Shivchander Sudalairaj, Guangxuan Xu, Kai Xu, and Akash Srivastava. A probabilistic inference approach to inference-time scaling of llms using particle-based monte carlo methods. arXiv preprint arXiv:2502.01618, 2025. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634–8652, 2023. Reaching Beyond the Mode: RL for Distributional Reasoning in LMs Alexander Shypula, Shuo Li, Botong Zhang, Vishakh Padmakumar, Kayo Yin, and Osbert Bastani. Evaluating the diversity and quality of llm generated content. arXiv preprint arXiv:2504.12522, 2025. Yuda Song, Julia Kempe, and R ́ emi Munos. Outcome-based exploration for LLM rea- soning. In NeurIPS 2025 Workshop: Second Workshop on Aligning Reinforcement Learning Experimentalists and Theorists, 2025. URLhttps://openreview.net/forum?id=VORSpYLBJ6. Paul Stangel, David Bani-Harouni, Chantal Pellegrini, Ege ̈ Ozsoy, Kamilia Zaripova, Matthias Keicher, and Nassir Navab. Rewarding doubt: A reinforcement learning ap- proach to confidence calibration of large language models. CoRR, abs/2503.02623, March 2025. URL https://doi.org/10.48550/arXiv.2503.02623. Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn. DDX- Plus: A new dataset for automatic medical diagnosis. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URLhttps: //openreview.net/forum?id=heBKnuV42O. Sergey Troshin, Irina Saparina, Antske Fokkens, and Vlad Niculae. Asking a language model for diverse responses. In Proceedings of the 2nd Workshop on Uncertainty-Aware NLP (UncertaiNLP 2025), p. 66–72, 2025. Benjamin Turtel, Danny Franklin, Kris Skotheim, Luke Hewitt, and Philipp Schoeneg- ger.Outcome-based reinforcement learning to predict the future.arXiv preprint arXiv:2505.17989, 2025. Christian Walder and Deep Karkhanis. Pass@ k policy optimization: Solving harder rein- forcement learning problems. arXiv preprint arXiv:2505.15201, 2025. Ante Wang, Weizhi Ma, and Yang Liu. Don’t miss the forest for the trees: In-depth confidence estimation for llms via reasoning over the answer space. arXiv preprint arXiv:2511.14275, 2025. Cheng Wang, Gyuri Szarvas, Georges Balazs, Pavel Danchenko, and Patrick Ernst. Calibrat- ing verbalized probabilities for large language models. arXiv preprint arXiv:2410.06707, 2024. Peter West and Christopher Potts. Base models beat aligned models at randomness and creativity. arXiv preprint arXiv:2505.00047, 2025. Fang Wu and Yejin Choi. The invisible leash: Why rlvr may not escape its origin. In 2nd AI for Math Workshop@ ICML 2025, 2025. Changyi Xiao, Mengdi Zhang, and Yixin Cao. Bnpo: Beta normalization policy optimization. arXiv preprint arXiv:2506.02864, 2025. Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems, 36:41618–41650, 2023. Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=gjeQKFxFpZ. Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao. Sayself: Teaching llms to express confidence with self-reflective rationales. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 5985–5998, 2024. Chenghao Yang and Ari Holtzman. How alignment shrinks the generative horizon. arXiv preprint arXiv:2506.17871, 2025. Reaching Beyond the Mode: RL for Distributional Reasoning in LMs Daniel Yang, Yao-Hung Hubert Tsai, and Makoto Yamada. On verbalized confidence scores for LLMs. In ICLR Workshop: Quantify Uncertainty and Hallucination in Foundation Models: The Next Frontier in Reliable AI, 2025. URLhttps://openreview.net/forum?id= CVRdNQvFPE. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 2369–2380, Brussels, Belgium, October-November 2018. As- sociation for Computational Linguistics. doi: 10.18653/v1/D18-1259. URLhttps: //aclanthology.org/D18-1259/. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837, 2025. Jiayi Zhang, Simon Yu, Derek Chong, Anthony Sicilia, Michael R Tomz, Christopher D Manning, and Weiyan Shi. Verbalized sampling: How to mitigate mode collapse and unlock llm diversity. arXiv preprint arXiv:2510.01171, 2025. Reaching Beyond the Mode: RL for Distributional Reasoning in LMs A System Prompts RLCR Single Prompt A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind, provides the user with the final answer, then analyzes its confidence about the solution and then provides the user with its confidence level. The confidence level is a number between 0 and 1 (inclusive) enclosed within <confidence> </confidence> tags. The final answer is enclosed between <answer> </answer> tags. The analysis about confidence and uncertainty is enclosed within <analysis> </analysis> tags. Here are some guidelines for the analysis: 1. Your task is to point out things where the model could be wrong in its thinking, or things where there might be ambiguity in the solution steps, or in the reasoning process itself. 2. You should not suggest ways of fixing the response, your job is only to reason about uncertainties. 3. For some questions, the response might be correct. In these cases, it is also okay to have only a small number of uncertainties and then explicitly say that I am unable to spot more uncertainties. 4. Uncertainties might be different from errors. 5. If there are alternate potential approaches that may lead to different answers, you should mention them. 6. List out plausible uncertainties, do not make generic statements, be as specific as possible. 7. Enclose this uncertainty analysis within <analysis> </analysis> tags. The final format that must be followed is: <think> reasoning process here </think> <answer> final answer here </answer> <analysis> analysis about confidence and uncertainty here </analysis> <confidence> confidence level here (number between 0 and 1) </confidence> RLVR Single Prompt A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think><answer> answer here </answer>. Do NOT put any sentences or reasoning process within the <answer> </answer> tags - only put the final answer that will be verified with exact match score within the <answer> </answer> tags. RLCR Multi Prompt A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The question may be ambiguous or difficult to answer, and you must propose multiple possible answers. You must assign a confidence score to each candidate. Make sure the confidences sum to less than or equal to 1. The confidences are allowed to sum to less than 1 if you are unsure about all candidates. You will be graded on how closely your confidences match the actual correctness of the candidates. Output EXACTLY K DISTINCT candidates with confidences that sum to less than or equal to 1. FORMAT ONLY (no extra text): <think> reasoning process about different candidate answers here </think> <answer1> candidate_answer_1 </answer1> <confidence1> confidence level for candidate 1 here (number between 0 and 1) </confidence1> ... exactly K pairs ... Reaching Beyond the Mode: RL for Distributional Reasoning in LMs RLVR Multi Prompt A conversation between User and Assistant. The user asks a question, and the Assistant solves it. You must propose multiple possible answers, not just one. For each candidate, think separately about why it could be correct or incorrect. Output EXACTLY K DISTINCT candidate answers. FORMAT ONLY (no extra text): <think> reasoning process about different candidate answers here </think> <answer1> candidate_answer_1 </answer1> <answer2> candidate_answer_2 </answer2> ... exactly K answers ... B Additional Training Details We sample 32 responses per prompt using a temperature of 0.7 and train with an effective batch size of 1536. Optimization uses a constant learning rate with linear warmup, a base learning rate of 1×10 −6 , and a warmup ratio of 0.05. All experiments are conducted on NVIDIA A100 and H100 GPUs, and we observe consistent performance trends across hardware types. Following prior work (Turtel et al., 2025), we remove the standard deviation normalization from the advantage computation, which can improve learning stability in the presence of extreme miscalibration. Training is performed with the BNPO objective, which aggregates token-level losses normalized by the number of active tokens in each local training batch (Xiao et al., 2025). For both datasets, we set a maximum completion length of 1536. We do 1 epoch of training. System prompts for RLCR Single, RLCR Multi, RLVR Single, and RLVR Multi are in Appendix A. Format Reward: We use a format reward to encourage adherence to the structured format required by the system prompts. In RLVR Single, models must format their output in <think>and<answer>tags. In RLCR Single, in addition to<think>and<answer>tags, we require a<confidence>tag for verbalized confidence. RLVR Multi requires a<think>tag and thenk <answeri>tags. RLCR Multi requires a<think>tag and thenksets of<answeri> and<confidencei>tags. A valid response must contain all these tags in the correct order. Both format and calibration rewards are weighted equally. In the Multi-RLCR and Multi-RLVR setting, we enforce the uniqueness required by the prompt by zeroing out all rewards if the normalized answers extracted from the<answeri> tags are not unique. Additionally, in the N = 1 setting where our dataset assigns only one correct answer to each question, the format reward zeroes out if the sum of the confidences extracted from the <confidencei> tags is more than 1. C Token Efficiency Comparison: Single Answer vs. Multi Answer RL Here, we provide a visual of the average token compute usage of a set of answers generated by RLVR Single, RLVR Multi, RLCR Single, and RLCR Multi. As one can see, Multi-Answer training significantly increases compute efficiency. Figure 7: Average token usage for RLVR and RLCR in single and multi settings on DDXPlus. Reaching Beyond the Mode: RL for Distributional Reasoning in LMs D Unique Answer Analysis: RLVR Single vs. RLVR Multi We create a Word Cloud comparing the answer diversity of RLVR Single and RLVR Multi on an example medical question. We run RLVR Single 30 times and RLVR Multi 10 times, with k =3 answers per set. As one can see, RLVR single, even though it also collects 30 answers, only collects 3 unique answers. RLVR Multi, on the other hand, admits significantly more. Figure 8: Word Cloud comparing Unique Answers gathered by RLVR Single and RLVR Multi E Increasing k in RLVR-Multi kEvaluation Coverage, RLVR Multi 20.78 30.68 40.65 50.62 Table 3: We report coverage, or the average number of correct answers generated in a set, on an evaluation set of 5, 000 questions as k increases. Reaching Beyond the Mode: RL for Distributional Reasoning in LMs F Full Example - Medical RLVR Single, RLVR Multi, RLCR Single, RLCR Multi Question ================================================================================ SYMPTOMS AND ANTECEDENTS: ================================================================================ Symptoms: 1. Symptom: Have you been coughing up blood? -> Yes 2. Symptom: Do you have pain somewhere, related to your reason for consulting? -> Yes 3. Symptom: Characterize your pain:: a knife stroke 4. Symptom: Do you feel pain somewhere?: lower chest 5. Symptom: Do you feel pain somewhere?: posterior chest wall(R) 6. Symptom: Do you feel pain somewhere?: posterior chest wall(L) 7. Symptom: How intense is the pain?: 5 8. Symptom: Does the pain radiate to another location?: nowhere 9. Symptom: How precisely is the pain located?: 6 10. Symptom: How fast did the pain appear?: 5 11. Symptom: Are you experiencing shortness of breath or difficulty breathing in a significant way? -> Yes 12. Symptom: Do you have a fever (either felt or measured with a thermometer)? -> Yes 13. Symptom: Have you had chills or shivers? -> Yes 14. Symptom: Do you have any lesions, redness or problems on your skin that you believe are related to the condition you are consulting for? -> Yes 15. Symptom: What color is the rash?: pink 16. Symptom: Do your lesions peel off?: Y 17. Symptom: Is the rash swollen?: 1 18. Symptom: Where is the affected region located?: commissure(L) 19. Symptom: How intense is the pain caused by the rash?: 0 20. Symptom: Is the lesion (or are the lesions) larger than 1cm?: N 21. Symptom: How severe is the itching?: 0 22. Symptom: Do you have diffuse (widespread) muscle pain? -> Yes 23. Symptom: Have you noticed any new fatigue, generalized and vague discomfort, diffuse (widespread) muscle aches or a change in your general well-being related to your consultation today? -> Yes 24. Symptom: Do you have a cough? -> Yes Antecedents: 1. Antecedent: Do you drink alcohol excessively or do you have an addiction to alcohol? -> Yes 2. Antecedent: Do you have heart failure? -> Yes 3. Antecedent: Do you have a chronic obstructive pulmonary disease (COPD)? -> Yes 4. Antecedent: Have you traveled out of the country in the last 4 weeks?: N 5. Antecedent: Is your BMI less than 18.5, or are you underweight? -> Yes -------------------------------------------------------------------- CORRECT / GOLD ANSWERS: 1. Pneumonia 2. Pulmonary neoplasm 3. Bronchitis 4. Tuberculosis 5. Possible NSTEMI / STEMI 6. GERD 7. Unstable angina 8. Pericarditis 9. Stable angina Reaching Beyond the Mode: RL for Distributional Reasoning in LMs RLVR Multi Okay, let's tackle this case. The patient is a 61-year-old male with several symptoms. He's coughing up blood, which is a red flag. He has chest pain described as a knife stroke, localized to the lower chest and right posterior chest wall. The pain is severe and sudden. He also has shortness of breath, fever, chills, and a rash on the left commissure that's pink and peeling. Additionally, he has diffuse muscle pain, fatigue, and a history of alcohol abuse, heart failure, COPD, and being underweight. First, the hemoptysis (coughing up blood) and chest pain could point to a pulmonary issue. Given his COPD, maybe something like pulmonary embolism? But he also has heart failure, which could contribute to pulmonary issues. However, the sudden onset of severe pain and the presence of fever and chills might suggest an infection, like pneumonia. But the rash is another clue. The rash is on the commissure, pink, peeling, and swollen. That sounds like a fungal infection, maybe oral thrush, but the location is a bit unclear. Alternatively, could it be a drug reaction? He's on alcohol and has heart failure, so maybe medications are involved. But the main issues are the chest pain and hemoptysis. Another possibility is tuberculosis, especially with the fever, night sweats (though not mentioned here), and hemoptysis. But the rash doesn't fit TB. Alternatively, a fungal infection like histoplasmosis, but that's more common in certain regions. The rash might be a secondary infection. Wait, the rash is on the commissure, which could be a sign of a fungal infection like candidiasis, but that's more oral. Maybe it's a skin manifestation of something else. Considering the heart failure and COPD, could there be a pulmonary infection like pneumonia? The fever, chills, and shortness of breath support that. But the hemoptysis is more concerning for something like a pulmonary embolism or a tumor. However, the sudden onset and severe pain might point to a pulmonary embolism. But the rash is confusing. Alternatively, maybe the rash is a sign of a systemic infection, like sepsis, but that's a broader picture. Another angle: the patient's alcohol use and underweight status could predispose him to infections or nutritional deficiencies. The diffuse muscle pain and fatigue could be due to systemic illness. The combination of hemoptysis, chest pain, fever, and rash makes me think of a fungal infection like coccidioidomycosis or histoplasmosis, but those usually have other symptoms. Alternatively, maybe a drug-induced reaction, but without knowing medications, it's hard. So the top three would be pulmonary embolism, pneumonia, and tuberculosis. Wait, the rash might be a clue for something else. Alternatively, the rash could be a sign of a systemic fungal infection. So maybe the three are pulmonary embolism, pneumonia, and fungal infection (like histoplasmosis). But the user wants three distinct diagnoses. Let me check again. The key symptoms are hemoptysis, chest pain, fever, chills, and rash. So possible diagnoses: 1. Pulmonary embolism, 2. Pneumonia, 3. Tuberculosis. Alternatively, 3. Fungal infection. But the rash might be more specific. Maybe the rash is a sign of a drug reaction, but without knowing medications, it's hard. So I'l go with pulmonary embolism, pneumonia, and tuberculosis. </think> <answer1> Pulmonary Embolism </answer1> <answer2> Pneumonia </answer2> <answer3> Tuberculosis </answer3> Reaching Beyond the Mode: RL for Distributional Reasoning in LMs RLCR Multi <think> Okay, let's tackle this case. The patient is a 61-year-old male with several symptoms. He's coughing up blood, which is a red flag. He has chest pain described as a knife stroke, localized to the lower chest and right posterior chest wall. The pain is severe (intensity 5) and sudden (appeared quickly). He also has shortness of breath, fever, chills, and a rash on the left commissure that's pink, peeling, and swollen. Additionally, he has diffuse muscle pain, fatigue, and a history of alcohol addiction, heart failure, COPD, and being underweight. First, the presence of hemoptysis (coughing up blood) and chest pain raises concerns for pulmonary issues. Given his COPD, maybe a pulmonary embolism? But he also has heart failure, which could contribute to pulmonary congestion. However, the sudden onset of pain and fever might point towards an infection or embolism. The rash is on the commissure, which is the corner of the mouth, and it's pink, peeling, and swollen. That could be a fungal infection like oral thrush, but given his alcohol use and underweight status, maybe something else. Wait, the rash is on the commissure, and he has a fever, which could indicate an infectious process. The diffuse muscle pain and fatigue could be part of a systemic infection or a viral illness. However, the combination of hemoptysis, chest pain, fever, and chills suggests a more serious condition. Pulmonary embolism (PE) is a possibility, especially with the sudden onset of chest pain and shortness of breath. But PE usually doesn't present with hemoptysis unless it's large. Alternatively, pneumonia could explain the fever, chills, and cough, but the hemoptysis is more concerning. Another angle: tuberculosis? But TB is more chronic. The rash might be a sign of a fungal infection, but I'm not sure. His history of heart failure could lead to pulmonary edema, which might cause hemoptysis, but the pain and fever are less typical. Another thought: maybe a lung abscess or infection. The sudden onset and systemic symptoms (fever, chills) could point towards an infection. However, the rash is localized to the mouth, which might be a separate issue. So, the top differentials would be pulmonary embolism, pneumonia, and maybe tuberculosis. But given the sudden onset, PE is higher. However, the presence of hemoptysis and chest pain could also suggest a pulmonary issue like a ruptured lung cyst or a malignancy. But with the fever and chills, infection is more likely. Alternatively, maybe a viral infection with secondary bacterial pneumonia. Another possibility is acute bronchitis, but that's less likely with hemoptysis. The diffuse muscle pain and fatigue could be part of a systemic infection. So, the most likely diagnoses are pulmonary embolism, pneumonia, and tuberculosis. But I need to rank them. Given the sudden onset of chest pain, shortness of breath, fever, and hemoptysis, pulmonary embolism is a strong candidate. Pneumonia could also fit, especially with the fever and chills. Tuberculosis is less likely without a chronic cough or weight loss, but he is underweight. However, the rash might be a red herring. So the three main possibilities are pulmonary embolism, pneumonia, and tuberculosis. But I need to check if there's a better fit. Alternatively, acute myocardial infarction? But the pain is described as a knife stroke, which could be pleuritic, pointing more to a pulmonary issue. So I'l go with pulmonary embolism, pneumonia, and tuberculosis as the top three. </think> <answer1> Pulmonary Embolism </answer1> <confidence1> 0.45 </confidence1> <answer2> Pneumonia </answer2> <confidence2> 0.35 </confidence2> <answer3> Tuberculosis </answer3> <confidence3> 0.20 </confidence3> Reaching Beyond the Mode: RL for Distributional Reasoning in LMs RLVR Single RLVR SINGLE: <think> Okay, let's tackle this medical case. The patient is a 61-year-old male with several symptoms. He's coughing up blood, which is a red flag. He has chest pain described as a knife stroke, localized to the lower chest and posterior walls on both sides. The pain is severe (intensity 5) and doesn't radiate. He also reports shortness of breath, fever, chills, and a pink rash that peels on the left commissure. Additionally, he has widespread muscle pain, fatigue, and a cough worse at night. His medical history includes alcohol addiction, heart failure, COPD, and being underweight. First, coughing up blood (hemoptysis) along with chest pain and respiratory symptoms makes me think of pulmonary issues. The presence of fever, chills, and night sweats could indicate an infection. Given his COPD, he's already at risk for respiratory infections. But the combination of hemoptysis, fever, and chest pain might point towards something more serious like tuberculosis, especially since he's underweight and has a history of alcohol use, which can weaken the immune system. However, tuberculosis typically presents with chronic symptoms, though acute cases can occur. Another possibility is pneumonia, but hemoptysis is less common in typical pneumonia. Then there's the rash-pink, peeling on the commissure. That could be a sign of a systemic infection or an autoimmune condition. However, the systemic symptoms like fever, muscle pain, and fatigue align more with an infectious process. His heart failure could contribute to fluid retention and make him more susceptible to infections. The combination of hemoptysis, fever, and chest pain also makes me consider lung cancer, but that's more common in smokers and might present with different symptoms like weight loss and chronic cough. However, the acute onset of symptoms and the presence of fever might lean more towards an infection. Tuberculosis should be considered, especially given the chronic nature of some symptoms, systemic involvement, and the patient's risk factors. The rash might be a secondary manifestation. Alternatively, fungal infections could be a possibility, especially in someone with COPD and immunocompromised state. But tuberculosis is more likely given the symptoms. I should also check for other causes like bronchiectasis, which can cause hemoptysis and recurrent infections. But again, the fever and systemic symptoms point towards an active infection. Considering all factors, the most probable diagnosis is tuberculosis, given the combination of hemoptysis, fever, night sweats, weight loss, and the presence of a rash as a possible extrapulmonary manifestation. </think> <answer>Tuberculosis </answer> <confidence>95. </confidence> Reaching Beyond the Mode: RL for Distributional Reasoning in LMs RLCR Single RLCR SINGLE (prefer correct; otherwise idx=0): <think> Okay, let's tackle this medical case. The patient is a 61-year-old male with several symptoms and antecedents. First, the key symptoms include coughing up blood, chest pain described as a knife stroke, shortness of breath, fever, chills, a pink rash that peels, muscle pain, fatigue, cough, night symptoms, and pain when inhaling. The antecedents are alcohol addiction, heart failure, COPD, and being underweight. Coughing up blood (hemoptysis) is a red flag. Combined with chest pain and shortness of breath, this could point to a respiratory issue. The pain being a knife stroke suggests sharp, localized pain, which might indicate something like a pulmonary embolism (PE) or pleuritis. However, the presence of fever, chills, and a rash complicates things. The rash is pink, peels, and is on the commissure, which might be related to a systemic condition. The patient has a history of COPD and heart failure, which are risk factors for PE. Alcohol use and underweight status could contribute to immune suppression or clotting issues. The diffuse muscle pain and fatigue could be due to a systemic infection or a condition like sepsis. However, the combination of hemoptysis, chest pain, and respiratory symptoms might lean towards tuberculosis or lung cancer, especially in someone with a history of smoking (though not mentioned here). Wait, the antecedents don't mention smoking, but COPD could be related. I need to consider the most likely diagnosis. Given the symptoms, pulmonary embolism seems possible, especially with the history of COPD and heart failure. However, the rash and systemic symptoms might point towards something else like a systemic infection or even a drug reaction. Alternatively, tuberculosis could present with hemoptysis, fever, and night sweats, but the rash is less typical. Another angle: the patient's underweight status and alcohol use could lead to immunocompromise, making infections more likely. But the acute onset of symptoms (pain appearing quickly) might favor PE over a chronic condition like TB. I should also check if the symptoms align with other conditions. The diffuse muscle pain and fatigue could be part of a systemic illness. The combination of hemoptysis, fever, chills, and chest pain is very concerning for PE. However, without imaging or more specific tests, it's challenging. The answer might be pulmonary embolism, but I'm not entirely sure due to the presence of the rash and other symptoms. Could it be something else like pneumonia with a rash? Or maybe a severe infection like sepsis with multi-system involvement. But the key symptom of hemoptysis points towards a pulmonary cause. I think PE is the most likely, but there's uncertainty due to the rash and other symptoms. </think> <answer> pulmonary embolism </answer> <analysis> The diagnosis of pulmonary embolism (PE) is strongly suggested by hemoptysis, chest pain with a knife-like character, shortness of breath, fever, chills, and a history of COPD and heart failure, all of which increase PE risk. However, the presence of a pink, peeling rash on the commissure and diffuse muscle pain introduces ambiguity. These could indicate an alternative diagnosis such as a systemic infection, drug reaction, or tuberculosis. The patient's underweight status and alcohol use may contribute to immune compromise, complicating differential diagnosis. While PE remains the most likely explanation given the acute onset and respiratory symptoms, the rash and systemic manifestations introduce uncertainty. </analysis> <confidence> 0.75 </confidence>