Paper deep dive
PVminerLLM2: Improving Structured Extraction of Patient Voice via Preference Optimization
Samah Fodeh, Linhai Ma, Ganesh Puthiaraju, Srivani Talakokkul, Afshan Khan, Elyas Irankhah, Sreeraj Ramachandran, Ashley Hagaman, Sarah Lowe, Aimee Roundtree
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 6/20/2026, 7:45:48 AM
Summary
PVminerLLM2 is an improved set of Large Language Models designed for structured extraction of patient-generated text (the 'patient voice'). It addresses limitations in standard Supervised Fine-Tuning (SFT) and sequence-level preference optimization (like DPO) by introducing three key mechanisms: a token-weighted preference objective, a confidence-gated stabilization term to prevent absolute likelihood degradation, and confusion-aware preference pair construction. The model is evaluated on the PV-Miner benchmark and demonstrates significant improvements in Code, Sub-code, and Span extraction accuracy across various model sizes (Llama and Qwen) compared to SFT and existing DPO variants.
Entities (8)
Relation Signals (5)
Patient Voice → extractedby → PVminerLLM2
confidence 100% · PVminerLLM2, an improved set of LLMs for structured patient voice extraction
PVminerLLM2 → improves → PV-Miner
confidence 100% · PVminerLLM2, an improved set of LLMs for structured patient voice extraction... We evaluate PVminerLLM2 on the PV-Miner benchmark
PVminerLLM2 → outperforms → DPO
confidence 100% · PVminerLLM2 consistently outperforms... existing DPO-based preference optimization methods.
PVminerLLM2 → uses → Confidence-gated stabilization
confidence 100% · We introduce a confidence-gated stabilization mechanism.
PVminerLLM2 → uses → Confusion-aware preference pair construction
confidence 100% · Our method introduces... (ii) confusion-aware preference pair construction
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Motivation: Patient-generated text contains critical information on patients' lived experiences, social context, and care engagement, but remains largely unstructured, limiting its use in patient-centered outcomes research. Prior work introduced the PV-Miner benchmark and PVMinerLLM models for structured extraction. However, supervised fine-tuning (SFT) alone struggles with rare, fine-grained, and unevenly distributed errors, particularly in token-critical structured outputs. Results: We present PVminerLLM2, an improved set of LLMs for structured patient voice extraction that applies preference optimization to address token-critical errors beyond the reach of supervised fine-tuning. Our method introduces (i) a preference objective with token-level gated stabilization term that prevents degradation of absolute token likelihood under preference optimization, and (ii) confusion-aware preference pair construction to better capture low-separation distinctions. We further incorporate token-importance weighting and inverse-frequency reweighing to address token imbalance and class skew. Across multiple model sizes, PVMinerLLM2 consistently outperforms strong baselines, achieving gains of up to 4.43% (Code), 3.50% (Sub-code), and 1.55% (Span), and outperforms baseline LLM trained with existing preference optimization methods. Availability and Implementation: The supplementary material, code, evaluation scripts, and trained models for PVminerLLM2 are publicly available at: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2606.16074v1
- Canonical: https://arxiv.org/abs/2606.16074v1
Trouble viewing inline? Open PDF directly →
Full Text
36,123 characters extracted from source content.
Expand or collapse full text
Bioinformatics, YEAR, , 1–7 DOI added during production Published: Date added during production Original Paper PVminerLLM2: Improving Structured Extraction of Patient Voice via Preference Optimization Samah Jamal Fodeh , 1,2∗ Linhai Ma, 1 Ganesh Puthiaraju, 1 Srivani Talakokkul, 1 Afshan Khan, 1 Elyas Irankhah, 1 Sreeraj Ramachandran, 1 Ashley Hagaman, 3 Sarah Lowe 3 and Aimee Roundtree 4 1 Department of Emergency Medicine, Yale School of Medicine, 464 Congress Ave, 06519, CT, USA 2 Department of Biomedical Informatics & Data Science, Yale School of Medicine, 100 College Street, 06510, CT, USA 3 Department of Social and Behavioral Sciences, Yale School of Public Health, 60 College Street, 06520, CT, USA 4 Division of Research, Texas State University, 601 University Dr., 78666, TX, USA ∗ Corresponding author. samah.fodeh@yale.edu Abstract Motivation: Patient-generated text contains critical information on patients’ lived experiences, social context, and care engagement, but remains largely unstructured, limiting its use in patient-centered outcomes research. Prior work introduced the PV-Miner benchmark and PVMinerLLM models for structured extraction. However, supervised fine-tuning (SFT) alone struggles with rare, fine-grained, and unevenly distributed errors, particularly in token-critical structured outputs. Results: We present PVminerLLM2, an improved set of LLMs for structured patient voice extraction that applies preference optimization to address token-critical errors beyond the reach of supervised fine-tuning. Our method introduces (i) a preference objective with a token-level gated stabilization term that prevents degradation of absolute token likelihood under preference optimization, and (i) confusion-aware preference pair construction to better capture low-separation distinctions. We further incorporate token-importance weighting and inverse-frequency reweighting to address token imbalance and class skew. Across multiple model sizes, PVMinerLLM2 consistently outperforms strong baselines, achieving gains of up to 4.43% (Code), 3.50% (Sub-code), and 1.55% (Span), and outperforms baseline LLM trained with existing preference optimization methods. Availability and Implementation: The supplementary material, code, evaluation scripts, and trained models for PVminerLLM2 are publicly available at: https://github.com/Data-Mining-Lab-Yale/PVminerLLM2. Contact: samah.fodeh@yale.edu Keywords Large Language Models, Reinforcement Learning from Human Feedback, Preference Optimization, Medical Annotation, Patient-Generated Text PVminerLLM SFT only Preference optimization PVminerLLM2 SFT + preference optimization Token-weighted objective Upweight Code, Sub-code & Span tokens Confusion-aware pair construction Label confusion, deletion, insertion Confidence-gated stabilization Prevent token likelihood degradation Class-imbalance reweighting Inverse-frequency weights for rare Codes Figure1OverviewofthePVminerLLM2trainingpipeline. PVminerLLM (SFT only) is extended via preference optimization, which incorporates four mechanisms:token-weighted objective, confusion-aware pair construction, confidence-gated stabilization, and class-imbalance reweighting. 1. Introduction Patient-generated data, including secure messages, survey responses, and interview narratives, provide a direct view into patients’ lived experiences outside traditional clinical encounters (Howie et al., 2014; Huba and Zhang, 2012; Shapiro et al., 2012; Tiase et al., 2020). Unlike structured clinical records, these texts capture how individuals describe their needs, constraints, emotions, and expectations in their own words. These expressions collectively form the patient voice, reflecting not only clinical concerns but also broader social, environmental, and relational factors that influence health outcomes, such as social determinants of health and patient engagement in care (Amineh and Asl, 2015; Bakhtin, 1994; Gherlone, 2016; Fodeh et al., 2026c,f,b,e,d). Recent work (Fodeh et al., 2026a) introduced the PV- Miner benchmark and showed that supervised fine-tuning (SFT) substantially improves over prompting-based approaches. However, SFT alone is insufficient: even small token-level errors, such as an incorrect Code or Sub-code label or a slight Span boundary deviation, can invalidate an otherwise well-formed output and cause downstream errors in clinical applications. These errors are rare, fine-grained, and non-uniform in severity, making them difficult to eliminate through likelihood-based training alone. Received: Date Month Year. Revised received: Date Month Year. Accepted: Date Month Year © The Author(s) YEAR. Copyright and licence statements to be updated by the publisher during production. arXiv:2606.16074v1 [cs.CL] 15 Jun 2026 2Bioinformatics, YEAR, Volume X, Issue x Preference optimization provides a complementary signal by explicitly comparing preferred and dispreferred structured outputs for the same input,enabling models to resolve confusable label assignments,reduce spurious or missing extractions, and improve Span grounding. However, standard sequence-levelmethods(e.g.,(Rafailovetal.,2023; Gheshlaghi Azar et al., 2024; Meng et al., 2024; Xiao et al., 2024; Pal et al., 2024)) are insufficient here. In PV-Miner, preferred and dispreferred outputs often differ in only a few schema-defining tokens, so sequence-level objectives dilute the learning signal across largely identical outputs. Moreover, preference optimization can degrade the absolute likelihood of preferred outputs, harming structured prediction accuracy. Existing token-level methods (Zeng et al., 2024; Yang et al., 2025) partially address credit assignment but do not stabilize absolute likelihood or target the construction of low-separation pairs. To address these challenges, we present PVminerLLM2, an improved set of LLMs for structured patient voice extraction that extends SFT with token-level preference optimization. Our proposed method targets the three failure modes identified above:learning signal dilution,likelihood degradation of preferred outputs, and class imbalance. We propose novel mechanisms at both the objective and data-construction levels, including confidence-gated stabilization and confusion-aware pair construction. To our knowledge, PVminerLLM2 is the first to integrate these two mechanisms for structured information extraction. We evaluate PVminerLLM2 on the PV-Miner benchmark against PVminerLLM and multiple DPO-based preference optimization methods across four model sizes. Contributions: 1. We introduce a confidence-gated token-level stabilization mechanism that prevents degradation of absolute token likelihood during preference optimization. Our ablation identifies this as the single most impactful component. 2. We propose a confusion-aware preference pair construction strategy that generates low-separation pairs from empirical SFT errors. This enables precise learning on token-critical distinctions. 3. We incorporate token-importance weighting to concentrate learning on Code, Sub-code, and Span tokens. We further apply example-level inverse-frequency reweighting to address class imbalance. 4. We provide a comprehensive empirical evaluation showing that PVminerLLM2 consistently outperforms PVminerLLM and existing DPO-based baselines across model sizes and evaluation levels. 2. Related Work 2.1. DPO and its variants DPO (Rafailov et al., 2023) learns from preference pairs by encouraging a model to assign higher likelihood to preferred outputs than to dispreferred ones. Given an input x and a pair (y + ,y − ), DPO defines a preference score based on log-likelihood differences and optimizes: L DPO =− logσ β(s + − s − ) ,(1) where s + = logπ θ (y + |x) and s − = logπ θ (y − |x). Here, σ(·) denotes the sigmoid function, and β is a scaling factor controlling the sharpness of the preference. In practice, DPO often incorporates a reference model (e.g.,a supervised fine-tuned model) to stabilize training by comparing relative likelihoods rather than absolute values. Building on DPO, several variants have been proposed. SImPO (Meng et al., 2024) replaces the explicit reference model with an implicit reward;IPO/ΨPO (Azar et al., 2024) substitutes a margin-based regression objective; Cal- DPO (Xiao et al., 2024) adds a calibration mechanism; and DPO-Positive emphasizes the absolute likelihood of preferred outputs. However, all operate at the completion level and weight every token uniformly, leaving them without mechanisms to concentrate learning on the few schema-defining tokens that determine correctness in structured extraction.Our approach addresses this gap through token-level supervision and confidence-gated stabilization. See Table 1 for a summary comparison and the Extended DPO variant discussion in supplementary materials for an extended discussion. 2.2. Task and Data We evaluate our method on patient generate datasets introduced in (Fodeh et al., 2026a) called PV-Miner dataset, which consists of text data collected from multiple sources, including secure messages, surveys, and interview transcripts. The dataset contains 1,137 messages annotated with a hierarchical schema comprising 8 Codes and 33 Sub-codes, each paired with supporting text spans. In total, the dataset includes 46,038 words, with an average message length of 40.5 words (standard deviation 32.8), ranging from short clarifications to longer narrative descriptions. The data is inherently imbalanced, with a small number of high-frequency categories and a long tail of rare labels, reflecting real-world clinical communication patterns (shown in Fig. 2 ). We follow the original train/test splits. 0100200300400500600700 Count SharedDecision (Provider) CareCoordination (Provider) SharedDecision (Patient) CareCoordination (Patient) SocioEmotional Behaviour Partnership (Provider) SDOH Partnership (Patient) 0200400600 Count Acknowledge Patient Expertise ApprovalofDecision/Reinforcement Education Access & Quality Approval/Reinforcement Share Options Build Trust Requests for Opinion Explore Options Checking Understanding Make Decision State Preferences Neighborhood & Built Env. Maintain Communication Alignment Seeking Approval Connection Invite Collaboration Social & Community Context Appreciation/Gratitude Economic Stability Express Opinions Healthcare Access & Quality Sign-off Active Participation Salutation Clinical Care Figure 2 Distribution of Codes (left) and Sub-codes (right). The image is in high resolution, please zoom in for better vision. Bioinformatics, YEAR, Volume X, Issue x3 Table 1. Comparison of PVminerLLM2 preference optimization with existing methods. Prior approaches rely on completion-level signals, which can dilute learning under low edit-distance settings. PVminerLLM2 introduces token-level supervision and confidence- gated stabilization, focusing updates on semantically critical tokens (e.g., labels and spans). MethodSignal LevelToken ImportanceLow Edit Dist.Calibration / StabilityProtects Critical Tokens DPOCompletionNoNoYes (ref reg.)No SimPOCompletionNoNoYes (implicit reward)No IPO /ΨPOCompletionNoNoYes (margin regression)No Cal-DPOCompletionNoNoYes (calibration)No DPO-PositiveCompletionNoPartialYes (chosen likelihood)No PVminerLLM2 (Ours)Token + CompletionYes (value-aware)YesYes (confidence-gated)Yes (token-wise) Table 2. Example of patient voice extraction from the PV-Miner dataset. Each annotation is a (Code, Sub-code, Span) triple shown in JSON-style format. “None” means the Code has no Sub-codes. SourceMessage (Input)Extracted Triples (Output) YNHHDo I call them or do they reach out when they get the referral? ["code": "PartnershipPatient", "subcode": "activeParticipation", "span": "Do I call them", "code": "SharedDecisionPatient", "subcode": "ExploreOptions", "span": "or do they reach out", "code": "CareCoordinationPatient", "subcode": "None", "span": "they get the referral"] Patient voice extraction is a structured information extraction task over patient-generated text. Given an input message, the goal is to extract all patient-centered communication elements as structured tuples of the form (Code, Sub-code, Span). Each tuple consists of a high-level category (Code), a more specific subtype (Sub-code), and the exact text span supporting the annotation. The task is multi-label, allowing multiple tuples per message, and is subject to schema constraints, where only valid Code–Sub-code combinations are permitted. Please find the Codebook in the supplementary materials. Formally, given an input text x, the model predicts a set of tuples Y(x) =(c i ,s i ,t i ) N i=1 ,(2) where c i denotes a Code, s i is a Sub-code constrained by c i , and t i is an exact span copied from the input text. We follow the task formulation, dataset construction, and annotation schema introduced in (Fodeh et al., 2026a), and refer readers to that work for full details. In practice, model outputs are generated in a structured JSON format to ensure schema validity and enable automatic evaluation. All datasets are in English. The messages were collected from patient portals and survey platforms where English is the primary communication language. Table 2 presents one example. Examples from all the fours sources can be found in Examples from all four data sources in supplementary materials. 3. Method 3.1. System overview We aim to improve preference optimization for structured extraction, where small token-level errors can invalidate the entire output. Our method extends standard DPO in three aspects: (i) focusing learning on task-critical tokens, (i) stabilizing preferred outputs during training, and (i) addressing class imbalance. In the training data perspective, we propose confusion-aware preference training data generation to better capture realistic errors. 3.2. Token-weighted preference objective Following standard DPO, we optimize a preference objective basedonlikelihooddifferencesbetweenpreferredand dispreferred outputs. However, in structured extraction, most tokens (e.g., formatting tokens in JSON outputs) do not affect correctness. Errors are typically localized to a small subset of tokens, such as Code, Sub-code, and Span values. Treating all tokens equally therefore dilutes the learning signal. To address this, we redefine the sequence score as a token- weighted sum: s θ (y) = X t w t logπ θ (y t | x,y <t ),(3) where the weights depend on token roles: w t = w Code I[t∈C] + w Sub-code I[t∈S] + w Span I[t∈P], (4) where C, S , and P denote the sets of token positions that fall within Code, Sub-code, and Span values in the serialized output, respectively. Tokens outside all three sets (e.g., JSON scaffolding) receive w t = 0; tokens belonging to multiple fields receive correspondingly higher weight. Then, we compare the model against a reference model (the supervised fine-tuned (SFT) model). Let s ref (y) denote the same token-weighted score computed using this reference model. ∆ = s θ (y + )− s θ (y − ) − s ref (y + )− s ref (y − ) ,(5) During the training process, the following objective is optimized: L pref =− logσ(β∆).(6) This reference-adjusted objective ensures that learning focuses on improvements over the SFT baseline rather than inheriting its biases. 4Bioinformatics, YEAR, Volume X, Issue x 3.3. Confidence-gated likelihood stabilization Preference optimization may reduce the absolute likelihood of preferred outputs, which can degrade structured formats or span accuracy. To mitigate this, we introduce a confidence-gated stabilization mechanism. We first identify unstable tokens using: g t = ( 1if logπ θ (y + t | x,y + <t ) < γ 0otherwise (7) which activates only for tokens whose predicted probability falls below a confidence level. Here, γ controls the minimum confidence required for a token to be considered stable. We then apply a weighted penalty: L stab = P t g t w t − logπ θ (y + t | x,y + <t ) P t g t w t .(8) This term selectively lowers the preference optimization loss for low-confidence tokens in preferred outputs, preventing degradation while avoiding unnecessary constraints on already stable tokens. 3.4. Class-imbalance reweighting The dataset exhibits a long-tail distribution (Fig. 2), where rare Codes receive limited training signal. To address this, we adopt a class-balanced weighting strategy that increases the contribution of examples containing underrepresented categories. Specifically, let n κ denote the frequency of Code κ in the training data. For each training example i, we assign an inverse-frequency weighting based on the least frequent Code it contains: ω i = max κ∈C i 1 n κ ,(9) where C i denotes the set of Codes in the example i. This formulation ensures that examples involving rare Codes receive higher weights, while frequent ones are down-weighted. In practice, we use the effective-number formulation (Cui et al., 2019) as a smooth variant of this inverse-frequency weighting. The final training objective is defined as: L = P i ω i L (i) pref + λL (i) stab P i ω i ,(10) where L pref is the preference loss and L stab is the stabilization loss. 3.5. Preference Pair Construction Because many PV-Miner Codes and Sub-codes are semantically overlapping, effective preference pairs must be low-separation: preferred and dispreferred outputs differ in only a small number of schema-defining tokens. Confusion-aware synthetic pairs are derived from empirical SFT errors on a held-out validation set. For each gold annotation Y + , three dispreferred outputs Y − are generated by applying three perturbations, each schema- validity-preserving: (i) label confusion: a Code or Sub-code is replaced with a commonly confused alternative while the Span is preserved; (i) missing annotation: one tuple is deleted to simulate under-extraction; (i) extra annotation: a spurious tuple is inserted to simulate over-extraction. Because Y + and Y − share the same JSON scaffold and differ in only a few semantic tokens, these pairs directly instantiate the low-edit- distance regime motivating the token-weighted objective in Section 3.2. The Preference Pair Construction Details can be found in supplementary materials. 4. Experiments 4.1. Experimental Setting All benchmark experiments are conducted using lm-eval (Xie et al., 2024; Gao et al., 2024) with a vLLM backend for fast, reproducible generation. We evaluate instruction-tuned LLMs spanning 1.5B–70B parameters, including Llama-3.3- 70B-Instruct,Llama-3.1-8B-Instruct,Llama-3.2-3B-Instruct, and Qwen2.5-1.5B-Instruct. For all models, we apply each model’s native chat template and use deterministic decoding (temperature = 0, no sampling) to produce schema-constrained JSON outputs. The full prompt is provided in Prompt for Training the LLM in the supplementary materials. 4.1.1. Metric Code and Sub-code prediction is evaluated as multi-label classification. Let ˆy i and y i denote the predicted and gold code sets for instance i. Micro-averaged precision, recall, and F1 are: precision = P i |ˆy i ∩ y i | P i |ˆy i | (11) recall = P i |ˆy i ∩ y i | P i |y i | (12) F 1 = 2× precision× recall precision + recall (13) For Span extraction, a predicted Span is a true positive (TP) if it fully contains, or is fully contained by, a gold Span, or if their token-level Jaccard similarity exceeds 0.6. Unmatched predictions are false positives (FP); unmatched gold Spans are false negatives (FN). Span F1 is computed analogously to Code F1 above. 4.1.2. Hyperparameters Evaluation uses zero-shot decoding (temperature = 0, max 8096 context tokens, max 1024 generated tokens) with schema- valid JSON output. Training uses QLoRA (Dettmers et al., 2023) with bfloat16 mixed precision, 4-bit quantization, gradient checkpointing, and AdamW via HuggingFace Trainer on two H200 GPUs. Preference optimization is initialized from the publicly released PVminerLLM checkpoints (Fodeh et al., 2026a), which also serve as the frozen reference policy in Eq. 3. SFT training details are described in the original work (Fodeh et al., 2026a). For token weighting, we upweight semantic fields within the JSON completion with (w Code ,w Sub-code ,w Span ) = (2, 3, 1.5), additionally upweighting tokens that differ between chosen/rejected by a factor of 2, and normalizing weight mass so that the mean active weight is approximately 1.0. γ is 0.66 and β is 0.85. 3 epochs of training is conducted. Bioinformatics, YEAR, Volume X, Issue x5 Table 3. Extraction performance (in %) (F1, Precision, Recall, and F1 std). ∆F1 denotes improvement from SFT to PO. F1 is the mean of 3 runs with different random seeds PVminerLLMPVminerLLM2∆F1 ModelPRF1StdPRF1Std Code Llama-3.3-70B 87.90 80.11 83.82 0.62 87.26 81.74 84.41 0.17 +0.59 Llama-3.1-8B85.04 78.12 81.43 0.60 84.46 81.56 82.98 0.36 +1.55 Llama-3.2-3B82.48 78.30 80.33 0.78 82.70 80.66 81.67 0.19 +1.34 Qwen2.5-1.5B83.19 71.61 76.97 2.71 85.63 77.58 81.40 0.26 +4.43 Sub-code Llama-3.3-70B 83.74 77.95 80.74 2.40 84.13 80.34 82.19 0.35 +1.45 Llama-3.1-8B79.19 76.33 77.73 0.53 79.16 79.14 79.15 0.38 +1.42 Llama-3.2-3B75.80 73.74 74.75 1.75 77.54 75.03 76.27 0.30 +1.52 Qwen2.5-1.5B77.86 66.88 71.96 1.75 76.15 74.77 75.46 0.46 +3.50 Span Llama-3.3-70B 88.02 86.07 87.03 1.30 88.94 88.22 88.58 0.37 +1.55 Llama-3.1-8B87.29 86.37 86.83 2.74 86.16 89.40 87.75 0.59 +0.92 Llama-3.2-3B85.29 84.34 84.81 0.61 85.73 86.25 85.99 0.45 +1.18 Qwen2.5-1.5B83.98 85.64 84.80 0.57 84.66 86.44 85.54 0.49 +0.74 4.2. PVminerLLM vs PVminerLLM2 Table 3 reports extraction performance across four model sizes at Code, Sub-code, and Span levels. PVminerLLM2 outperforms PVminerLLM on all models and all evaluation levels, with gains ranging from +0.59% to +4.43% F1 on Code, +1.42% to +3.50% on Sub-code, and +0.74% to +1.55% on Span. The largest absolute gains occur on Qwen2.5-1.5B (+4.43% Code, +3.50% Sub-code), while the smallest occur on Llama-3.3-70B (+0.59% Code). Alongside F1 improvements, PVminerLLM2 consistently reduces run-to-run variance: Qwen2.5-1.5B’s Code F1 standard deviation drops from 2.71 to 0.26, and Llama-3.3- 70B’s Sub-code standard deviation from 2.40 to 0.35. 4.3. Comparison with Other DPO Variants We compare PVminerLLM trained with DPO, IPO, Cal-DPO, and DPO-Positive against PVminerLLM2 on Qwen2.5-1.5B- Instruct (Table 4). SimPO failed to converge under our training configuration and is therefore excluded. PVminerLLM2 achieves the highest F1 at all three evaluation levels. IPO yields the lowest performance (Code F1 54.48%), falling well below the SFT baseline. DPO (70.24%) and Cal-DPO (66.80%) also underperform SFT, indicating that reference regularization and calibration alone are insufficient in this low-edit-distance setting. DPO-Positive (72.43%) comes closest to PVminerLLM2 (81.40%) among the baselines but still falls short by 8.97% on Code, 6.39% on Sub-code, and 0.63% on Span. PVminerLLM2 also exhibits the lowest run-to-run variance on Code (0.26) and Sub-code (0.46). 4.4. Ablation Study Table 5 reports the contribution of each component on Qwen2.5- 1.5B-Instruct. The base configuration (no CB, no TW, no CG) achieves a mean F1 of 72.79%. Enabling CG alone Table 4. Performance comparison (F1 score, %) across methods on Qwen2.5-1.5B-Instruct. Std is over 3 runs with different random seeds. CodeSub-codeSpan MethodF1StdF1StdF1Std PVminerLLM (SFT)76.97 2.71 71.96 1.75 84.80 0.57 PVminerLLM + IPO54.48 1.74 48.27 5.17 68.01 1.65 PVminerLLM + DPO70.24 0.63 64.44 0.54 83.27 0.11 PVminerLLM + Cal-DPO 66.80 0.29 65.76 1.60 82.90 2.15 PVminerLLM + DPO-P72.43 0.52 69.07 0.38 84.91 0.30 PVminerLLM2 (Ours)81.40 0.26 75.46 0.46 85.54 0.49 Table 5. PVminerLLM2 ablation study on Qwen2.5-1.5B- Instruct. Results are F1 (%). Mean is the average of Code/Sub- code/Span (in %). CB: class balance weighting; TW: token weighting; CG: confidence-gated. T: true. F: false. CBTWCGCodeSub-codeSpanMean F70.2464.4483.2772.65 FFT77.8874.3584.1378.79 TFF72.4367.5284.1074.68 FTF74.9768.0183.0775.35 T81.4075.4685.5480.80 Table6.Code-levelperformancecomparisonbetween PVminerLLMandPVminerLLM2F1.∆F1denotes improvement. ClassFreqPVminerLLMPVminerLLM2∆F1 SharedDecisionPatient16331.1159.02+27.91 SocioEmotionalBehaviour24263.4679.66+16.20 CareCoordinationPatient18262.5072.22+9.72 SharedDecisionProvider10842.4245.83+3.41 PartnershipProvider37887.7290.61+2.89 PartnershipPatient67185.7188.41+2.70 SDOH53683.0585.34+2.29 CareCoordinationProvider16277.5072.73-4.77 yields the largest gain (+6.00%, to 78.79%), and achieves the highest score in every column, confirming that confidence- gated stabilization is the most critical mechanism. CB and TW contribute more modest improvements of +1.89% and +2.56% respectively, reflecting the benefit of token-importance concentration on the token-critical PV-Miner task. Because of the multi-label nature of the task, class-balance reweighting yields more modest gains for this long-tailed situation. (F, F, F) configuration is DPO with confusion-aware pairs but no TW, no CG, no CB and it falls below SFT, which further shows that the components are necessary. 4.5. Class-level analysis on Qwen2.5-1.5B Table 6 shows Code-level results on Qwen2.5-1.5B. The largest gains are on SharedDecisionPatient (+27.91%, from 31.11% to 59.02%) and SocioEmotionalBehaviour (+16.20%, from 63.46% to 79.66%). CareCoordinationPatient improves by +9.72%; remaining Codes show smaller but consistent gains. The one regression is CareCoordinationProvider (−4.77%). 6Bioinformatics, YEAR, Volume X, Issue x Table 7. Sub-code level performance comparison between PVminerLLMandPVminerLLM2F1.∆F1denotes improvement.Only Sub-codes with ∆ F1 > 10% and ∆F1 < 0 % are shown because the full list is long. The full table can be found in Full Sub-code performance table in supplementary materials. ClassFreqPVminerLLMPVminerLLM2∆F1 checkingUnder standing/clarification 6111.1138.71+27.60 Neighborhood AndBuiltEnvironment 9128.5748.65+20.08 EducationAccess AndQuality 1366.6785.71+19.04 requestsForOpinion3928.5747.06+18.49 ExploreOptions6114.2931.58+17.29 SeekingApproval10733.3350.00+16.67 connection12165.1277.55+12.43 SocialAnd CommunityContext 12970.5981.36+10.77 expressOpinions17758.6769.14+10.47 signoff40692.7491.02-1.72 statePreferences7460.8756.00-4.87 EconomicStability16083.3377.33-6.00 build trust3750.0033.33-16.67 Table 7 shows Sub-code results. The largest gains occur on low-frequency Sub-codes: checkingUnderstanding/clarification (+27.60%), NeighbourhoodAndBuiltEnvironment (+20.08%), EducationAccessAndQuality (+19.04%), and requestsForOpinion (+18.49%). Three Sub-codes regress: build trust (−16.67%), EconomicStability (−6.00%), and statePreferences (−4.87%). 5. Discussion 5.1. Interpreting the Results Sequence-level preference optimization methods spread the learning signal across hundreds of identical scaffolding tokens, providing almost no gradient on the schema-defining tokens that determine correctness. This explains why IPO collapses below SFT, and why DPO and Cal-DPO only partially recover through reference regularization and calibration:neither is token- selective. PVminerLLM2 succeeds by concentrating updates on Code, Sub-code, and Span tokens while the confidence-gated mechanism prevents their absolute likelihood from degrading under contrastive training. The ablation confirms that likelihood degradation, not insufficient learning signal, is the primary failure mode: without stabilization, even confusion-aware pairs with token weighting fall below SFT. Gains are larger on smaller models because they produce more frequent and diverse token- level errors after SFT, providing richer signal for confusion-aware pairs to exploit; larger models leave less room for correction. At the class level, the largest improvements occur on mid- to-low frequency Codes with high label confusion under SFT. The regressions on CareCoordinationProvider and build trust reflect a conservatism trade-off: confusion-aware pairs sharpen inter-label distinctions but can over-correct on closely related categories, as evidenced by build trust regressing while the semantically adjacent connection gains. 5.2. Clinical and Social Implications PVminerLLM2 extends PVminerLLM with more reliable extraction of precisely the categories most relevant to health equity. The improved performance on SharedDecisionPatient, SocioEmotionalBehaviour, and rare SDOH Sub-codes means that signals of decisional uncertainty, emotional distress, and social barriers can now be detected at scale with substantially higher reliability. Recovering these signals enables systematic population-level monitoring of patient voice, supporting earlier identification of at-risk groups and more targeted care coordination. That the largest gains occur on smaller models further suggests that reliable patient voice extraction is accessible to resource-constrained health systems where these signals are most needed. 5.3. Limitations and Future Work PVminerLLM2 demonstrates strong extraction capability but leaves room for improvement. The over-correction pattern identified in Section 5.1 shows that confusion-aware pair construction can degrade performance on high-overlap classes; future work will investigate asymmetric margin strategies to mitigate this. The current prompt design is also necessarily complex due to the hierarchical schema and disambiguation requirements, and future work will explore multi-agent inference frameworks where distinct agents handle label selection and Span verification separately, which may reduce prompt complexity while improving robustness (Srivastava; Parmar et al., 2025; Zhou et al., 2025). 6. Conclusion We present PVminerLLM2, an improved set of LLMs for structured patient voice extraction that applies preference optimization to address the token-critical,low-separation failure modes that SFT alone cannot resolve. PVminerLLM2 incorporates confusion-aware preference construction with confidence-gated token-level stabilization,and consistently improves over PVminerLLM and existing DPO-based baselines across model sizes, with especially strong gains on rare Sub- codes and reduced run-to-run variance. Together, the PV-Miner benchmark and PVminerLLM2 provide a practical testbed for studying preference optimization in real-world structured clinical NLP. 7. Conflicts of interest The authors declare that they have no competing interests. 8. Author contributions statement S.F. conceptualized the study, designed the methodology and data analysis plan, and contributed to the manuscript revision. L.M. designed the methodology, conducted the experiments, and prepared the manuscript. G.P. , A.K. , E.I. , S.R. assisted with the experiments, data processing, manuscript revision. S.T. and L.M. performed data annotation and quality assurance. S.F. , A.H. , S.L. , A.R. and S.T. provided domain expertise in patient-provider communication and contributed to Bioinformatics, YEAR, Volume X, Issue x7 the development and revision of the annotation framework. All authors reviewed and approved the final manuscript. 9. Acknowledgments This work was supported by the Patient-Centered Outcomes Research Institute (PCORI) under Award No. ME-2023C2- 31367 (to S.F.). References R. J. Amineh and H. D. Asl. Review of constructivism and social constructivism. Journal of Social Sciences, Literature and Languages, 1(1):9–16, 2015. M. G. Azar, Z. D. Guo, B. Piot, et al. A general theoretical paradigm to understand learning from human preferences. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238, pages 4447–4455. PMLR, 2024.URL https://proceedings.mlr.press/ v238/gheshlaghi-azar24a.html. M. M. Bakhtin.The Bakhtin Reader: Selected Writings of Bakhtin, Medvedev, and Voloshinov. 1994. Y. Cui, M. Jia, T.-Y. Lin, et al. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9268–9277, 2019. T. Dettmers, A. Pagnoni, A. Holtzman, et al. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023. URL https://arxiv.org/abs/2305.14314. S. Fodeh, L. Ma, G. Puthiaraju, et al. Pvminerllm: Structured extraction of patient voice from patient-generated text using large language models.arXiv preprint arXiv:2603.05776, 2026a. S. Fodeh,L. Ma,G. Puthiaraju,et al.Tab-po: Preference optimization with a token-level adaptive barrier for token-critical structured generation.arXiv preprint arXiv:2603.00025, 2026b. S. Fodeh, L. Ma, Y. Wang, et al. Pvminer: A domain-specific tool to detect the patient voice in patient generated data. arXiv preprint arXiv:2602.21165, 2026c. S. Fodeh, G. Puthiaraju, E. Irankhah, et al. Star-dro: Stateful tsallis reweighting for group-robust structured prediction. arXiv preprint arXiv:2604.09737, 2026d. S. Fodeh, S. Ramachandran, E. Irankhah, M. Arif, A. Khan, G. Puthiaraju, L. Ma, S. Talakokkul, J. Alpert, and S. Schellhorn. Eppc-oasis: Ontology-aware adaptation and structured inference refinement for electronic patient-provider communication mining in secure messages.arXiv preprint arXiv:2605.24172, 2026e. S. J. Fodeh, Y. Wang, L. Ma, et al.Eppcminerben: A novel benchmark for evaluating large language models on electronic patient-provider communication via the patient portal. Artificial Intelligence in Medicine, page 103429, 2026f. L. Gao, J. Tow, B. Abbasi, et al. The language model evaluation harness, 07 2024.URL https://zenodo.org/records/ 12608602. L. Gherlone. Vygotsky, bakhtin, lotman: Towards a theory of communication in the horizon of the other. Semiotica, 2016 (213):75–90, 2016. M. Gheshlaghi Azar, Z. Daniel Guo, B. Piot, et al.A general theoretical paradigm to understand learning from human preferences. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 4447–4455. PMLR, 2024. URL https://proceedings.mlr. press/v238/gheshlaghi-azar24a.html. L. Howie, B. Hirsch, T. Locklear, et al. Assessing the value of patient-generated data to comparative effectiveness research. Health Affairs, 33(7):1220–1228, 2014. N. Huba and Y. Zhang. Designing patient-centered personal health records (phrs): health care professionals’ perspective on patient-generated data. Journal of Medical Systems, 36 (6):3893–3905, 2012. Y. Meng, M. Xia, and D. Chen.Simpo: Simple preference optimization with a reference-free reward, 2024. A. Pal, D. Karkhanis, S. Dooley, et al. Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024. M. Parmar, X. Liu, P. Goyal, et al. Plangen: A multi-agent framework for generating planning and reasoning trajectories for complex problem solving. arXiv preprint arXiv:2502.16111, 2025. R. Rafailov, A. Sharma, E. Mitchell, et al. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2023. M. Shapiro, D. Johnston, J. Wald, et al. Patient-generated health data. Technical report, RTI International, 2012. M. Srivastava.Echo: A multi-agent ai system for patient- centered pharmacovigilance. In Open Conference of AI Agents for Science 2025. V. L. Tiase, W. Hull, M. M. McFarland, et al. Patient-generated health data and electronic health record integration: a scoping review. JAMIA Open, 2020. T. Xiao, Y. Yuan, H. Zhu, et al. Cal-dpo: Calibrated direct preference optimization for language model alignment, 2024. Accepted by NeurIPS 2024 Main. Q. Xie, W. Han, Z. Chen, et al. Finben: A holistic financial benchmark for large language models. Advances in Neural Information Processing Systems, 37:95716–95743, 2024. N. Yang, H. Lin, Y. Liu, et al. Token-importance guided direct preference optimization.arXiv preprint arXiv:2505.19653, 2025. Y. Zeng, G. Liu, W. Ma, et al. Token-level direct preference optimization. arXiv preprint arXiv:2404.11999, 2024. Y. Zhou, L. Song, and J. Shen.Mam: Modular multi- agent framework for multi-modal medical diagnosis via role- specialized collaboration.arXiv preprint arXiv:2506.19835, 2025.