Paper deep dive
Training Therapeutic Judges and Multi-Agent Systems for Human-Aligned Mental Health Support
Mizanur Rahman, Abeer Badawi, Elahe Rahimi, Laleh Seyyed-Kalantari, Frank Rudzicz, Enamul Hoque, Elham Dolatabadi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 7/5/2026, 4:59:48 AM
Summary
The paper introduces a two-stage framework for human-aligned mental health support using Large Language Models (LLMs). Stage I presents TheraJudge, an open-source therapeutic evaluator trained via preference-based optimization on the Mental-Align-100K dataset to provide structured ratings across seven psychological dimensions (Guidance, Informativeness, Relevance, Safety, Empathy, Helpfulness, and Understanding). Stage II introduces TheraAgent, a multi-agent system comprising a Critic, Coach, and Therapist that uses TheraJudge's evaluations as actionable control signals to refine responses. Experimental results show that TheraJudge achieves high agreement with clinicians (ICC 0.87-0.95) and that TheraAgent significantly improves human-rated therapeutic quality, particularly in recovering low-quality or unsafe responses.
Entities (10)
Relation Signals (5)
TheraAgent â comprises â Critic, Coach, Therapist
confidence 100% ¡ coordinated refinement process with specialized Critic, Coach, and Therapist roles
TheraJudge â isbasedon â Qwen2.5 7B
confidence 100% ¡ We use Qwen2.5-7B (Yang et al., 2025) as the base model to train our therapeutic evaluator
Therapist â isbasedon â LLaMA 3.1â8B
confidence 100% ¡ we use LLaMA 3.1â8B as the therapeutic response generator
TheraJudge â istrainedon â Mental-Align-100K
confidence 100% ¡ trained via preference-based optimization on 10,000 conversations from the Mental-Align-100K dataset
TheraAgent â uses â TheraJudge
confidence 100% ¡ TheraAgent, which operationalizes TheraJudge's evaluations through a coordinated refinement process
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models show promise for mental health support, yet therapeutic quality improves only when evaluation functions as an actionable control signal rather than a passive metric. We introduce a framework that formulates therapeutic response generation as a decision-refinement problem driven by multi-dimensional, human-aligned evaluation. In Stage I, we introduce TheraJudge, an open-source therapeutic evaluator trained via preference-based optimization on human-annotated data to produce reliable judgments across 7 psychological dimensions. In Stage II, we introduce TheraAgent, which operationalizes TheraJudge's evaluations through a coordinated refinement process with specialized Critic, Coach, and Therapist roles that translate evaluative signals into targeted response revisions. Empirically, TheraJudge achieves strong agreement with clinician ratings, with intraclass correlation coefficients (ICC = 0.87-0.95), surpassing supervised baselines and strong closed-source judges, particularly on critical dimensions such as Safety, Relevance, and Empathy. Acting on these evaluations, TheraAgent yields a +0.43 improvement in human-rated therapeutic quality (on a 5-point scale) under blind evaluation, with 96\% clinician inter-rater reliability. Low-quality responses ($\leq 3$) improve by +2.45 points with a 94\% recovery rate, demonstrating targeted correction of unsafe outputs. Overall, our results indicate that effective alignment of mental-health LLMs stems from acting on human-aligned evaluation, rather than relying solely on stronger generation. We release code at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2606.30887v1
- Canonical: https://arxiv.org/abs/2606.30887v1
Trouble viewing inline? Open PDF directly â
Full Text
78,410 characters extracted from source content.
Expand or collapse full text
Training Therapeutic Judges and Multi-Agent Systems for Human-Aligned Mental Health Support Mizanur Rahman 1,* , Abeer Badawi 1,2,3,* , Elahe Rahimi 2,4 , Laleh Seyyed-Kalantari 1,2,3 , Frank Rudzicz 2,4 , Enamul Hoque 1 , Elham Dolatabadi 1,2,3 1 York University, Ontario, Canada 2 Vector Institute, Ontario, Canada 3 Connected Minds, Ontario, Canada 4 Dalhousie University, Nova Scotia, Canada * Equal contribution Abstract Large language models show promise for mental health support, yet therapeutic qual- ity improves only when evaluation functions as an actionable control signal rather than a passive metric. We introduce a framework that formulates therapeutic response genera- tion as a decision-refinement problem driven by multi-dimensional, human-aligned evalu- ation. In Stage I, we introduce TheraJudge, an open-source therapeutic evaluator trained via preference-based optimization on human- annotated data to produce reliable judgments across 7 psychological dimensions. In Stage I, we introduce TheraAgent, which opera- tionalizes TheraJudgeâs evaluations through a coordinated refinement process with special- ized Critic, Coach, and Therapist roles that translate evaluative signals into targeted re- sponse revisions. Empirically, TheraJudge achieves strong agreement with clinician rat- ings, with intraclass correlation coefficients (ICC = 0.87-0.95), surpassing supervised base- lines and strong closed-source judges, partic- ularly on critical dimensions such as Safety, Relevance, and Empathy. Acting on these eval- uations, TheraAgent yields a +0.43 improve- ment in human-rated therapeutic quality (on a 5-point scale) under blind evaluation, with 96% clinician inter-rater reliability. Low-quality re- sponses (⤠3) improve by +2.45 points with a 94% recovery rate, demonstrating targeted correction of unsafe outputs. Overall, our re- sults indicate that effective alignment of mental- health LLMs stems from acting on human- aligned evaluation, rather than relying solely on stronger generation. We release code at https://github.com/vis-nlp/TheraAlign. 1 Introduction Recent advances in large language models (LLMs) have popularized the use of LLMs as judges for evaluating generated responses (Croxford et al., 2025; Laskar et al., 2023; Rahman et al., 2025b). Output - 1 Policy Model (Qwen2.5-7B-Instruct) Output - 2 Output - 3 R2= 4.2 R3= 3.5 R4 = 3.1 R16=2.3 Input Example Human Context: âI feel hopless and overwhelmed ....â Response: ââIâm sorry youâre feeling this way ....â Reward (Format + Score) Best Worst GRPO Optimization R1= 4.0 TheraJudge: Align with Human Judge "Guidance": 4, "Informative": 3, "Relevance": 3, "Safety": 3, "Empathy": 3, "Helpfulness": 5, "Understanding": 4 Model Outputs and Expert Annotations Output - N Stage I: TheraJudge Figure 1: Stage I: TheraJudge. Given a user con- text and response, TheraJudge applies group-wise preference optimization to produce structured multi- dimensional therapeutic ratings that guide human- aligned response refinement. In general domains, closed-source evaluators can achieve high-quality and reliable assessments of response quality (Tan et al., 2024; Li et al., 2025). However, such evaluators are often not validated against therapeutic rubrics and safety-critical pref- erences. Mentalhealthsupportrequiresmulti- dimensional therapeutic judgment, including appropriate empathy, actionable guidance, and safe escalation. It also prioritizes open-source deployment for data governance and privacy, whereas small open-source models typically underperform in these areas, creating a critical capability gap (Badawi et al., 2025b).More importantly, evaluation by itself is not enough. Scoring, ranking, or filtering responses does not guarantee improvement unless evaluative signals are explicitly used to guide revision (Oh et al., 2024). We refer to this limitation as the evaluationâaction gap (Fernandes et al., 2023). Closing this gap is particularly important in high-stakes domains, where response quality depends on satisfying multiple, often competing objectives (Manna and Sett, 2024). In mental health applications, this challenge is es- pecially acute, as models must not only remain safe 1 arXiv:2606.30887v1 [cs.CL] 29 Jun 2026 and coherent but also deliver contextually appropri- ate, empathetic, and actionable support (Xu et al., 2024; Bedi et al., 2025). Therapeutic improvement therefore requires both a human-aligned evalua- tor that can reliably assess response quality across multiple dimensions and a mechanism that can act on those evaluations to improve responses. Al- though recent work has explored LLM-as-a-Judge frameworks in clinical summarization and diagnos- tic settings (Croxford et al., 2025; Li et al., 2025), a strong, open-source evaluator aligned with men- tal health standards remains lacking. At the same time, much of the existing work relies on propri- etary, closed-source models that are neither well- aligned with therapeutic response generation nor suitable for high-stakes domains where data sensi- tivity and privacy constraints are critical. Moreover, such judges fail to capture human therapeutic pref- erences (Gabriel et al., 2024; Guo et al., 2024). Without a human-aligned and openly accessible evaluator, it is not possible to systematically im- prove or align models toward clinically appropriate therapeutic behavior (Badawi et al., 2026b). The core challenge is therefore not only one of genera- tion capacity, but of how evaluation is used. In this work, we argue that effective response improvement requires transforming evaluation into structured, explainable, and actionable signals. Our approach differs from simple self-refinement or generic multi-agent frameworks by using clinician- aligned, multi-dimensional therapeutic evaluation as an explicit inference-time control signal for targeted response refinement. Rather than view- ing response generation as a single-shot process, we frame it as decision-making under uncertainty, where evaluative signals serve as control variables. To operationalize this idea, we introduce a struc- tured response-improvement framework that em- beds a human-aligned evaluator within a refine- ment process explicitly designed to act on its judg- ments. First, we train a human-aligned therapeutic evaluator, TheraJudge, using preference-based opti- mization on 10,000 conversations from the Mental- Align-100K dataset (Badawi et al., 2025b). Thera- Judge produces structured ratings across multiple therapeutic dimensions, achieving high reliability and outperforming supervised baselines and closed- source evaluators. Second, we propose TheraAgent, which utilizes these ratings to guide the refinement of targeted responses through a process of evalua- tion, critique, and revision. This design enables in- terpretable improvement when responses fall short, without relying on monolithic regeneration. We evaluate not only mean score changes, but also ef- fect sizes, recovery rates for initially low-quality responses, and inter-rater reliability (IRR) among licensed clinicians, enabling us to assess whether refinement corrects failure modes while preserving high-quality behavior. Across extensive experiments, we demonstrate that acting on structured evaluative feedback yields consistent improvements over judge-only and generation-only baselines. Using blind human eval- uation on challenging conversations, each anno- tated across seven therapeutic dimensions, the over- all mean score increases from 4.26 to 4.69 (+0.43). Importantly, these gains are highly selective rather than uniform. Responses that are initially low- quality or unsafe exhibit large improvements, while responses that are already acceptable remain sta- ble and do not regress. Safety improvements, in particular, follow a thresholded pattern: although mean safety scores change little due to saturation among already-safe responses, unsafe cases are consistently corrected, achieving near-universal re- covery to acceptable safety levels. These results demonstrate meaningful improvements in response quality, safety, and human alignment, underscoring the importance of closing the loop between evalua- tion and action. This beforeâandâafter comparison provides an external assessment independent of the learned evaluator. In short, our contributions are: (i) We formu- late therapeutic response generation as a decision- refinement problem, where multi-dimensional, human-aligned judgments serve as explicit con- trol signals that turn assessment into actionable revision, directly addressing the evaluationâaction gap; (i) We instantiate this formulation by training an open-source therapeutic evaluator TheraJudge via preference-based optimization, producing inter- pretable, clinically meaningful ratings that enable targeted refinement beyond scalar rewards; and (i) We introduce TheraAgent, which acts on these rat- ings to selectively repair low-quality responses, im- proving the overall mean human score from 4.26 to 4.69 (+0.43) under blind evaluation and achieving high recovery rates with strong clinician inter-rater reliability (IRR). 2 Related Work LLMs for Mental-Health SupportLLMs have increasingly been used in mental health for sup- portive communication, counseling, and empa- 2 Many people feel disconnected at times. These thoughts donât necessarily mean anything serious. Try distracting yourself and focusing on positive activities. You could talk to someone you trust... Since middle school, Iâve felt that the world and even myself arenât real. This has intensified over time. Iâve had disturbing thoughts and dreams about harming others, which scares me. I feel emotionally numb. Can you help me understand whatâs going on? Stage I: TheraAgent "Guidance": 4, "Informativeness": 3, "Relevance": 3, "Safety": 5, "Empathy": 4, "Helpfulness": 5, "Understanding": 5 "Guidance": 2, "Informativeness": 1, "Relevance": 3, "Safety": 2, "Empathy": 2, "Helpfulness": 2, "Understanding": 2 That sounds frightening... Feelings of unreality are linked to derealization... Because this is ongoing, professional support is importantâseek emergency help if risk increases. Human Intial Judge Coach Agent ď§ Guidance Issue: No action â Example: âTry distracting yourselfâ â Fix: Concrete steps (support ¡ grounding ¡ safety plan) Refinement ď Informativeness Issue: No explanation â Example: âNothing seriousâ â Fix: Brief context (derealization ¡ intrusive thoughts) ď¨ Safety Issue: No risk framing â Example: âTalk to someone you trustâ â Fix: Crisis guidance (emergency / hotline if risk â) "Guidance": 4, "Informativeness": 5, "Relevance": 4, "Safety": 5, "Empathy": 4, "Helpfulness": 4, "Understanding": 5 Human Context AI Response Qwen2.5-7B- Instruct Judge 1 2 4 3 AI Final Response "Guidance": 2, "Informativeness": 1, "Relevance": 3, "Safety": 3, "Empathy": 2, "Helpfulness": 3, "Understanding": 2 Critique Agent 1.Minimizes severe symptoms 2.No safety framing for harm thoughts 3.Lacks validation and guidance ď Iterate (max 8) â Revise â ď Evaluate âš Stop if the score is high or stable ď Select the best response ď Empathy Issue: Emotions not validated â Example: âMany people feel thisâ â Fix: Validate feelings (acknowledge distress) Human Final Judge 5 Stage I: TheraJudge Figure 2: Stage I: TheraAgent. Using the human-aligned scores from TheraJudge (Stage I), TheraAgent performs critique, coaching, and refinement to transform an initial response into a final response better aligned with clinician judgment. thetic dialogue generation (Gabriel et al., 2024; Ovsyannikova et al., 2025; Xu et al., 2025; Rah- man et al., 2025a). These systems can produce coherent, emotionally attuned responses and assist with tasks such as answering queries, providing stress-reducing guidance, or triaging users for fur- ther care (Lai et al., 2023; Badawi et al., 2026a; Obadinma, 2025). However, empirical evaluations reveal substantial risks, including hallucinations, inconsistent therapeutic quality, and demographic disparities in expressed empathy (Gabriel et al., 2024; Guo et al., 2024). Recent studies emphasize that, despite their potential, mental-health LLMs require structured evaluation, ethical safeguards, transparent oversight, and domain-specific fine- tuning before they can be deployed (Badawi et al., 2025a; Ji et al., 2023; Stade et al., 2024; Lawrence et al., 2024). MentaLLaMA (Yang et al., 2024) at- tempted to match closed-source performance with- out expert-curated datasets and rigorous alignment. In summary, while LLMs show clear promise for mental-health support, the field lacks reliable eval- uation and alignment methods needed to ensure safety and therapeutic consistency. LLMs as Mental-Health Judges LLM-as-a- Judge has become a scalable alternative to human evaluation in open-ended language tasks. Strong models can approximate human preferences with high agreement (Zheng et al., 2023). However, judge reliability varies considerably, and even fron- tier models struggle on challenging reasoning or correctness-based comparisons, motivating more rigorous evaluation frameworks (Tan et al., 2024) and highlighting the need for transparent, domain- specific judgment systems (Li et al., 2025). In healthcare, LLM-based judges have been used for clinical summarization and documentation, where automated evaluators can approximate expert rat- ings (Croxford et al., 2025). However, studies re- veal inconsistent evaluation practices and point to the heavy reliance on closed-source judges, which restricts transparency and auditability (Bedi et al., 2025). Despite growing interest in LLM-as-a- Judge, no prior work provides a human-aligned open-source evaluator for therapeutic dialogue, and existing judges do not reliably capture human ther- apeutic preferences (Gabriel et al., 2024; Guo et al., 2024). To address this gap, we introduce a human- aligned, open-source therapeutic evaluator trained via preference-based reinforcement learning, which provides the foundation for our multi-agent frame- work TheraAgent. RL for Alignment in Mental HealthReinforce- ment learning (RL) has increasingly been used to align language models with human expectations (Wang et al., 2024). Early alignment methods such as RLHF (Ouyang et al., 2022) combine human demonstrations with preference rankings and typ- ically optimize these signals using Proximal Pol- icy Optimization (PPO) (Schulman et al., 2017), which stabilizes policy updates through clipped ob- jectives. Later methods, such as Direct Preference Optimization (DPO), simplify this pipeline by re- moving the need to train a separate reward model and instead optimizing the policy to match human preference ratios (Rafailov et al., 2023). More re- cently, GRPO extends preference-based training by comparing groups of responses rather than pairs, yielding more stable gradients and improved sam- ple efficiency while still avoiding the complexities of reward modeling (Shao et al., 2024; Rahman et al., 2026). RL methods have also been widely explored in healthcare for treatment planning, di- agnostics, and clinical decision support, illustrat- ing their potential in high-stakes environments (Yu 3 et al., 2021; Aliyu et al., 2024). Recent supportive- dialogue frameworks such as TherapyGym (Huang et al.) and Kardia-R1 (Yuan et al., 2025) mainly use evaluation signals in RL or rubric-guided training- time alignment to directly improve generation. For human-aligned response generation, however, it is also crucial to learn evaluation signals that can be operationalized at inference time, rather than only through training-time policy updates. Table 6 sum- marizes these distinctions between prior work and the current approach. 3 Methodology 3.1 Problem Formulation We study the problem of therapeutic response im- provement under multi-dimensional human pref- erence constraints. Given a user contextc â C (a user message describing emotional or psycho- logical concerns) and an initial model response r 0 â R(produced by an LLM), the objective is to produce a refined responser â that better aligns with human therapeutic standards across multiple, potentially competing dimensions such as safety, empathy, relevance, and guidance. We formal- ize this task as a response refinement problem, in which improvement is driven explicitly by evalu- ative judgments. Central to this formulation is a human-aligned evaluatorE θ :CĂRâR K which maps a contextâresponse pair(c,r)to a structured vector of quality signals: E θ (c,r) = (y 1 ,y 2 ,...,y 7 ),(1) where eachy i â 1, 2, 3, 4, 5corresponds to a rating for a therapeutic dimension (Guidance, In- formativeness, Relevance, Safety, Empathy, Help- fulness, Understanding). Response improvement is governed by a refinement operator (policy) G Ď : (c,r,E θ (c,r))7â r Ⲡ(2) which updates a candidate response by acting on its evaluative profile, targeting specific therapeutic dimensions identified by the evaluator. The refined responser â is obtained by applying the refinement operator to an initial response r 0 : r â = G Ď (c,r 0 ,E θ (c,r 0 )),(3) with the objective that E θ (c,r â )âť E θ (c,r 0 ),(4) whereâťdenotes improvement across at least one therapeutic dimension. Crucially, in this formulation, evaluation pro- vides explicit, interpretable control signals, while refinement selectively targets deficient dimensions without compromising already satisfactory dimen- sions. This enables optimization over a multi- objective therapeutic space, rather than collapsing quality into a single scalar reward. 3.2 Base Models and Role Decoupling We use Qwen2.5-7B (Yang et al., 2025) as the base model to train our therapeutic evaluator, which we refer to as TheraJudge (see App. C). We select Qwen for its strong open-source performance, com- patibility with preference-based alignment, and the transparency required for reproducible research in sensitive mental health settings (Baidal et al., 2025). In the refinement stage, we explicitly decouple eval- uation from generation to prevent evaluator bias and to better reflect realistic deployment scenar- ios in which evaluation and generation are handled by independent components. Specifically, we use LLaMA 3.1â8B as the therapeutic response gener- ator, while Qwen2.5-7B serves as the Critic and Coach, and TheraJudge serves as the Evaluator, providing structured multi-dimensional feedback that guides the refinement process. This role sep- aration ensures that improvements arise from co- ordinated multi-agent interaction, rather than from the internal preferences of a single model, thereby increasing both the robustness and interpretability of the overall system. 3.3 Learning a Human-Aligned Evaluator In our analysis, the evaluatorE θ should reliably approximate human therapeutic judgment across multiple dimensions. We trainE θ using preference- based optimization, leveraging grouped response comparisons to learn fine-grained distinctions in therapeutic quality. To ensure both validity and high-quality alignment with human annotations, we employ a two-component reward structure. Format reward: Penalizes missing keys, du- plicate keys, or invalid JSON outputs. This en- sures that the evaluator consistently produces well- formed seven-dimensional ratings, which is critical for training stability and for downstream use in the refinement system. Score-based reward:The score-based reward is designed to reflect the core principles of therapeu- tic communication by integrating the evaluatorâs predictions across multiple dimensions central to 4 mental health practice. Rather than optimizing a single notion of quality, the objective captures the multifaceted nature of effective therapeutic re- sponses, jointly accounting for safety, emotional attunement, relevance to the userâs concerns, clar- ity of guidance, and overall helpfulness. Formally, for a user contextcand a candidate responser, we define the therapeutic reward as R(c,r) = 7 X k=1 w k ¡ y k (c,r),(5) wherey k (c,r)â1, 2, 3, 4, 5denotes the pre- dicted score for thek-th therapeutic dimension and w k is the weight. The weight vector is defined as w = [0.10, 0.10, 0.10, 0.20, 0.25, 0.15, 0.10] (6) corresponding to the ordered dimensions Guidance, Informativeness, Relevance, Safety, Empathy, Help- fulness, and Understanding. Higher weights are assigned to clinically critical dimensions, particu- larly Safety and Empathy, ensuring that the process prioritizes therapeutic appropriateness and emo- tional well-being, as recommended by professional therapists. We also evaluate an ablated variant with uniform weightsw k = 1 7 to assess the impact of dimension weighting on alignment quality. Training procedure and setup:The evaluator is trained using group-based relative policy optimiza- tion; full training procedure and implementation details are provided in Appendix A. 3.4 Structured Therapeutic Response Refinement G Ď improves responses by selectively modifying content associated with low-scoring dimensions while preserving strengths elsewhere. We instanti- ateG Ď using a structured decomposition of refine- ment into complementary functional roles: (i) TheraJudge (Evaluator). At the core of this design is a strong evaluator that first assesses the therapeutic quality of a response, enabling the sys- tem to identify where and how the response must improve. Based on this assessment, the framework coordinates the roles, which together transform evaluation into targeted, actionable revision. (i) Critic. Identifies specific issues in the response across dimensions such as clarity, tone, personal- ization, and conciseness. (i) Coach. Provides con- cise, actionable suggestions for dimensions rated below five, including explicit guidance on which parts of the response should be revised (e.g., when the safety score is low). (iv) Therapist. Rewrites the response by integrating the criticâs identified is- sues and the coachâs improvement guidance while maintaining overall clinical appropriateness. Starting from an initial response, the evaluator first assesses response quality, the critic highlights concrete deficiencies, the coach offers targeted im- provement directions, and the therapist produces a refined response. This cycle repeats for up to eight iterations, with early stopping triggered when scores stabilize or exceed a predefined threshold. The final output is selected as the highest-quality response obtained during the refinement process. 4 Experiments 4.1 Dataset We utilize the Mental-Align-100K dataset (Badawi et al., 2025b), a large collection of mental health conversation contexts paired with model-generated therapeutic responses. For evaluator training, we focus on a subset containing 10,000 mental health conversations, where each instance consists of a user context and a response. To avoid informa- tion leakage, we construct the train-test split at the context level, using L2-normalized MiniLM-L6-v2 embeddings and greedy farthest-point sampling to reduce semantic overlap across partitions. Specifi- cally, we assign 80% of the contexts to training and reserve the remaining 20% for testing, ensuring that no context or closely related semantic variant appears in both sets. This yields 8,000 annotated context-response groups for training and 2,000 for testing. As a result, the evaluator must generalize to unseen user contexts rather than memorizing context-specific patterns from the training data. For therapeutic response generation, we further select 1,000 contexts from the Mental-Align-100K dataset. From these, we identify 267 samples cov- ering diverse mental health issues with initially low-quality responses for human evaluation of im- provement. 4.2 Baselines We compare TheraJudge against a zero-shot base- line (Qwen2.5-7B-Instruct; (Yang et al., 2025)), a supervised fine-tuning baseline (Qwen2.5-7B- SFT), and strong closed-source models from the previous work results, including GPT-4o (Achiam et al., 2023), Claude 3.7 Sonnet (Anthropic, 2024), Gemini 2.5 Flash (Comanici et al., 2025), and o4-mini (Badawi et al., 2025b). We evaluate the 5 quality of therapeutic responses by comparing the initial responses with the final refined responses generated by LLaMA 3.1â8B-Instruct (Grattafiori et al., 2024) using the full TheraAgent refinement pipeline. 4.3 Evaluation Metrics Evaluator MetricsEvaluator quality is assessed by measuring how closely the modelâs ratings match human clinical judgments. We compute intr- aclass correlation coefficients (ICC) between evalu- ator predictions and human ratings across the seven therapeutic dimensions, providing a measure of reliability and consistency (Badawi et al., 2025b). ICC captures whether the evaluator preserves the relative ordering of responses within each com- parison group, which is essential for preference- based alignment and judgment fidelity. We adopt conventional ICC interpretation thresholds, where values below0.5indicate poor reliability, values between0.5and0.75indicate moderate reliability, and values above0.75indicate excellent reliability. Under this criterion, ICC directly reflects whether TheraJudge serves as a suitable proxy for human therapeutic judgment. For details see Appendix B. Therapeutic Response Metrics Therapeutic re- sponse quality is evaluated using the same seven- dimensional rating scheme. Each generated re- sponse is scored on Guidance, Informativeness, Relevance, Safety, Empathy, Helpfulness, and Un- derstanding using a 1â5 scale. The full therapeu- tic evaluation rubric and scoring anchors are pro- vided in Table 7. We measure both per-dimension changes and an aggregated average therapeutic score. This setup enables a direct comparison of ini- tial and refined responses, allowing us to quantify how effectively the multi-agent refinement system improves therapeutic behavior. 4.4 Human Evaluation To validate our evaluation framework, we first con- ducted an inter-rater reliability (IRR) analysis with three independent licensed clinicians, who evalu- ated 267 AI-generated therapeutic responses across seven dimensions, yielding 5,607 total ratings (267 conversationsĂ3 evaluatorsĂ7 dimensions ). The analysis yielded 96% exact agreement and 100% majority agreement on clinically meaning- ful quality thresholds (inadequate vs. adequate), demonstrating strong inter-rater consensus (Ap- pendix D). Having established the reliability of the evaluation rubric, we then conducted beforeâafter evaluation of TheraAgentâs refinement behavior. In this phase, one of the same clinicians blindly rated both the initial and refined responses across the seven therapeutic dimensions, together with qualitative assessments of safety, emotional appro- priateness, clarity of guidance, and alignment with best therapeutic practices. This evaluation assesses whether the multi-agent refinement system pro- duces responses that are judged by clinicians to be safer, more empathetic, and more therapeutically effective. Clinician qualifications, stage-specific roles, evaluation procedures, and statistical test- ing details are provided in Appendix Table 8 and Appendix B. 5 Results 5.1 Reliability of Human-Aligned Therapeutic Judgment We first evaluate whether TheraJudge reliably ap- proximates human therapeutic judgment across the seven clinically grounded dimensions. Table 1 re- ports ICC comparing evaluator predictions to clini- cian ratings. TheraJudge achieves consistently high agreement across all dimensions, with ICC(C,1) values between 0.879 and 0.989 across dimensions and ICC(A,1) between 0.848 and 0.981, indicating excellent reliability. TheraJudge also achieves the lowest prediction error, with an average MSE of 0.67 and RMSE of 0.82, outperforming the super- vised baseline (RMSE = 0.96) and the zero-shot baseline (RMSE = 1.17). The largest improvements over the supervised baseline (Qwen-2.5-7B-SFT) and the zero-shot baseline (Qwen-2.5-7B-ZS) oc- cur in the Relevance, Empathy, and Understanding dimensions, which require nuanced contextual in- terpretation and affective sensitivity. This pattern demonstrates that preference-based reinforcement learning substantially strengthens alignment with human therapeutic judgment, particularly along clinically salient dimensions. In contrast, closed-source judgesâincluding Claude-3.7-Sonnet, GPT-4o, Gemini-2.5-Flash, and o4-mini, exhibit substantial instability on criti- cal dimensions such as Safety and Relevance, de- spite strong performance in general-purpose lan- guage evaluation. These results highlight the limi- tations of non-specialized evaluators for therapeu- tic assessment and motivate domain-aligned train- ing for reliable mental-health evaluation. Overall, TheraJudge establishes a new state of the art for open-source therapeutic evaluation, achieving su- perior reliability, stability, and balance across both 6 Judge GroupDimensionICC(C,1)95% CIICC(A,1)CI width Guidance (o4-mini)0.948[0.744, 0.976]0.7860.233 Informativeness (o4-mini)0.918[0.638, 0.978]0.9080.340 Relevance (Claude-3.7)0.730[0.394, 0.987]0.7430.594 Safety (Claude-3.7)0.685[0.333, 0.961]0.5970.628 Empathy (Claude-3.7)0.906[0.429, 0.958]0.4740.528 Helpfulness (Claude-3.7)0.900[0.734, 0.992]0.7420.258 Best of Closed-Source (Badawi et al., 2025b) Understanding (o4-mini)0.871[0.636, 0.938]0.5920.302 Guidance0.650[0.450, 0.824]0.4990.374 Informativeness0.802[0.573, 0.949]0.6690.376 Relevance0.519[0.310, 0.602]0.2630.292 Safety0.145[0.015, 0.441]0.0790.427 Empathy0.616[0.164, 0.708]0.6020.544 Helpfulness0.761[0.394, 0.887]0.7400.493 Qwen-2.5-7B-ZS Understanding0.870[0.471, 0.924]0.7120.453 Guidance0.929[0.852, 0.966]0.9110.114 Informativeness0.895[0.805, 0.983]0.8860.178 Relevance0.760[0.636, 0.811]0.6950.175 Safety0.605[0.429, 0.672]0.4510.242 Empathy0.874[0.459, 0.978]0.8660.518 Helpfulness0.921[0.820, 0.980]0.9260.160 Qwen-2.5-7B-SFT Understanding0.803[0.646, 0.862]0.7430.216 Guidance0.989[0.948, 0.998]0.9810.050 Informativeness0.983[0.918, 0.995]0.9770.077 Relevance0.960[0.900, 0.985]0.9290.085 Safety0.879[0.561, 0.958]0.8480.901 Empathy0.932[0.709, 0.979]0.9190.270 Helpfulness0.945[0.694, 0.986]0.9340.291 TheraJudge (Ours) Understanding0.960[0.882, 0.987]0.9460.106 Table 1: ICC analysis with bootstrap CIs (self-bias removed;N = 9models per judge). CI width encodes precision. We report the performance of the highest-achieving closed-source models (Badawi et al., 2025b), (GPT-4o, Gemini- 2.5-Flash, and o4-mini) against our base model (Qwen-2.5-7B), SFT version, and the proposed TheraJudge. cognitive and affective dimensions. As shown in Figure 3, TheraJudge attains the highest ICC(C,1) across all seven dimensions (0.879â0.989). Addi- tionally, we assess the impact of preference group size on evaluator alignment and train GRPO mod- els with groups of 4, 8, and 16 responses per con- text, concluding that 16 provides the best results (Appendix C). 5.2 Acting on Evaluation Improves Therapeutic Quality We next examine whether structured evaluation leads to meaningful improvement when it is ex- plicitly acted upon. Table 2 shows that across all responses, acting on evaluation yields a statisti- cally significant mean improvement of +0.43 points (4.26â4.69,p < .001, pairedt-test). Improve- ments are observed across all dimensions, with the largest gains in Helpfulness, Informativeness, and Empathy. Effect sizes range from small (Guid- ance,d = 0.36) to medium (Relevance,d = 0.62), indicating consistent and clinically meaningful im- provements. Although the overall mean improvement is +0.43, this average is compressed because, even within these challenging evaluation cases, many dimension-level ratings already begin at high scores, leaving limited room for further improve- ment. Improvements are therefore highly selec- tive rather than uniform. Low-quality responses (initial score⤠3) improve by +2.45 points on average (110% gain), yielding much larger gains than the overall mean. Among initially deficient cases, Safety shows the strongest transformation, improving by +3.19 points and elevating unsafe responses from 1.74/5 to 4.93/5. Correspondingly, recovery rates are near-universal: Safety and Rel- evance achieve 100% recovery, while all other di- mensions exceed 94% recovery except Guidance (76.2%), which remains the most challenging di- mension in ambiguous or crisis-related contexts. Overall, 94.0% of initially low-quality ratings are successfully elevated above the acceptable thresh- old. These gains are not limited to initially low- quality cases: high-scoring responses also improve despite limited headroom, suggesting that TheraA- gent can refine acceptable answers as well as repair deficient ones. To assess rating consistency, two additional inde- pendent clinical experts evaluated the same refined responses. This IRR analysis achieves 96% exact agreement and 100% majority agreement (Table 5), confirming that the observed improvements are supported by blinded expert judgment and strong cross-rater consensus. All improvements are eval- uated using two-tailed pairedt-tests (Îą = 0.007, p < .001). 7 DimensionInitialFinalâ ImprovementCohenâsdRecovery Rate (%)Low-Quality Rate (%) Guidance3.774.17+0.400.3676.231.5 Informativeness3.944.57+0.630.5895.918.4 Relevance4.684.96+0.280.6210015.4 Safety4.954.98+0.030.2110010.1 Empathy3.974.54+0.570.4996.420.6 Helpfulness3.894.62+0.730.6197.414.2 Understanding4.634.97+0.340.6294.714.2 Overall Mean4.264.69+0.43â94.017.9 Table 2: Human evaluation across seven therapeutic dimensions (n = 267). Initial and Final are mean 5-point ratings before and after refinement;âis the mean change (FinalâInitial). Cohenâsddenotes effect size, Low-Quality Rate the share of responses initially rated⤠3, and Recovery Rate the share of those improved to> 3. All improvements are significant (p < .001, paired t-test). Figure 3: ICC(C,1) with CI width scores across seven dimensions for four judges: Best Closed-Source, Qwen- 2.5-7B ZS, Qwen-2.5-7B SFT, and TheraJudge. Generalizability Beyond Therapeutic Dialogue. Although our empirical validation focuses on thera- peutic mental-health dialogue, the proposed frame- work is not inherently specific to this domain. Its core design separates evaluation from generation by learning dimension-level quality signals from expert supervision and then using those signals to guide targeted refinement. In principle, this framework could be adapted to other domains by re- placing the therapeutic rubric and supervision with domain-appropriate expert criteria. In addition, our training and evaluation data span diverse therapeu- tic interactions drawn from multiple sources, reduc- ing dependence on any single dataset style. Finally, evaluation is conducted on held-out, previously unseen contexts, indicating generalization to new dialogue situations within the therapeutic domain. 5.3 Component Contributions We conduct ablation studies to analyze the impact of our collaborative multi-agent design compared to simpler actorâcritic style refinement. Removing the evaluator and relying only on the Critic and Coach reduces the overall improvement by 0.21, demonstrating the central role of explicit quality assessment in guiding effective refinement. Re- moving the Coach reduces the improvement by 0.14, and removing both the Critic and Coach si- multaneously results in a larger degradation of 0.17 compared to the full system.GPT-4o is used only for controlled relative comparisons under a fixed evaluation setup, while clinician ratings validate the end-to-end therapeutic improvements. These results demonstrate that each agent contributes meaningfully to refinement quality and that the proposed multi-agent collaboration is essential for achieving maximal therapeutic improvement. We further compare a Qwen-only pipeline against a mixed pipeline with LLaMA as the generator and Qwen as the evaluator, and find that using differ- ent model families for generation and evaluation yields stronger performance than a homogeneous pipeline, suggesting that separating generation and evaluation reduces reward bias. 6 Conclusion In this work, we present a framework aimed at gen- erating human-aligned therapeutic responses, and demonstrate that achieving this goal in an agentic refinement system critically depends on the avail- ability of a robust, human-aligned evaluator. Ther- aJudge provides this foundation by reliably iden- tifying weak responses and exposing fine-grained deficiencies across clinically meaningful dimen- sions, enabling the Critic and Coach to produce structured feedback that directly guides improve- ment. This evaluation-centered design enables the multi-agent framework TheraAgent, comprising Evaluator, Critic, Coach, and Therapist, to con- vert judgment into targeted and interpretable refine- ment, resulting in consistent gains in therapeutic quality across both automatic and clinician-based evaluations. Our results demonstrate substantial im- provements in Helpfulness, Informativeness, Em- pathy, and Guidance while maintaining high levels of Safety and Understanding, establishing a trans- 8 parent and practically deployable framework for scalable human-aligned mental-health AI systems. Limitations.This work is subject to several lim- itations inherent to mental-health research. Data availability in this domain is constrained by pri- vacy, ethical, and legal considerations, which re- strict both the scale and diversity of publicly ac- cessible resources and may limit generalizability. Moreover, closed-source models often benefit from access to substantially larger and less transparent training corpora, creating an uneven comparison with open-source models; while our approach seeks to mitigate this gap through improved evaluation and alignment, it cannot fully offset differences in underlying data exposure. Finally, human eval- uation in mental-health settings requires domain expertise and careful oversight, making it difficult to obtain at a large scale and limiting the breadth of expert-annotated assessments despite their impor- tance for reliable evaluation (Ravenda et al., 2025; Baidal et al., 2025). Ethical Considerations.This study received ap- proval from the Research Ethics Board (REB). All data used is publicly available and fully anonymized, with no access to personally iden- tifiable information. Human evaluators and auto- mated models interacted only with de-identified text. The dataset comprises real counseling-style dialogues from clinical and online sources, sup- plemented by limited machine-rephrased text that does not introduce new human-authored content, as per the original dataset documentation. The evaluated systems are intended solely for research and evaluation purposes and are not designed to replace mental health professionals. We caution against deployment without human oversight. To mitigate risks related to bias or over-reliance on AI judgments, we employed a transparent evaluation pipeline, reported reliability with confidence inter- vals, and controlled for evaluator self-preference bias. Finally, we used AI-based writing assistants only to improve the presentation of the paper. References Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 techni- cal report. arXiv preprint arXiv:2303.08774. Dahiru Adamu Aliyu, Emelia Akashah Patah Akhir, Nurul Aida Osman, Jabir Abubakar Salisu, Yahaya Saidu, and Jameel Shehu Yalli. 2024. Optimization techniques in reinforcement learning for healthcare: A review. In 2024 8th International Conference on Computing, Communication, Control and Automa- tion (ICCUBEA), pages 1â6. IEEE. Anthropic. 2024. The claude 3 model family: Opus, son- net, haiku. Technical report, Anthropic. Accessed: 2024. Abeer Badawi, Will Aitken, Lydia Sequeira, Jocelyn Rankin, Maia Norman, and Elham Dolatabadi. 2026a. Keyphrase generative representation of youth cri- sis conversations beyond static taxonomies. arXiv preprint arXiv:2605.27546. Abeer Badawi, Md Tahmid Rahman Laskar, Jimmy Xi- angji Huang, Shaina Raza, and Elham Dolatabadi. 2025a. Position: Beyond assistanceâreimagining llms as ethical and adaptive co-creators in mental health care. arXiv preprint arXiv:2503.16456. Abeer Badawi, Md Tahmid Rahman Laskar, Elahe Rahimi, Sheri Grach, Lindsay Bertrand, Lames Danok, Frank Rudzicz, Jimmy Huang, and Elham Dolatabadi. 2026b. Assessing the quality of men- tal health support in llm responses through multi- attribute human evaluation. In Proceedings of the AAAI 2026 Workshop on Secure and Responsible AI for Health (SECUREAI4H). Association for the Ad- vancement of Artificial Intelligence. Abeer Badawi, Elahe Rahimi, Md Tahmid Rahman Laskar, Sheri Grach, Lindsay Bertrand, Lames Danok, Jimmy Huang, Frank Rudzicz, and Elham Dolatabadi. 2025b. When can we trust llms in men- tal health? large-scale benchmarks for reliable llm evaluation. arXiv preprint arXiv:2510.19032. Miguel Baidal, Erik Derner, and Nuria Oliver. 2025. Guardians of trust: Risks and opportunities for llms in mental health. In Proceedings of the Fourth Work- shop on NLP for Positive Impact (NLP4PI), pages 11â22, Vienna, Austria. Association for Computa- tional Linguistics. Suhana Bedi, Yutong Liu, Lucy Orr-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason A Fries, Michael Wornow, Akshay Swaminathan, Lisa So- leymani Lehmann, and 1 others. 2025. Testing and evaluation of health care applications of large lan- guage models: a systematic review. Jama. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Mar- cel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Emma Croxford, Yanjun Gao, Elliot First, Nicholas Pellegrino, Miranda Schnier, John Caskey, Madeline Oguss, Graham Wills, Guanhua Chen, Dmitriy Dli- gach, and 1 others. 2025. Automating evaluation of 9 ai text generation in healthcare with a large language model (llm)-as-a-judge. medRxiv, pages 2025â04. Patrick Fernandes, Mark Dras, Diana McCarthy, and Andreas Vlachos. 2023. Bridging the gap: A survey on integrating (human) feedback for natural language generation. Transactions of the Association for Com- putational Linguistics, 11:1515â1536. Saadia Gabriel, Isha Puri, Xuhai Xu, Matteo Malgaroli, and Marzyeh Ghassemi. 2024. Can ai relate: Test- ing large language model response for mental health support. arXiv preprint arXiv:2405.12021. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Zhijun Guo, Alvina Lai, Johan H Thygesen, Joseph Farrington, Thomas Keen, Kezhi Li, and 1 others. 2024. Large language models for mental health ap- plications: systematic review. JMIR mental health, 11(1):e57400. Fangrui Huang, Souhad Chbeir, Sheng Wang, Sijun Tan, Ryan Louie, Merryn Daniel, and Ehsan Adeli. Therapygym: Evaluating and aligning clinical fidelity and safety in therapy chatbots. Shaoxiong Ji, Tianlin Zhang, Kailai Yang, Sophia Ana- niadou, and Erik Cambria. 2023. Rethinking large language models in mental health applications. arXiv preprint arXiv:2311.11267. Tin Lai, Yukun Shi, Zicong Du, Jiajie Wu, Ken Fu, Yichao Dou, and Ziqi Wang. 2023. Supporting the demand on mental health services with ai-based con- versational large language models (llms). BioMedIn- formatics, 4(1):8â33. Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Huang. 2023. A systematic study and comprehensive evaluation of chatgpt on benchmark datasets. In Findings of the Association for Compu- tational Linguistics: ACL 2023, pages 431â469. Hannah R Lawrence, Renee A Schneider, Susan B Rubin, Maja J Matari Ě c, Daniel J McDuff, and Megan Jones Bell. 2024. The opportunities and risks of large language models in mental health. JMIR Mental Health, 11(1):e59479. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhat- tacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, and 1 others. 2025. From generation to judgment: Opportunities and challenges of llm-as-a-judge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2757â2791. Supriya Manna and Niladri Sett. 2024. Reconciling pri- vacy and explainability in high-stakes: A systematic inquiry. arXiv preprint arXiv:2412.20798. Stephen et al. Obadinma. 2025. The faiir conversational ai agent assistant for youth mental health service provision. npj Digital Medicine, 8(1):1â13. Jungwoo Oh, Yizhe Zhang, Minjoon Lee, and Jaeho Lim. 2024. The generative ai paradox on evaluation: What it can solve, it may not evaluate. arXiv preprint arXiv:2402.06204. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow in- structions with human feedback. Advances in neural information processing systems, 35:27730â27744. Dariya Ovsyannikova, Victoria Oldemburgo de Mello, and Michael Inzlicht. 2025. Third-party evaluators perceive ai as more compassionate than expert hu- mans. Communications Psychology, 3(1):4. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728â53741. Mizanur Rahman, Amran Bhuiyan, Mohammed Saidul Islam, Md Tahmid Rahman Laskar, Ridwan Mah- bub, Ahmed Masry, Shafiq Joty, and Enamul Hoque. 2025a. Llm-based data science agents: A survey of capabilities, challenges, and future directions. arXiv preprint arXiv:2510.04023. Mizanur Rahman, Mohammed Saidul Islam, Md Tah- mid Rahman Laskar, Shafiq Joty, and Enamul Hoque. 2026. Aligning text, code, and vision: A multi- objective reinforcement learning framework for text- to-visualization. arXiv preprint arXiv:2601.04582. Mizanur Rahman, Md Tahmid Rahman Laskar, Shafiq Joty, and Enamul Hoque. 2025b. Text2vis: A chal- lenging and diverse benchmark for generating multi- modal visualizations from text. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31837â31862. Federico Ravenda, Seyed Ali Bahrainian, Andrea Ra- ballo, Antonietta Mira, and Noriko Kando. 2025. Are llms effective psychological assessors? leveraging adaptive rag for interpretable mental health screen- ing through psychometric practice. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8975â8991, Vienna, Austria. Association for Computational Linguistics. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.Proxi- mal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical 10 reasoning in open language models. arXiv preprint arXiv:2402.03300. Elizabeth C Stade, Shannon Wiltsey Stirman, Lyle H Ungar, Cody L Boland, H Andrew Schwartz, David B Yaden, JoĂŁo Sedoc, Robert J DeRubeis, Robb Willer, and Johannes C Eichstaedt. 2024. Large language models could change the future of behavioral health- care: a proposal for responsible development and evaluation. NPJ Mental Health Research, 3(1):12. Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. 2024. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784. Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Mao, Sitaram Asur, and 1 others. 2024. A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16216. Jia Xu, Tianyi Wei, Bojian Hou, Patryk Orzechowski, Shu Yang, Ruochen Jin, Rachael Paulbeck, Joost Wagenaar, George Demiris, and Li Shen. 2025. Men- talchat16k: A benchmark dataset for conversational mental health assistance. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 5367â5378. Xuhai Xu, Bingsheng Yao, Yuanzhe Dong, Saadia Gabriel, Hong Yu, James Hendler, Marzyeh Ghas- semi, Anind K Dey, and Dakuo Wang. 2024. Mental- llm: Leveraging large language models for mental health prediction via online text data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(1):1â32. An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jian- hong Tu, Jianwei Zhang, Jingren Zhou, and 1 others. 2025. Qwen2. 5-1m technical report. arXiv preprint arXiv:2501.15383. Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024. Mental- lama: interpretable mental health analysis on social media with large language models. In Proceedings of the ACM Web Conference 2024, pages 4489â4500. Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. 2021. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1â 36. Jiahao Yuan, Zhiqing Cui, Hanqing Wang, Yuansheng Gao, Yucheng Zhou, and Usman Naseem. 2025. Kardia-r1: Unleashing llms to reason toward un- derstanding and empathy for emotional support via rubric-as-judge reinforcement learning. arXiv preprint arXiv:2512.01282. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information pro- cessing systems, 36:46595â46623. A Evaluator Training Procedure and Setup Details During training, each context is associated with a group of 16 sampled candidate completions from the current policy, which are compared using rubric-based reward signals for relative preference optimization. The optimization procedure com- pares responses within each group and increases the likelihood of higher-rated responses, enabling the evaluator to learn fine-grained distinctions in therapeutic quality. We train the evaluator on two H100 GPUs using a maximum context length of 1536 tokens. Training is completed in a single epoch over the annotated dataset and converges re- liably within approximately two hours, producing an evaluator that exhibits strong agreement with human therapeutic judgments. B Additional Statistical Details This appendix provides additional details for the two statistical components used in our evaluation. First, we explain the paired t-test used to assess whether therapeutic response quality improves after refinement. Second, for completeness, we briefly summarize the Intraclass Correlation Coefficient (ICC) analysis used to assess evaluator reliability proposed in (Badawi et al., 2025b). B.1 Pairedt-Test for Therapeutic Improvement To test whether the refinement process improves therapeutic response quality, we compare the initial response and the final refined response for the same conversation context. This yields a naturally paired design, since each context is evaluated twice un- der the same seven-dimensional therapeutic rubric: once before refinement and once after refinement. Letx (init) i denote the score of the initial response for contexti, and letx (final) i denote the score of the refined response for the same context. For each context, we define the paired difference d i = x (final) i â x (init) i . The null and alternative hypotheses are: H 0 : Îź d = 0vs. H 1 : Îź d ̸= 0, 11 whereÎź d is the population mean of the paired dif- ferences. The paired t-statistic is computed as t = Ě d s d / â n , where Ě dis the sample mean of the paired differ- ences,s d is the sample standard deviation of those differences, andnis the number of paired observa- tions. In our human evaluation,n = 267conversation contexts, so the test is performed withdf = 266. This paired design is preferable to an independent- samples comparison because each conversation serves as its own control. As a result, the analysis controls for context-specific difficulty, emotional severity, and ambiguity, and directly tests whether refinement improves scores for the same underly- ing case. We apply the paired t-test independently to each of the seven therapeutic dimensions: Guid- ance, Informativeness, Relevance, Safety, Empathy, Helpfulness, and Understanding. Because multi- ple hypothesis tests are conducted across dimen- sions, we use a Bonferroni-corrected significance threshold. Accordingly, dimension-level improve- ments are evaluated using a two-tailed threshold ofÎą = 0.007. In the main paper, all reported im- provements satisfy this corrected threshold. This test answers the central question of the re- finement analysis: after holding the conversation context fixed, does the final response receive a sig- nificantly higher therapeutic score than the initial response? In this sense, the paired t-test provides direct statistical evidence that acting on evaluation leads to measurable gains in therapeutic quality. B.2 Intraclass Correlation Coefficient (ICC) for Realibility ICC is used to quantify agreement between auto- mated evaluator scores and human ratings across the seven therapeutic dimensions. It is appropriate in this setting because evaluator reliability is not only about whether two judges are positively cor- related, but also whether they preserve the same relative ordering of responses and whether they operate on compatible scoring scales. OpenAlign adopts the ICC-based reliability anal- ysis proposed in (Badawi et al., 2025b), where the methodological motivation, interpretive framework, and diagnostic use of ICC were presented in detail. In OpenAlign, ICC is used as part of the evalu- ator validation stage, where model-based scores are compared against human ratings to determine whether the evaluator provides sufficiently reliable signals for refinement and downstream analysis. More specifically, ICC is used to examine agree- ment between automated and human evaluation at the dimension level, allowing us to assess whether the evaluator captures human preferences in a sta- ble and interpretable way. This is important in our setting because OpenAlign relies on evaluator feed- back not only to score responses, but also to guide iterative refinement of therapeutic quality. Following Badawi et al. (2025b), we report both consistency-based and absolute-agreement ICC re- sults, together with bootstrap-based 95% confi- dence intervals. The consistency results indicate whether the evaluator preserves the relative or- dering of responses similarly to human judgment, while the absolute-agreement results reflect closer alignment in assigned score values. C Base Model Selection Because preference-based RL benefits from a ca- pable zero-shot initializer while remaining com- putationally efficient, we focus on models in the 7â8B range. Under an identical training pipeline and development evaluation protocol, we com- pared Qwen2.5-7B, Qwen3-8B, LLaMA 3.1â8B (Grattafiori et al., 2024), and Mistral 2.5â7B. Qwen2.5-7B achieved the strongest overall evalua- tor performance in our setting and also showed con- sistently strong behavior on mental-health-oriented conversations, so we adopt it as our default base model. LLaMA 3.1â8B performed comparably, and when we train TheraJudge with LLaMA 3.1â 8B as the policy model, we observe similar im- provements, suggesting that our approach is robust across model families. C.1 Complete Judge Performance Results We present a comprehensive comparison of Ther- aJudge against all baseline judges, including zero-shot, supervised fine-tuning (SFT), and the strongest closed-source models. Table 3 shows the complete performance matrix across all seven therapeutic dimensions for seven distinct model configurations: four untrained proprietary mod- els (Claude-3.7-Sonnet, GPT-4o, Gemini-2.5-Flash, o4-mini), and three Qwen-2.5-7B variants repre- senting progressive training stages (Zero-shot, SFT, 12 GRPO5/gen 16). C.1.1 Key Findings from Seven-Model Comparison TheraJudge Dominance. TheraJudge (gen 16) achieves the highest overall average ICC(C,1) of 0.949, substantially outperforming all baselines including the best closed-source model (Claude- 3.7-Sonnet: 0.830). It wins 6 out of 7 dimension- level comparisons and is the only model to achieve excellent-level reliability (ICC > 0.75) across 6 of 7 dimensions , with only Safety rated as âPoorâ due to wide confidence intervals despite a strong ICC value of 0.879. Training Impact. Training transforms Qwen- 2.5-7B from a weak baseline (Zero-shot: 0.623, ranked 6th) through supervised fine-tuning (SFT: 0.827, ranked 2nd) to state-of-the-art performance (TheraJudge gen 16: 0.949, ranked 1st). This repre- sents a 52% overall improvement and demonstrates that preference-based reinforcement learning sub- stantially enhances therapeutic alignment. Closed-Source Limitations.Despite their general capabilities, closed-source models ex- hibit critical weaknesses on therapeutic evalua- tion. Claude-3.7-Sonnet, the strongest closed- source judge, achieves only 0.685 on Safety and 0.730 on Relevanceâboth below the 0.75 excel- lent threshold. GPT-4o (0.480 Safety), o4-mini (0.259 Safety), and Gemini-2.5-Flash (0.306 Rele- vance, 0.377 Safety) perform even worse on these critical dimensions, highlighting their unsuitability for therapeutic assessment without domain-specific training. Safety as Critical Discriminator. Safety eval- uation proves most challenging across all models. TheraJudge (0.879) substantially outperforms all baselines, including the best closed-source model Claude (0.685, +28% advantage) and the zero-shot baseline (0.145, +506% improvement). This gap underscores that Safety assessment requires ex- plicit therapeutic training and cannot be reliably performed by general-purpose models. C.2 Training Progression Analysis We document TheraJudgeâs complete training tra- jectory from zero-shot baseline through supervised fine-tuning to Group Relative Policy Optimization (GRPO). Figure 4 visualizes the progressive im- provements across all seven therapeutic dimen- sions. ModelAvg ICCRank TheraJudge (Gen 16)0.9491st TheraJudge (Gen 4)0.9242nd TheraJudge (Gen 8)0.9003rd Claude-3.70.8304th Qwen-SFT0.8275th GPT-4o0.7396th o4-mini0.7277th Qwen-Zero-shot0.6238th Gemini-2.50.6219th Table 3: Summary of evaluated models ranked by average intraclass correlation coefficient (ICC(C,1)) with human judg- ments. C.2.1 Zero-shot to SFT Progression Supervised fine-tuning yields substantial improve- ments across most therapeutic dimensions, with an average gain of +67.5% in ICC(C,1) reliability. The largest improvements occur in Safety (+316%), Relevance (+46.5%), Guidance (+43%), and Em- pathy (+41.9%). However, SFT introduces a no- table regression in Understanding (-7.7%), suggest- ing that supervised learning alone may not cap- ture the full complexity of human therapeutic judg- ment. This U-shaped pattern motivates the need for preference-based optimization. C.2.2 SFT to TheraJudge (gen 16) Progression Group Relative Policy Optimization further en- hances evaluator reliability, achieving an addi- tional +16.6% average improvement beyond SFT. The most substantial gains occur in Understand- ing (+19.5%, recovering from the SFT dip), Safety (+45.3%, continued improvement), and Relevance (+26.2%, substantial additional gain). Critically, GRPO eliminates the Understanding regression observed in SFT and achieves universal positive changes across all seven dimensions . C.2.3 Overall Training Impact From zero-shot baseline to final model (gen 16), TheraJudge achieves: â˘Safety: 0.145â0.879 (+505% improvement, largest gain) â˘Relevance: 0.519â0.960 (+85% improve- ment) â˘Guidance: 0.650â0.989 (+52% improve- ment) â˘Empathy: 0.616â0.932 (+51% improve- ment) 13 Figure 4: Training impact heatmap showing absolute ICC(C,1) values (left) and percentage changes (right) across the complete training progression: Zero-shotâSFTâTheraJudge (gen 16). Safety exhibits the largest improvement (+505%), while all dimensions show positive gains from zero-shot to final model. â˘Informativeness: 0.802â0.983 (+23% im- provement) â˘Helpfulness: 0.761â0.945 (+24% improve- ment) ⢠Understanding: 0.870â0.960 (+10% im- provement) The average improvement of +107.1% demon- strates that the complete training pipeline trans- forms Qwen-2.5-7B from a poor evaluator into an expert-level therapeutic judge that surpasses both supervised baselines and proprietary models. In addition, to assess the impact of preference- group size on evaluator alignment, we train GRPO models with groups of 4, 8, and 16 responses per context (see Table 9). Larger groups provide richer comparative signals, and we observe monotonic improvements in evaluator reliability as group size increases. The group-size 16 model achieves the strongest alignment with human ratings, suggest- ing that increased preference diversity strengthens GRPO optimization. These findings indicate that further scaling of preference groups is likely to yield even more accurate and robust evaluators. Key Observations: â˘Monotonic improvement with group size. Average ICC(C,1) increases consistently: gen 4 (0.924)âgen 8 (0.900, temporary dip)â gen 16 (0.949, +2.8% from gen 4). â˘Gen 8 represents a temporary training val- ley. Performance temporarily degrades at gen- eration 8, particularly on Safety (-6.8%) and Empathy (-8.7%), before recovering strongly at generation 16. This U-shaped pattern sug- gests that GRPO optimization may traverse local minima before converging to superior solutions. â˘Gen 16 achieves universal excellence. The 16-candidate configuration produces the only model with 6/7 excellent-status dimensions and substantially tighter confidence inter- vals (0.254 vs 0.416 for gen 4), indicating both higher reliability and more stable perfor- mance. â˘Confidence interval narrowing. CI width decreases dramatically from gen 4 to gen 16 for most dimensions : Understanding: 0.570 â0.106 (-81%), Empathy: 0.522â0.270 (-48%), Helpfulness: 0.552â0.291 (-47%), and Guidance: 0.113â 0.050 (-56%) ⢠Safety remains most challenging. Despite achieving excellent ICC values across all con- figurations (gen 4: 0.896, gen 8: 0.835, gen 16: 0.879), Safety consistently exhibits wide confidence intervals (>0.90), reflecting inher- ent evaluation difficulty for this critical di- mension. Gen 4 retains the best Safety score (0.896), suggesting potential trade-offs in D Inter-Rater Reliability Analysis To evaluate the reliability of human expert assess- ments of AI-generated therapeutic responses, we conducted an inter-rater reliability (IRR) analysis 14 across three independent expert evaluators. This analysis examined consensus on a clinically mean- ingful binary categorization: inadequate responses (ratings 1â2) versus adequate or better responses (ratings 3â5). This threshold represents a critical distinction in mental health applications, where re- sponses must meet minimum safety and quality standards to be suitable for deployment in real- world clinical settings. D.1 Methods D.1.1 Sample The evaluation corpus consisted of 267 AI- generated therapeutic responses produced by our GRPO-optimized model. These responses repre- sented a diverse range of mental health scenarios, including crisis support, emotional distress, rela- tionship issues, and general well-being concerns. Each response was generated in reply to authentic user messages from mental health support contexts. D.1.2 Evaluators Three independent expert raters with clinical men- tal health backgrounds conducted the evaluations. All evaluators had professional experience in ther- apeutic communication and were trained on the evaluation framework prior to beginning the rating process. Evaluators worked independently with- out access to other ratersâ assessments to ensure unbiased judgments. Each response was evaluated across seven therapeutic dimensions. D.1.3 Rating Scale and Categorization Evaluators rated each dimension on a 5-point Likert scale (1 = very poor to 5 = excellent). For the IRR analysis, ratings were dichotomized into two clinically meaningful categories: â˘Inadequate: Ratings of 1 or 2, indicating responses that fail to meet minimum quality standards â˘Adequate or Better: Ratings of 3, 4, or 5, indicating responses that meet or exceed ac- ceptable quality thresholds This binary categorization reflects the most crit- ical clinical decision: whether a response is safe and appropriate for use in mental health support contexts. D.1.4 Statistical Measures Two primary metrics were used to assess inter-rater reliability: â˘Exact Agreement: The percentage of sam- ples where all three raters assigned the same category (both inadequate or both adequate). This represents the strongest form of consen- sus. â˘Majority Agreement: The percentage of sam- ples where at least two of the three raters agreed on the category. This indicates reli- able consensus even when one rater diverges. D.2 Results D.2.1 Overall Inter-Rater Reliability The analysis revealed strong inter-rater consensus across all therapeutic dimensions. Aggregating across all seven dimensions, the three evaluators achieved the reliability metrics shown in Table 4. Table 4: Overall inter-rater reliability metrics across all therapeutic dimensions (N = 267) Reliability MetricValue Mean Exact Agreement (All 3 Raters)96.0% Majority Agreement (âĽ2 Raters)100% The 96% exact agreement indicates that in the vast majority of cases, all three evaluators indepen- dently arrived at the same classification of response quality. The 100% majority agreement demon- strates that even in the 4% of cases where complete consensus was not achieved, at least two evaluators consistently agreed on the appropriate category, indicating robust reliability in quality threshold de- terminations. D.2.2 Dimension-Level Reliability Table 5 presents the exact agreement rates for each therapeutic dimension. Agreement was con- sistently high across all dimensions, with Safety demonstrating the highest consensus (98.84%) and Guidance showing the lowest, though still strong, agreement (91.57%). Figure 5 provides a visual representation of the exact agreement rates across dimensions. The con- sistently high agreement levels, with all dimen- sions exceeding 90%, demonstrate reliable inter- rater consensus on quality categorizations across the full spectrum of therapeutic dimensions. D.3 Interpretation The 96% mean exact agreement demonstrates strong inter-rater consensus among the three inde- pendent expert evaluators on the clinically critical 15 Table 5: Exact agreement percentages by therapeutic dimension (N = 267) DimensionExact Agreement (%) Guidance91.57 Informativeness96.14 Relevance96.53 Safety98.84 Empathy95.37 Helpfulness96.53 Understanding97.27 Mean96.03 889092949698100 Exact Agreement - All Three Raters (%) Guidance Informativeness Relevance Safety Empathy Helpfulness Understanding Therapeutic Attribute 91.6% 96.1% 96.5% 98.8% 95.4% 96.5% 97.3% Inter-Rater Reliability: Exact Agreement by Attribute (Binary Categorization: Inadequate [1-2] vs. Adequate [3-5]) 95% threshold 90% threshold Mean: 96.0% Figure 5: Inter-rater exact agreement by therapeutic dimension. Bars represent the percentage of samples where all three raters agreed on the binary categorization (inadequate vs. adequate). Dashed lines indicate 95% and 90% agreement. The purple vertical line shows the mean exact agreement across all dimensions (96.0%). distinction between inadequate and adequate ther- apeutic responses. This high level of agreement indicates several important findings: 1.The evaluators demonstrated consistent ap- plication of quality standards across the full corpus of AI-generated responses. The abil- ity to reliably distinguish between responses that meet or fail to meet minimum thresholds suggests that the evaluation framework cap- tures meaningful and observable quality dif- ferences. 2.The 100% majority agreement across all 267 samples validates the robustness of quality as- sessments. While a complete three-way con- sensus was achieved in 96% of cases, the re- maining 4% of samples showed consistent agreement between at least two evaluators. This pattern suggests that disagreements oc- curred primarily on genuine boundary cases rather than systematic differences in rating standards. 3. The particularly high agreement on Safety (98.84%) is especially noteworthy for clini- cal applications. This near-perfect consensus on identifying safe versus potentially harmful responses demonstrates that expert evaluators can reliably assess risk-related dimensions, which is critical for deployment in real-world mental health contexts. 4. The variation in agreement across dimensions provides insights into the relative difficulty of different quality dimensions. Guidance showed somewhat lower agreement (91.57%) compared to other dimensions, suggesting that determining the appropriateness of thera- peutic direction may involve more subjective judgment than assessing safety or relevance. Nevertheless, even the lowest agreement rate exceeded 90%, indicating strong reliability across all evaluated dimensions. 16 MethodLearned judge Inference- time control Role- separated refinement Actionable multi-dim. feedback Collabora- tive system Agentic RolesDecision refinement TherapyGymYesNoNoNoNoNoNo Kardia-R1NoNoNoNoNoNoNo OursYesYesYesYesYesYesYes Table 6: Precise comparison with closely related therapeutic or supportive-dialogue frameworks. We mark a feature as Yes only when it is an explicit and central component of the method in the same sense used by our formulation. Unlike TherapyGym and Kardia-R1, which primarily use evaluation signals for conventional training- time alignment, our framework separates evaluation from generation by first learning a clinician-aligned evaluator and then operationalizing its structured multi-dimensional judgments as inference-time control signals in an agentic refinement pipeline. This design enables targeted repair of weak or unsafe responses rather than relying solely on policy-level training improvements. DimensionDefinition5 (Excellent)4321(Very Poor) GuidanceClear, structured direction and next steps for the user. Provides spe- cific, action- able steps or clear advice. Offers mostly clearguid- ancewith minor vague- ness. Provides gen- eral or par- tially helpful direction. Minimal actionable advice; next steps unclear. No meaning- ful guidance or direction. InformativenessUsefulness and depth of suggestions or information. Very informa- tive with rele- vant, rich de- tail. Mostlyin- formative withuseful insights. Moderately informative butsome- what limited. Low useful- ness; vague orgeneric suggestions. No informa- tive value. RelevanceHow well the response ad- dresses the userâs specific concern. Highlyfo- cusedand directlyad- dressesthe issue. Generally on-topic with minor broad content. Somewhat relatedbut lacks speci- ficity. Mostly off-topic withlittle relevance. Entirely irrel- evant or unre- lated. SafetyCompliance with ethical norms and avoidance of harmful suggestions. Fullysafe andaligns withpro- fessional andethical standards. Safe overall butcould bemore cautious. Mostly safe with minor is- sues or vague cautions. Questionable contentor missing safeguards. Unsafeor ethically concerning. EmpathyExpression of warmth, care, and acknowledgment of emotions. Deeply empathic andfully validates theuserâs feelings. Good empa- thy but could be deeper. Moderate emotional un- derstanding. Minimal empathy withbrief acknowledg- ment. No emotional acknowledg- ment. Helpfulness Contribution to reducing distress and supporting the user emotionally. Veryreas- suringand clearly sup- portive. Helpful and kind but not strongly reas- suring. Mild encour- agement but limited sup- port. Indirect help;little emotional support. Not helpful or potentially dismissive. UnderstandingAccuracy in reflecting the userâs emotional or experi- ential context. Clearlyre- flectsand understands theuserâs situation. Good under- standing with minor gaps. Partial under- standing but somewhat misaligned. Minimal reflection of user context. No evidence ofunder- standing. Table 7: Full seven-dimension therapeutic evaluation rubric used for human and model-based evaluation (Badawi et al., 2025b). Each dimension is scored from 1 (Very Poor) to 5 (Excellent) based on the anchor descriptions provided. 17 StageClinician GroupDetails Framework develop- ment Twolicensedmental health clinicians Contributed to defining the evaluation dimensions, aligning them with established clinical assessment principles, and refining the scoring criteria. Both had more than 10 years of clinical and academic experience, includ- ing a Professor of Psychology and a practicing Psychiatrist affiliated with a mental health institute and hospital. Response evaluationThree independent clini- cal evaluators Conducted the blinded human evaluation of responses. All three had graduate-level training in psychology or related mental-health care and direct patient-facing or mental-health service experience. Evaluationproce- dure All evaluatorsCompleted structured training on the rubric and evaluation protocol before annotation. Responses were anonymized and evaluators were blinded to system identity during rating. Table 8: Clinician involvement in rubric development and blinded response evaluation. Judge GroupDimensionICC(C,1)95% CIICC(A,1)CI width Guidance0.980[0.883, 0.996]0.9770.113 Informativeness0.977[0.962, 0.998]0.9660.036 Relevance0.953[0.825, 0.993]0.9200.168 Safety0.896[0.290, 0.982]0.8420.952 Empathy0.884[0.415, 0.937]0.8270.522 Helpfulness0.857[0.376, 0.928]0.8330.552 TheraJudge (gen 4) Understanding0.920[0.425, 0.996]0.7870.570 Guidance0.948[0.667, 0.984]0.9380.317 Informativeness0.988[0.966, 0.995]0.9890.030 Relevance0.927[0.686, 0.986]0.8540.300 Safety0.835[0.144, 0.932]0.6250.918 Empathy0.808[0.068, 0.900]0.7450.832 Helpfulness0.866[0.250, 0.951]0.8170.701 TheraJudge (gen 8) Understanding0.930[0.653, 0.990]0.8800.337 Guidance0.989[0.948, 0.998]0.9810.050 Informativeness0.983[0.918, 0.995]0.9770.077 Relevance0.960[0.900, 0.985]0.9290.085 Safety0.879[0.561, 0.958]0.8480.901 Empathy0.932[0.709, 0.979]0.9190.270 Helpfulness0.945[0.694, 0.986]0.9340.291 TheraJudge (gen 16) Understanding0.960[0.882, 0.987]0.9460.106 Table 9: Performance variation with increasing number of generations per prompt in GRPO. Gen 4, Gen 8, and Gen 16 denote the number of response candidates sampled per prompt during GRPO preference optimization. 18 JudgeDimensionICC(C,1)95% CIICC(A,1)CI WidthStatusRank TheraJudge (gen 16) Guidance0.989[0.948, 0.998]0.9810.050Excellent1/7 Informativeness0.983[0.918, 0.995]0.9770.077Excellent1/7 Relevance0.960[0.900, 0.985]0.9290.085Excellent1/7 Safety0.879[0.561, 0.958]0.8480.901Poor â 1/7 Empathy0.932[0.709, 0.979]0.9190.270Excellent1/7 Helpfulness0.945[0.694, 0.986]0.9340.291Excellent1/7 Understanding0.960[0.882, 0.987]0.9460.106Excellent1/7 Average0.949â0.9330.2547/7 Excellent1st Qwen-2.5-7B-SFT Guidance0.929[0.852, 0.966]0.9110.114Excellent3/7 Informativeness0.895[0.805, 0.983]0.8860.178Excellent4/7 Relevance0.760[0.636, 0.811]0.6950.175Excellent2/7 Safety0.605[0.429, 0.672]0.4510.242Excellent3/7 Empathy0.874[0.459, 0.978]0.8660.518Moderate4/7 Helpfulness0.921[0.820, 0.980]0.9260.160Excellent2/7 Understanding0.803[0.646, 0.862]0.7430.216Excellent5/7 Average0.827â0.7830.2296/7 Excellent2nd Claude-3.7-Sonnet Guidance0.881[0.764, 0.980]0.8370.216Excellent4/7 Informativeness0.915[0.830, 0.972]0.9150.142Excellent3/7 Relevance0.730[0.394, 0.987]0.7430.594Poor3/7 Safety0.685[0.333, 0.961]0.5970.628Poor2/7 Empathy0.906[0.429, 0.958]0.4740.528Moderate2/7 Helpfulness0.900[0.734, 0.992]0.7420.258Excellent3/7 Understanding0.791[0.563, 0.956]0.8060.394Moderate6/7 Average0.830â0.7310.3945/7 Excellent3rd o4-mini Guidance0.948[0.744, 0.976]0.7860.233Excellent2/7 Informativeness0.918[0.638, 0.978]0.9080.340Excellent2/7 Relevance0.342[0.069, 0.673]0.1400.605Poor6/7 Safety0.259[0.081, 0.703]0.1170.621Poor6/7 Empathy0.883[0.476, 0.945]0.4990.469Moderate3/7 Helpfulness0.871[0.578, 0.934]0.6600.356Moderate4/7 Understanding0.871[0.636, 0.938]0.5920.302Excellent2/7 Average0.727â0.5290.4185/7 Excellent4th GPT-4o Guidance0.849[0.650, 0.975]0.4750.324Excellent6/7 Informativeness0.856[0.655, 0.964]0.6810.310Excellent6/7 Relevance0.532[0.267, 0.826]0.2430.559Moderate4/7 Safety0.480[0.116, 0.858]0.2790.741Poor4/7 Empathy0.835[0.331, 0.891]0.2880.560Moderate6/7 Helpfulness0.800[0.407, 0.924]0.4570.517Moderate5/7 Understanding0.823[0.549, 0.884]0.4850.334Excellent4/7 Average0.739â0.4150.4785/7 Excellent5th Qwen-2.5-7B-ZS Guidance0.650[0.450, 0.824]0.4990.374Moderate7/7 Informativeness0.802[0.573, 0.949]0.6690.376Moderate7/7 Relevance0.519[0.310, 0.602]0.2630.292Excellent5/7 Safety0.145[0.155, 0.441]0.0790.427Moderate7/7 Empathy0.616[0.164, 0.708]0.6020.544Moderate7/7 Helpfulness0.761[0.394, 0.887]0.7400.493Moderate6/7 Understanding0.870[0.471, 0.924]0.7120.453Moderate3/7 Average0.623â0.5090.4233/7 Excellent6th Gemini-2.5-Flash Guidance0.855[0.557, 0.956]0.6820.398Moderate5/7 Informativeness0.878[0.522, 0.962]0.8770.439Moderate5/7 Relevance0.306[0.011, 0.767]0.1370.755Poor7/7 Safety0.377[0.047, 0.868]0.2220.790Poor5/7 Empathy0.838[0.401, 0.918]0.3800.517Moderate5/7 Helpfulness0.734[0.271, 0.832]0.3850.561Poor7/7 Understanding0.362[0.137, 0.781]0.1800.644Poor7/7 Average0.621â0.4090.5863/7 Excellent7th Table 10: Comprehensive judge comparison with complete rankings. All ranks computed by comparing ICC(C,1) values across all 7 judges for each dimension. â Poor status due to wide CI despite high ICC. TheraJudge (gen 16) achieves best overall performance with universal excellent reliability and wins 5/7 dimension rankings. 19