Paper deep dive
ESCRAG-R1: Retrieval-Augmented Reinforcement Learning for Emotional Support Conversation
Weichu Liu, Yuxuan Hu, Yirong Sun, Ningning Mao, Ziyun Zhang, Jian Chen, Mingyang Xu, Qishan Zhong, Chengming Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/25/2026, 7:41:10 AM
Summary
The paper introduces ESCRAG-R1, a retrieval-augmented reinforcement learning framework for Emotional Support Conversation (ESC) that integrates psychological guidance into Group Relative Policy Optimization (GRPO). It addresses the lack of structured, stage-aware reasoning in existing ESC systems by using a novel dataset, ESC-Preference, constructed via a Client-Counselor-Judge evaluation framework. Experiments show ESCRAG-R1 significantly outperforms baselines in empathy and therapeutic alignment.
Entities (8)
Relation Signals (6)
ESC-Preference → evaluatedby → Client-Counselor-Judge
confidence 95% · ESC-Preference, a high-quality dataset based on a Client--Counselor--Judge evaluation framework
ESCRAG-R1 → uses → GRPO
confidence 95% · ESCRAG-R1, a unified framework that integrates retrieval-based psychological guidance into Group Relative Policy Optimization (GRPO).
ESCRAG-R1 → uses → ESC-Preference
confidence 95% · To provide the reliable supervision required for this optimization, we construct ESC-Preference... To provide reliable reward supervision for this optimization, we construct a Client–Counselor–Judge evaluation framework and build ESC-Preference.
ESC-Preference → derivedfrom → ESConv
confidence 90% · Based on ESConv (19), we simulate multi-turn client–counselor interactions to construct counseling dialogues.
ESCRAG-R1 → outperforms → existing baselines
confidence 90% · Extensive experiments demonstrate that ESCRAG-R1 significantly outperforms existing baselines
Qwen-2.5-7B → usedas → reward_model_backbone
confidence 90% · For reward model, we adopt Qwen-2.5-7B (33; 26) as the backbone.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Emotional Support Conversation (ESC) systems aim to provide holistic support by balancing professional therapeutic competence with natural empathy. However, existing methods struggle to simultaneously achieve structured, stage-aware reasoning and seamless empathy-expertise alignment, often resulting in an artificial splicing of clinical strategies and generic reassurance. To overcome these limitations, we propose ESCRAG-R1, a unified framework that integrates retrieval-based psychological guidance into Group Relative Policy Optimization (GRPO). By incorporating retrieval into the reinforcement learning loop, ESCRAG-R1 transforms external knowledge into a robust learning signal that stimulates explicit internal reasoning prior to generation and fundamentally reshapes the model's internal policy. To provide the reliable supervision required for this optimization, we construct ESC-Preference, a high-quality dataset based on a Client--Counselor--Judge evaluation framework that delivers precise, empathy-aware reward signals. Extensive experiments demonstrate that ESCRAG-R1 significantly outperforms existing baselines by mitigating superficial splicing and realizing a natural integration of professional guidance and empathetic expression. Code and datasets are released at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.21925v1
- Canonical: https://arxiv.org/abs/2608.21925v1
Trouble viewing inline? Open PDF directly →
Full Text
54,359 characters extracted from source content.
Expand or collapse full text
ESCRAG-R1: Retrieval-Augmented Reinforcement Learning for Emotional Support Conversation Weichu Liu Affiliation: Beijing Institute of Technology, Shenzhen MSU-BIT University, Yuxuan Hu Yirong Sun Affiliation: City University of Hong Kong, Shenzhen University of Advanced Technology, Ningning Mao Ziyun Zhang Jian Chen Affiliation: Beijing Normal University, The University of Hong Kong liuwc@bit.edu.cn, yuxuanhu7-c@my.cityu.edu.hk, licm@smbu.edu.cn Mingyang Xu Affiliation: Beijing Institute of Technology, Shenzhen MSU-BIT University, Qishan Zhong Affiliation: Beijing Institute of Technology, Shenzhen MSU-BIT University, Chengming Li Affiliation: Beijing Institute of Technology, Shenzhen MSU-BIT University, Abstract Emotional Support Conversation (ESC) systems aim to provide holistic support by balancing professional therapeutic competence with natural empathy. However, existing methods struggle to simultaneously achieve structured, stage-aware reasoning and seamless empathy-expertise alignment, often resulting in an artificial splicing of clinical strategies and generic reassurance. To overcome these limitations, we propose ESCRAG-R1, a unified framework that integrates retrieval-based psychological guidance into Group Relative Policy Optimization (GRPO). By incorporating retrieval into the reinforcement learning loop, ESCRAG-R1 transforms external knowledge into a robust learning signal that stimulates explicit internal reasoning prior to generation and fundamentally reshapes the model’s internal policy. To provide the reliable supervision required for this optimization, we construct ESC-Preference, a high-quality dataset based on a Client–Counselor–Judge evaluation framework that delivers precise, empathy-aware reward signals. Extensive experiments demonstrate that ESCRAG-R1 significantly outperforms existing baselines by mitigating superficial splicing and realizing a natural integration of professional guidance and empathetic expression. Code and datasets are released at https://github.com/Matcha-Liu/ESCRAG-R1. 11footnotetext: Equal contribution.22footnotetext: Corresponding author. 1 Introduction Figure 1: Comparison of GRPO and ESCRAG-R1. By integrating retrieved guidance into GRPO, ESCRAG-R1 produces more grounded and empathetic responses. Driven by growing psychological needs and LLM advancements (35; 15), Emotional Support Conversation (ESC) systems have become vital resources for users seeking comfort and guidance (6; 23; 5). In real-world scenarios, users often share their experiences and emotional states with ESC systems, expecting empathetic and constructive responses (25; 17). The development of ESC systems is advancing toward deeper Emotional Intelligence, aiming to provide users with comprehensive emotional support that balances professional competence and empathy (41; 28; 2; 12). To achieve deep emotional intelligence, an effective ESC system must possess two core capabilities. The first is structured, stage-aware reasoning. Because emotional support is inherently process-oriented, generating a high-quality response requires inferring the client’s counseling stage, assessing their psychological state, and selecting appropriate interventions (1; 10; 7). This explicit reasoning ensures that empathy and guidance emerge from a coherent framework rather than isolated, reactive utterances. The second capability is empathy-expertise alignment. Psychologically grounded guidance must be delivered in a supportive, non-didactic manner. Responses that are emotionally warm but clinically ungrounded offer only superficial reassurance, while technically accurate yet affectively detached replies risk alienating vulnerable users (21). Ultimately, effective support hinges on seamlessly integrating rigorous psychological reasoning with carefully calibrated empathetic language. Existing ESC research generally follows two paradigms, both struggling to simultaneously fulfill these ideal capabilities. (i) Parameter-based approaches use supervised fine-tuning (4; 36) or reinforcement learning (39; 27) to mimic empathetic patterns. However, they critically lack explicit guidance for structured, stage-aware reasoning, relying on surface-level imitation rather than dynamically adapting to the client’s evolving psychological state. (i) Inference-augmented approaches incorporate agents (42; 37) or retrieval-augmented generation (13; 9) to inject psychological knowledge. Yet, this knowledge typically serves merely as external auxiliary guidance at inference time, rather than a definitive learning signal reshaping the model’s internal policy. Consequently, avoiding this artificial splicing requires a unified framework that integrates explicit reasoning guidance directly into the model’s internal policy. To overcome these limitations, we propose ESCRAG-R1, a unified framework that integrates retrieval-based psychological guidance into Group Relative Policy Optimization (GRPO), as shown in Figure 1. ESCRAG-R1 leverages retrieved counseling strategies to stimulate explicit, stage-aware reasoning prior to generation, ensuring the model conducts an internal psychological assessment rather than surface-level imitation. Furthermore, by incorporating this retrieval process into the GRPO loop, external knowledge acts as a robust learning signal that reshapes the model’s internal policy, seamlessly integrating professional therapeutic strategies with empathetic expression. To provide reliable reward supervision for this optimization, we construct a Client–Counselor–Judge evaluation framework and build ESC-Preference. This dataset delivers precise reward signals for GRPO, while its preferred responses serve as practice-grounded exemplars in the retrieval corpus for both training and inference. Our contributions are summarized as: • We propose ESCRAG-R1, a retrieval-augmented reinforcement learning framework that incorporates psychological guidance into GRPO for stage-aware emotional support. • We construct ESC-Preference based on a three-dimensional Client–Counselor–Judge evaluation framework, supporting both reward modeling and retrieval grounding. • Extensive experiments show that ESCRAG-R1 improves empathy-expertise alignment and generates more grounded, supportive, and therapeutically appropriate responses. Figure 2: The left panel shows the stage-aware annotation pipeline for ESC-Preference, and the right panel illustrates how retrieval-augmented guidance is integrated into GRPO to optimize grounded, empathetic support responses. 2 Related Work Parameter-based ESC Systems. Parameter-based approaches internalize supportive behaviors through optimization on specialized datasets. As the foundation for training, studies propose psychological counseling dialogue datasets with diverse topics and strategies to support subsequent model training (36; 19; 40). Many works adopt supervised fine-tuning (SFT) on counseling dialogues, enabling models to learn therapeutic patterns from the data and thereby deliver better services (4; 42). In reinforcement learning, some studies construct preference pair data and employ Direct Preference Optimization (DPO) (24) to align models with human preferences (39). With the emergence of Group Relative Policy Optimization (GRPO), several works have conducted preliminary explorations, though these attempts rely entirely on LLMs to explore strategies autonomously, lacking external guidance or structured intervention (29; 34; 27). However, these parameter-based methods mainly learn supportive patterns from fixed training data, without explicitly incorporating practice-grounded counseling knowledge into policy optimization. As a result, they may lack guidance for stage-aware reasoning and context-sensitive intervention, motivating our retrieval-augmented reinforcement learning framework. Inference-augmented ESC Systems. Inference-augmented approaches enhance ESC systems by introducing external knowledge or structured reasoning during inference without modifying model parameters. For example, some studies develop dialogue agent frameworks to construct ESC systems that are more empathetic and helpful (37; 38; 14). Beyond reasoning-focused approaches, a growing body of research draws upon established psychological theories to ground system behavior in clinically validated principles (30; 32). Additionally, recent studies adopt retrieval-augmented generation (RAG) frameworks to incorporate external knowledge or relevant dialogue exemplars during inference (31; 16; 20). Despite their strengths, the augmented information in these approaches often remains peripheral to the model’s core decision-making, serving as auxiliary input rather than actively shaping the underlying intervention policy. This motivates us to integrate retrieval-based psychological guidance into reinforcement learning, so that external counseling knowledge can participate in policy optimization rather than only assisting inference. 3 Method The overall framework of ESCRAG-R1 consists of two main components: ESC-Preference Construction and Retrieval-Guided Policy Optimization, as shown in Figure 2. ESC-Preference is constructed to provide both preference supervision for reward modeling and high-quality counseling exemplars for retrieval grounding. Based on this dataset, we first train a multi-perspective reward model, and then optimize the ESC policy through supervised fine-tuning and retrieval-augmented GRPO. 3.1 Task Definition Emotional Support Conversation (ESC) is formulated as a turn-level counselor response generation task. A dialogue consists of alternating user and counselor utterances, denoted as =u1,c1,u2,c2,…,uT,cTD=\u_1,c_1,u_2,c_2,…,u_T,c_T\, where uiu_i is the i-th user utterance, cic_i is the corresponding counselor response, and T is the number of counselor turns. At the i-th counselor turn, the model observes the dialogue history ℋi=u1,c1,…,ui−1,ci−1,uiH_i=\u_1,c_1,…,u_i-1,c_i-1,u_i\ and generates a response cic_i according to the counselor policy πθ(⋅∣ℋi) _θ(· _i), where θ denotes the policy parameters. The goal is to learn a counselor policy πθ _θ that maximizes the expected reward of generated responses over observed dialogue histories: πθ∗=argmaxπθℋi,ci∼πθ(⋅∣ℋi)[R(ℋi,ci)], _θ = _ _θ\;E_H_i,\,c_i _θ(· _i) [R(H_i,c_i) ], (1) where R(ℋi,ci)R(H_i,c_i) denotes the reward function that evaluates the emotional support quality of response cic_i under dialogue history ℋiH_i. 3.2 ESC-Preference Construction To obtain reward signals that balance professional competence and empathetic quality, as well as practice-grounded guidance for both training and inference, we construct the ESC-Preference dataset, which provides multi-perspective preference pairs and high-quality counseling exemplars. Based on ESConv (19), we simulate multi-turn client–counselor interactions to construct counseling dialogues. As shown in Figure 2, candidate models perform three-stage counseling inference to build contrasting reasoning paths, from which reliable preference pairs are obtained through stage-wise selection and final preference filtering. The prompts are provided in Appendix A. Stage-wise Response Reasoning. Given a dialogue history, we construct candidate responses through a three-stage counseling reasoning framework. This design follows the progressive nature of counseling interactions, where the counselor first understands the counseling context, then assesses the client’s psychological state, and finally selects an intervention strategy. This process grounds response generation in counseling-oriented reasoning rather than relying only on surface dialogue context. To obtain contrasting response candidates, three candidate models, including GPT-4o (22), Gemini-2.5-Pro (5), and DeepSeek-V3 (6), independently perform the three-stage inference. At each stage, intermediate selection is applied to retain both high- and low-quality reasoning states. These selected states are propagated through the subsequent stages, producing two contrasting reasoning trajectories, which are then used by GPT-4o to generate paired counselor responses. Multi-perspective Preference Evaluation. To evaluate the paired responses beyond a single overall score, we use GPT-5-chat (23) as the evaluator and adopt a multi-perspective evaluation framework covering the client, counselor, and judge viewpoints. The client perspective focuses on perceived emotional support quality, the counselor perspective evaluates therapeutic coherence and professional grounding, and the judge perspective assesses whether the response fits the client’s state and targets the core problem. Based on these evaluation results, we apply a dominance-based filtering criterion: the preferred response must achieve non-negative improvement across all metrics and strictly outperform the alternative response in at least three dimensions. Only pairs satisfying these conditions are retained. Through this process, we obtain 3,667 high-confidence preference pairs, containing 7,334 responses in total. 3.3 Retrieval-Guided Policy Optimization To move beyond inference-only guidance and enable retrieved counseling knowledge to shape the model policy, we use ESC-Preference to guide policy optimization. The preference pairs are used to train a reward model, and the preferred responses are organized as a retrieval corpus to provide counseling guidance. We first initialize the policy with a supervised reasoning-response pattern, and then integrate retrieval augmentation into GRPO. In this way, retrieved counseling exemplars are used during both policy optimization and inference, rather than serving only as inference-time guidance. Preference-Guided Reward Learning. To provide reward signals for policy optimization, we train a reward model on the preference pairs in ESC-Preference. Each preference pair is represented as (ℋi,ci+,ci−)(H_i,c_i^+,c_i^-), where ci+c_i^+ and ci−c_i^- denote the preferred and rejected responses under the same dialogue state ℋiH_i, respectively. The reward model estimates the relative quality of candidate responses by mapping each dialogue state and response to a scalar reward score. Specifically, it consists of a pretrained backbone fθ(⋅)f_θ(·) and a linear scoring head gψ(⋅)g_ψ(·), and the reward score riRMr_i^RM for a candidate response is computed as: riRM=gψ(fθ(ℋi,ci)).r_i^RM=g_ψ\! (f_θ(H_i,c_i) ). (2) During training, the backbone parameters θ are frozen and only the scoring head parameters ψ are optimized. For each preference pair, the reward model produces scores riRM+=Rϕ(ℋi,ci+)r_i^RM+=R_φ(H_i,c_i^+) and riRM−=Rϕ(ℋi,ci−)r_i^RM-=R_φ(H_i,c_i^-) for the preferred and rejected responses respectively. The loss function of the reward model is defined as: ℒRM=1N∑i=1N[−logσ(riRM+−riRM−) _RM= 1N _i=1^N [- σ(r_i^RM+-r_i^RM-) (3) +λ(riRM++riRM−)2], +λ(r_i^RM++r_i^RM-)^2 ], where σ(⋅)σ(·) denotes the sigmoid function and λ is a regularization coefficient. The first term encourages the reward model to assign higher scores to preferred responses, while the second term constrains the reward scale to prevent score drifting during training. During policy optimization, we further introduce an auxiliary format reward to encourage the policy model to follow the structured output format. Instead of directly generating the final counselor response, the model is expected to first organize its counseling reasoning and then provide the final supportive response. The format reward riformatr_i^format is defined as: riformat=1,if format satisfied;0,otherwise.r_i^format= cases1,&if format satisfied;\\ 0,&otherwise. cases (4) The final reward rir_i used during reinforcement learning is defined as: ri=riRM+riformat.r_i=r_i^RM+r_i^format. (5) Supervised Pattern Initialization. Before reinforcement learning, we initialize the policy model using a supervised fine-tuning (SFT) stage to establish a structured reasoning pattern for emotional support responses. Specifically, we generate 500 cold-start demonstrations using GPT-4o (22). Following the prompt design in Appendix B.2, each demonstration consists of a reasoning segment and a final response, formatted as <think></think> and <response></response>. Given a dialogue state ℋiH_i, let yiy_i denote the target structured output in the cold-start demonstration, which contains both the reasoning segment and the final response. The policy model πθ _θ is trained to generate yiy_i conditioned on ℋiH_i. The supervised fine-tuning objective is defined as: ℒSFT=−(ℋi,yi)[logπθ(yi∣ℋi)].L_SFT=-E_(H_i,y_i) [ _θ(y_i _i) ]. (6) This stage enables the model to learn the reasoning–response output format and provides a stable initialization for subsequent reinforcement learning with GRPO. Retrieval-Augmented GRPO. After supervised pattern initialization, we further optimize the counselor policy using GRPO with retrieval augmentation. Different from inference-only RAG, our goal is to expose the policy to retrieved counseling exemplars during the rollout process, so that retrieval-based guidance can influence both response generation and policy updates. Given a dialogue state ℋiH_i, the policy first generates an initial response c^i c_i: c^i∼πθ(⋅∣ℋi). c_i _θ(· _i). (7) The initial response is used as a query to retrieve relevant counseling exemplars from the retrieval corpus constructed during dataset generation. Let ℰi=e1,…,ekE_i=\e_1,…,e_k\ denote the retrieved exemplars. The policy then generates the final response conditioned on the dialogue state, the initial response, and the retrieved exemplars: ci∼πθ(⋅∣ℋi,c^i,ℰi).c_i _θ(· _i, c_i,E_i). (8) Model Client Counselor Judge Avg EI NE TA SC TH SF PT General-Purpose Models (Raw) GPT-4o 8.19 8.61 8.07 8.87 7.90 8.40 7.56 8.23 GPT-4o with RAG 8.79 8.96 8.56 8.95 8.13 8.76 7.96 8.59 Llama-3.1-8B-Instruct 8.20 8.48 8.06 8.71 7.78 8.27 7.65 8.16 Llama-3.1-8B-Instruct with RAG 8.73 8.50 8.59 8.09 7.71 8.29 7.57 8.21 Qwen-2.5-7B-Instruct 7.62 7.63 7.51 8.49 7.36 7.63 6.92 7.59 Qwen-2.5-7B-Instruct with RAG 8.31 8.50 8.12 8.22 7.49 8.00 7.18 7.97 Specialized ESC models ChatCounselor-7B 6.22 5.59 6.25 7.06 6.07 6.09 5.56 6.12 PsyLLM-8B 8.59 9.18 8.34 8.68 7.87 8.55 7.65 8.41 ESCRAG-R1 ESCRAG-R1-3B with RAG 8.68 9.31 8.55 8.80 8.02 8.58 7.56 8.50 ESCRAG-R1-7B with RAG 8.74 9.60 8.68 8.98 8.09 8.66 7.65 8.63 Table 1: Evaluation results from three perspectives. Client: EI (Emotional Impact), NE (Non-Lecturing Empathy), TA (Therapeutic Alliance); Counselor: SC (Strategic Coherence), TH (Theoretical Application); Judge: SF (Strategy-Client Fit), PT (Problem Targeting). The best and second-best results are bolded and underlined, respectively. During training, for each dialogue state ℋiH_i, the current policy samples a group of N retrieval-augmented candidate responses cijj=1N\c_i^j\_j=1^N, and each response is evaluated using the reward signal defined above. The group-relative advantage AijA_i^j for each sampled response is computed as: Aij=rij−1N∑k=1Nrik1N∑k=1N(rik−1N∑l=1Nril)2+ϵ,A_i^j= r_i^j- 1N _k=1^Nr_i^k 1N _k=1^N (r_i^k- 1N _l=1^Nr_i^l )^2+ε, (9) where rijr_i^j denotes the reward of the j-th sampled response, while rikr_i^k is used as the summation term over all responses in the same group. The constant ϵε is used for numerical stability. We define the importance sampling ratio ρij _i^j between the current policy and the old policy for the j-th sampled response as: ρij=πθ(cij∣ℋi,c^i,ℰi)πθold(cij∣ℋi,c^i,ℰi), _i^j= _θ(c_i^j _i, c_i,E_i) _ _old(c_i^j _i, c_i,E_i), (10) where ρij _i^j measures how the likelihood of the sampled response changes under the current policy πθ _θ compared with the old policy πθold _ _old. Based on the importance sampling ratio ρij _i^j and the group-relative advantage AijA_i^j, the policy is optimized with the following clipped GRPO objective: ℒGRPO(θ) _GRPO(θ) =−ℋi,cijj=1N[1N∑j=1N =-E_H_i,\c_i^j\_j=1^N [ 1N _j=1^N (11) min(ρijAij,clip(ρij,1−δ,1+δ)Aij)], ( _i^jA_i^j,\,clip( _i^j,1-δ,1+δ)A_i^j ) ], where clip(ρij,1−δ,1+δ)clip( _i^j,1-δ,1+δ) constrains the policy ratio within [1−δ,1+δ][1-δ,1+δ], and δ is a clipping coefficient that limits excessively large policy updates. The overall procedure of retrieval-augmented GRPO and RAG inference is summarized in Appendix D. 4 Experiment 4.1 Experiment Settings Dataset. We conduct experiments on ESConv (19) using its default training and test split. The training set contains 910 dialogues with 10,939 turns, which are used for reward model training, retrieval corpus construction, and GRPO-based policy optimization. During evaluation, we assess model performance on the test split, which contains 195 dialogues with a total of 2,505 turns. Evaluation Metrics. We evaluate model performance from three complementary perspectives: Client, Counselor, and Judge. For automatic evaluation, we use GPT-5-chat as the evaluator under a unified scoring prompt. Detailed descriptions of the metrics are provided in Appendix A.2. Implementation Details. For reward model, we adopt Qwen-2.5-7B (33; 26) as the backbone. For policy optimization, we use Qwen-2.5-3B-Instruct and Qwen-2.5-7B-Instruct as the backbone policy models. For the retrieval component, we use bge-m3 (3) to encode dialogue chunks and construct the vector database. For fairness, all RAG-based experiments use the same retrieval number, with N=3N=3 retrieved exemplars. Detailed hyperparameter settings are provided in Appendix C. Model Client Counselor Judge Avg EI NE TA SC TH SF PT Ablation on Framework Components Qwen-2.5-3B-Instruct 7.18 7.20 7.18 7.97 6.92 7.08 6.31 7.12 Qwen-2.5-3B-Instruct with RAG 7.93 8.16 7.81 7.18 6.90 7.00 6.12 7.30 ↑0.18 0.18 Cold-Start SFT 8.14 8.46 8.00 8.88 7.88 8.21 7.43 8.14 ↑1.02 1.02 + Vanilla-GRPO 8.19 9.06 8.13 9.00 8.02 8.52 7.55 8.35 ↑1.23 1.23 + Vanilla-GRPO + RAG Inference 8.49 9.30 8.35 8.96 8.03 8.46 7.52 8.44 ↑1.32 1.32 ESCRAG-R1-3B + RAG Inference 8.68 9.31 8.55 8.80 8.02 8.58 7.56 8.50 ↑1.38 1.38 Table 2: Ablation on framework components. Vanilla-GRPO uses standard GRPO training, RAG Inference applies retrieval augmentation only at test time, and ESCRAG-R1-3B uses our full joint training framework with retrieval augmentation during both training and test time. Baselines. We compare our method against two categories of baselines: general-purpose large language models and specialized emotional support dialogue models. The general-purpose models include GPT-4o (22), Llama-3.1-8B-Instruct (8), and Qwen-2.5-7B-Instruct(33; 26), while the specialized models include ChatCounselor-7B (18) and PsyLLM-8B (11). Considering the different architectures and usage settings of the models, we additionally evaluate the RAG setting only for general-purpose models. 4.2 Main Results Table 1 presents the overall comparison results from the Client, Counselor, and Judge perspectives. The first key finding is that ESCRAG-R1 achieves the strongest overall performance among all compared models. Specifically, ESCRAG-R1-7B with RAG obtains the highest average score of 8.63, while ESCRAG-R1-3B with RAG also achieves a competitive score of 8.50. These results demonstrate that the proposed framework is effective across different model scales and consistently improves the overall quality of ESC responses. The second key finding is that ESCRAG-R1 achieves a better balance between empathetic expression and professional grounding. Compared with GPT-4o with RAG, ESCRAG-R1-7B obtains a substantially higher score on Non-Lecturing Empathy (9.60 vs. 8.96) and a higher score on Therapeutic Alliance (8.68 vs. 8.56), indicating that our model produces responses that are perceived as more supportive and less didactic even when compared with a strong retrieval-augmented general-purpose model. Meanwhile, ESCRAG-R1-7B also maintains strong Counselor-side performance, with 8.98 on Strategic Coherence and 8.09 on Theoretical Application. These results show that ESCRAG-R1 improves empathetic expression while preserving professional therapeutic reasoning, thereby better supporting empathy-expertise alignment. The third key finding is that specialized ESC models still show limitations in jointly maintaining empathy and professional grounding. Although PsyLLM-8B achieves a strong average score of 8.41 and performs well on empathy-related metrics, ESCRAG-R1-7B further improves all three Client-side metrics and achieves higher Counselor-side scores, increasing Strategic Coherence from 8.68 to 8.98 and Theoretical Application from 7.87 to 8.09. Meanwhile, ChatCounselor-7B obtains a much lower average score of 6.12, with weak performance across both Client and Judge dimensions. These results suggest that counseling-oriented training alone may be insufficient to consistently produce responses that are both empathetic and strategically appropriate, highlighting the importance of integrating structured reasoning and preference-based policy optimization. Client: I feel like my friend group judges me for simply being myself. Although they are probably joking, I sometimes take their words personally, and it makes me upset. Counselor: It sounds like your friends’ jokes can feel judgmental to you, and even if they do not mean harm, their words still end up hurting you. Client: Should I still hang out with them if they keep making me feel this way? Response Initial Response (Vanilla-GRPO) It sounds like this situation is really tough for you. It’s understandable that you feel upset when people you care about are not being supportive. Maybe we can explore together what being in this situation feels like for you and what boundaries, if any, you might want to set. Would you be comfortable discussing how you want to approach your friend group moving forward? Final Response (ESCRAG-R1) It sounds like this situation is really affecting you, and I want to acknowledge how difficult that must be. It makes sense that you feel upset when people you care about say things that hurt, even if they are joking. Would you like to talk more about how this is affecting you? We can take our time to understand your feelings and what matters most to you. Remember, you’re not alone in this, and I’m here to support you as you explore your options. Retrieved Chunks [Chunk 1] History: The client is hurt and confused after a friend said she hated them while they were also dealing with family problems. Response: The counselor validates the client’s pain and confusion, encourages them to process feelings at their own pace, and reassures them that they do not have to go through this alone. [Chunk 2] History: The client feels rejected because friends are cold and unwilling to meet, then asks what they should do. Response: The counselor acknowledges the hurt, avoids direct advice, invites the client to explore how the coldness affects them, and supports taking things one step at a time. Table 3: Case study of ESCRAG-R1. Retrieved chunks guide the model to produce a more supportive final response, with highlighted text showing improved counseling behaviors. During training, the reward of such retrieval-guided responses is used to update the policy, allowing external guidance to shape model parameters. 4.3 Ablation Analysis Table 2 reports the ablation study on the core components of the framework, examining how each module contributes to the overall performance. Figure 3 further investigates how varying the number of retrieved exemplars affects model performance. Ablation on Framework Components. Table 2 shows that each component of ESCRAG-R1 contributes to the overall performance. Starting from the backbone, Qwen-2.5-3B-Instruct achieves an average score of 7.12, and adding RAG brings a modest improvement to 7.30, suggesting that retrieved exemplars provide useful supportive cues but are insufficient when used only as external inference-time context. After Cold-Start SFT establishes a structured reasoning-and-response pattern, Vanilla-GRPO further improves the average score from 8.14 to 8.35, showing that reinforcement learning can enhance the general quality of emotional support generation. Adding RAG Inference on top of Vanilla-GRPO further raises the score to 8.44, indicating that retrieved examples can provide additional guidance during generation. The full ESCRAG-R1-3B framework achieves the best overall score of 8.50, outperforming both Vanilla-GRPO and Vanilla-GRPO + RAG Inference. Notably, compared with Vanilla-GRPO + RAG Inference, ESCRAG-R1 further improves the Client-side metrics such as Emotional Impact and Therapeutic Alliance, as well as the Judge-side metrics of Strategy–Client Fit and Problem Targeting. These improvements suggest that integrating retrieval into policy optimization helps the model better internalize practice-grounded counseling strategies, leading to more supportive responses that are better aligned with the client’s state and core problem. Figure 3: Performance of ESCRAG-R1-7B with different retrieval numbers across evaluation dimensions. Ablation on Retrieval Number. Figure 3 illustrates the effect of different retrieval numbers on ESCRAG-R1-7B. Overall, increasing the retrieval number from n=2n=2 to n=3n=3 leads to consistent improvements across all evaluation dimensions, raising the average score from 8.44 to 8.63. This suggests that using too few retrieved exemplars may provide insufficient counseling guidance for response refinement. When the retrieval number further increases to n=4n=4, the overall performance remains competitive but slightly decreases to 8.61, indicating that additional retrieved examples do not necessarily bring further gains and may introduce redundant or less focused guidance. Figure 4: Human evaluation results on 100 sampled dialogue contexts from the ESConv test set. 4.4 Human Evaluation We further conduct human evaluation on 100 randomly sampled dialogue contexts from the ESConv test set. As shown in Figure 4, ESCRAG-R1-7B with RAG consistently achieves more wins than losses against all three baselines, obtaining 57 wins against GPT-4o with RAG, 54 wins against PsyLLM-8B without RAG, and 63 wins against Llama-3.1-8B-Instruct with RAG. Interestingly, although GPT-4o with RAG performs strongly in LLM-as-Judge evaluation, human annotators still prefer ESCRAG-R1 in more cases, and the comparison with PsyLLM-8B is relatively close. This suggests that models whose policies are updated with psychological counseling data may produce responses that better match human expectations. 4.5 Case Study Table 3 illustrates how retrieval augmentation helps ESCRAG-R1 refine its response. Compared with the initial response, the final response provides warmer emotional validation, avoids premature advice, and maintains a more supportive stance. The retrieved chunks guide the model to acknowledge the client’s hurt and encourage further exploration rather than directly prescribing solutions. Since retrieval-guided responses are evaluated during GRPO training, such guidance can further shape the policy beyond inference-time generation. 5 Conclusion In this paper, we propose ESCRAG-R1, a retrieval-augmented reinforcement learning framework for emotional support conversation. We construct ESC-Preference with a multi-perspective evaluation framework to support reward modeling and retrieval grounding. Experimental results show that ESCRAG-R1 improves emotional support quality across model scales and achieves stronger empathy-expertise alignment than general-purpose and specialized ESC models. 6 Limitations Despite the effectiveness of ESCRAG-R1, this work still has several limitations. First, due to computational resource constraints, our experiments are mainly conducted on 3B and 7B policy models. Although the results demonstrate consistent improvements across different model scales, we have not fully investigated how the proposed framework performs on larger-scale LLMs. Second, this work focuses on text-only Emotional Support Conversation. However, real-world emotional support often involves multimodal signals, such as facial expressions, speech tone, and other behavioral cues, which may provide important information about the user’s emotional state. 7 Ethical Considerations This work aims to improve the performance of Emotional Support Conversation (ESC) models and is intended for research purposes only. All data and models used in this work are based on publicly available and open-source resources, and our experiments do not involve private user data or personally identifiable information. During dataset construction, we carefully design the annotation and generation pipeline to reduce the risk of producing harmful or inappropriate content. The prompts used for client simulation, counselor reasoning, and response evaluation are designed to guide the models toward supportive, non-harmful, and therapeutically appropriate interactions, as detailed in Appendix A. Nevertheless, we acknowledge that automatically generated emotional support responses may still carry potential risks if used in real-world settings without proper safeguards. Therefore, ESCRAG-R1 and the constructed dataset are intended only for academic research. They should not be directly used as a substitute for professional psychological counselors or mental health services. Any future practical deployment should include strict safety mechanisms, human oversight, and clear instructions for users to seek professional help in high-risk situations. 8 Acknowledgements This work was supported in part by Guangdong Province Science and Technology Foundation (No. 2026A1515011806, No. 2024TQ08X559), in part by Innovation Team Project of Guangdong Province (No. 2024KCXTD017), in part by Shenzhen Science and Technology Foundation (No. JCYJ20240813145816022). References Beck (2020) J. S. Beck Cognitive behavior therapy: basics and beyond. Guilford Publications. Cited by: §1. Chen et al. (2025) J. Chen, Y. Hu, H. Lu, W. Wang, M. Yang, C. Li, and X. Hu MGHFT: multi-granularity hierarchical fusion transformer for cross-modal sticker emotion recognition. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 5794–5803. Cited by: §1. Chen et al. (2024) J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu BGE m3-embedding: multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. External Links: 2402.03216 Cited by: §4.1. Chen et al. (2023) Y. Chen, X. Xing, J. Lin, H. Zheng, Z. Wang, Q. Liu, and X. Xu SoulChat: improving llms’ empathy, listening, and comfort abilities through fine-tuning with multi-turn empathy conversations. arXiv preprint arXiv:2311.00273. Cited by: §1, §2. Comanici et al. (2025) G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §1, §3.2. DeepSeek-AI (2024) DeepSeek-AI DeepSeek-v3 technical report. External Links: 2412.19437, Link Cited by: §1, §3.2. Elliott (2002) R. Elliott The effectiveness of humanistic therapies: a meta-analysis.. Cited by: §1. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, et al. The llama 3 herd of models. External Links: 2407.21783, Link Cited by: §4.1. Guo et al. (2024) Q. Guo, J. Tang, W. Sun, H. Tang, Y. Shang, and W. Wang Soullmate: an application enhancing diverse mental health support with adaptive llms, prompt engineering, and rag techniques. arXiv preprint arXiv:2410.16322. Cited by: §1. Hayes et al. (2006) S. C. Hayes, J. B. Luoma, F. W. Bond, A. Masuda, and J. Lillis Acceptance and commitment therapy: model, processes and outcomes. Behaviour research and therapy 44 (1), p. 1–25. Cited by: §1. Hu et al. (2025) H. Hu, Y. Zhou, J. Si, Q. Wang, H. Zhang, F. Ren, F. Ma, L. Cui, and Q. Tian Beyond empathy: integrating diagnostic and therapeutic reasoning with large language models for mental health counseling. arXiv preprint arXiv:2505.15715. Cited by: §4.1. Hu et al. (2026) Y. Hu, J. Chen, Y. Wang, Z. Li, J. Xiong, P. Jia, W. Wang, C. Li, and X. Zhao Emotion and intention guided multi-modal learning for sticker response selection. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 14883–14891. Cited by: §1. Hu et al. (2024) Y. Hu, M. Tan, C. Zhang, Z. Li, X. Liang, M. Yang, C. Li, and X. Hu Aptness: incorporating appraisal theory and emotion support strategies for empathetic response generation. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, p. 900–909. Cited by: §1. Li et al. (2024a) J. Li, B. Peng, Y. Hsu, and C. Huang Be helpful but don’t talk too much-enhancing helpfulness in conversations through relevance in multi-turn emotional support. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 1976–1988. Cited by: §2. Li et al. (2025) Z. Li, B. Geng, J. Xiong, Y. He, Y. Hu, J. Chen, D. Chen, X. Chang, N. Wong, L. Zhang, et al. Ctr-sink: attention sink for language models in click-through rate prediction. arXiv preprint arXiv:2508.03668. Cited by: §1. Li et al. (2024b) Z. Li, J. Xiong, F. Ye, C. Zheng, X. Wu, J. Lu, Z. Wan, X. Liang, C. Li, Z. Sun, et al. Uncertaintyrag: span-level uncertainty enhanced long-context modeling for retrieval-augmented generation. arXiv preprint arXiv:2410.02719. Cited by: §2. Lin et al. (2019) Z. Lin, A. Madotto, J. Shin, P. Xu, and P. Fung Moel: mixture of empathetic listeners. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 121–132. Cited by: §1. Liu et al. (2023) J. M. Liu, D. Li, H. Cao, T. Ren, Z. Liao, and J. Wu Chatcounselor: a large language models for mental health support. arXiv preprint arXiv:2309.15461. Cited by: §4.1. Liu et al. (2021) S. Liu, C. Zheng, O. Demasi, S. Sabour, Y. Li, Z. Yu, Y. Jiang, and M. Huang Towards emotional support dialog systems. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: Long papers), p. 3469–3483. Cited by: §2, §3.2, §4.1. Liu et al. (2025) W. Liu, J. Xiong, Y. Hu, Z. Li, M. Tan, N. Mao, H. Shen, W. Xu, C. Tao, M. Yang, et al. LongEmotion: measuring emotional intelligence of large language models in long-context interaction. arXiv preprint arXiv:2509.07403. Cited by: §2. Nienhuis et al. (2018) J. B. Nienhuis, J. Owen, J. C. Valentine, S. Winkeljohn Black, T. C. Halford, S. E. Parazak, S. Budge, and M. Hilsenroth Therapeutic alliance, empathy, and genuineness in individual adult psychotherapy: a meta-analytic review. Psychotherapy Research 28 (4), p. 593–605. Cited by: §1. OpenAI (2024) OpenAI OpenAI: Hello GPT-4o. Note: https://openai.com/zh-Hans-CN/index/hello-gpt-4o/Accessed: 2025-07-24 Cited by: §3.2, §3.3, §4.1. OpenAI (2025) OpenAI GPT-5 system card. External Links: Link Cited by: §1, §3.2. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §2. Rashkin et al. (2019) H. Rashkin, E. M. Smith, M. Li, and Y. Boureau Towards empathetic open-domain conversation models: a new benchmark and dataset. In Proceedings of the 57th annual meeting of the association for computational linguistics, p. 5370–5381. Cited by: §1. Team (2024) Q. Team Qwen2.5: a party of foundation models. External Links: Link Cited by: §4.1, §4.1. Wang et al. (2025a) P. Wang, R. Ma, B. Zhang, X. Chen, Z. He, K. Luo, Q. Lv, Q. Jiang, Z. Xie, S. Wang, et al. Rlver: reinforcement learning with verifiable emotion rewards for empathetic agents. arXiv preprint arXiv:2507.03112. Cited by: §1, §2. Wang et al. (2023) X. Wang, X. Li, Z. Yin, Y. Wu, and J. Liu Emotional intelligence of large language models. Journal of Pacific Rim Psychology 17, p. 18344909231213958. Cited by: §1. Wang et al. (2025b) Y. Wang, M. Liu, W. Liu, K. Jiang, B. Wen, F. Yang, T. Gao, G. Zhou, and L. Nie COMPEER: controllable empathetic reinforcement reasoning for emotional support conversation. arXiv preprint arXiv:2508.09521. Cited by: §2. Xiao et al. (2024) M. Xiao, Q. Xie, Z. Kuang, Z. Liu, K. Yang, M. Peng, W. Han, and J. Huang HealMe: harnessing cognitive reframing in large language models for psychotherapy. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1707–1725. Cited by: §2. Xiong et al. (2024) J. Xiong, Z. Li, C. Zheng, Z. Guo, Y. Yin, E. Xie, Z. Yang, Q. Cao, H. Wang, X. Han, et al. Dq-lore: dual queries with low rank approximation re-ranking for in-context learning. In International Conference on Learning Representations, Vol. 2024, p. 41179–41203. Cited by: §2. Xu et al. (2025) A. Xu, D. Yang, R. Li, J. Zhu, M. Tan, M. Yang, W. Qiu, M. Ma, H. Wu, B. Li, et al. Autocbt: an autonomous multi-agent framework for cognitive behavioral therapy in psychological counseling. arXiv preprint arXiv:2501.09426. Cited by: §2. Yang et al. (2024) A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, and Z. Fan Qwen2 technical report. arXiv preprint arXiv:2407.10671. Cited by: §4.1, §4.1. Yang et al. (2025) T. Yang, L. Chen, and H. Wang Towards open-ended emotional support conversations in llms via reinforcement learning with future-oriented rewards. arXiv preprint arXiv:2508.12935. Cited by: §2. Yi et al. (2025) Z. Yi, J. Ouyang, Z. Xu, Y. Liu, T. Liao, H. Luo, and Y. Shen A survey on recent advances in llm-based multi-turn dialogue systems. ACM Computing Surveys 58 (6), p. 1–38. Cited by: §1. Zhang et al. (2024a) C. Zhang, R. Li, M. Tan, M. Yang, J. Zhu, D. Yang, J. Zhao, G. Ye, C. Li, and X. Hu Cpsycoun: a report-based multi-turn dialogue reconstruction and evaluation framework for chinese psychological counseling. arXiv preprint arXiv:2405.16433. Cited by: §1, §2. Zhang et al. (2024b) T. Zhang, X. Zhang, J. Zhao, L. Zhou, and Q. Jin Escot: towards interpretable emotional support dialogue systems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 13395–13412. Cited by: §1, §2. Zhang et al. (2025) X. Zhang, W. Wang, and Q. Jin IntentionESC: an intention-centered framework for enhancing emotional support in dialogue systems. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 26494–26516. External Links: Link, Document, ISBN 979-8-89176-256-5 Cited by: §2. Zhao et al. (2025) W. Zhao, X. Sui, X. Han, Y. Deng, Y. Hu, J. Guo, L. Qin, Q. Du, S. Wang, Y. Zhao, B. Qin, and T. Liu Chain of strategy optimization makes large language models better emotional supporter. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 15361–15381. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §1, §2. Zheng et al. (2023) Z. Zheng, L. Liao, Y. Deng, and L. Nie Building emotional support chatbots in the era of llms. arXiv preprint arXiv:2308.11584. Cited by: §2. Zhong et al. (2021) P. Zhong, D. Wang, P. Li, C. Zhang, H. Wang, and C. Miao CARE: commonsense-aware emotional response generation with latent concepts. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, p. 14577–14585. Cited by: §1. Zhou et al. (2025) Y. Zhou, N. Zhou, Q. Chen, J. Zhou, A. Zhou, and L. He DiaCBT: a long-periodic dialogue corpus guided by cognitive conceptualization diagram for cbt-based psychological counseling. arXiv preprint arXiv:2509.02999. Cited by: §1, §2. Appendix A Prompt Design for ESC-Preference Construction A.1 Client Simulator Prompt We design a role-playing prompt to simulate the client as a patient seeking emotional support. The prompt enforces the client to remain consistent with the given emotional and situational context, respond naturally to the therapist, and generate only one short and succinct sentence for the next turn without additional explanation. The prompt design can be seen in Figure 5. Figure 5: Client role-playing prompt. A.2 Evaluator Simulator Prompt We design an evaluator prompt to evaluate candidate responses from multiple perspectives and provide reliable preference signals. The evaluation considers three complementary roles: client, counselor, and judge, ensuring both empathetic quality and professional appropriateness. During automatic evaluation, we also adopt the same multi-perspective criteria in the LLM-as-a-Judge setting. The prompt design can be seen in Figure 6. Figure 6: Evaluator role-playing prompt. A.3 Counselor Simulator Prompt We design a stage-aware prompting framework that decomposes the counselor’s reasoning into three stages, each implemented with a dedicated prompt. Counseling stage identification. The model infers the client’s current counseling phase based on the dialogue context. The prompt design can be seen in Figure 7. Figure 7: Counselor role-playing prompt for counseling stage identification. Psychological state assessment. The model evaluates the client’s psychological state in terms of distress, resilience, and motivation. The prompt design can be seen in Figure 8. Figure 8: Counselor role-playing prompt for psychological state assessment. Therapeutic strategy selection. The model selects appropriate counseling strategies conditioned on the inferred stage and psychological state. The prompt can be seen in Figure 9 and Figure 10. Figure 9: Counselor role-playing prompt for therapeutic strategy selection (part 1). Figure 10: Counselor role-playing prompt for therapeutic strategy selection (part 2). Appendix B Prompt Design for ESCRAG-R1 Training Framework B.1 Reward Model Prompt The reward model is trained to evaluate whether a counselor response effectively addresses the client’s emotional problem under a given conversation history. Reward Model Prompt Instruction: You are a helpful assistant tasked with evaluating whether a patient’s emotional problem has been effectively addressed following a conversation with a therapist. Conversation History: conversation_history Candidate Response: candidate_response B.2 Policy Model Prompt The policy model is prompted to generate responses in a structured reasoning–response format. During retrieval-augmented generation, the model receives the dialogue history, an initial response, and retrieved dialogue exemplars from the external corpus, and then generates the final counselor response. Policy Model Prompt System Prompt: Now enter the role-playing mode. In the following conversation, you will play as a therapist in a counselling conversation with a patient. Your goal is to help the patient reduce their emotional distress and support them working through their challenges. You first think about the reasoning process in the mind and then provide the patient with the response. The reasoning process and response are enclosed within <think> </think> and <response> </response> tags, respectively, i.e., <think> reasoning process here </think> <response> supportive response here </response>. User Prompt: You are an empathetic and helpful assistant. Given the following dialogue context, your initial response and response guidelines retrieved from an external library, your task is to generate a final response to the user. This response should incorporate empathy and understanding, provide helpful guidance or suggestions, and be conversational and natural. [Dialogue History]: dialogue_history [Initial Response]: initial_response [Retrieved Chunks from External Library]: retrieved_dialogue Appendix C Parameter Setting Table 4 summarizes the main parameter settings used for reward model training and Retrieval-Augmented GRPO. The reward model is built on Qwen-2.5-7B and trained using two A800 GPUs. For policy optimization, ESCRAG-R1-3B is trained on two A800 GPUs, while ESCRAG-R1-7B is trained on four A800 GPUs. Parameter Value Reward Model Training Epochs 5 Per-device batch size 1 Gradient accumulation steps 16 Learning rate 5×10−55× 10^-5 Reward centering coefficient 0.01 Retrieval-Augmented GRPO Batch size 128 Epochs 2 Learning rate 1×10−61× 10^-6 KL coefficient 0.05 Number of sample generation 4 Number of retrieved chunks 3 Temperature 0.9 Top-p 0.95 Table 4: Hyperparameter settings for reward model training and Retrieval-Augmented GRPO. Appendix D Retrieval-Augmented GRPO and RAG Inference Algorithm 1 summarizes how retrieval is used in ESCRAG-R1 during both training and inference. In retrieval-augmented GRPO, retrieved counseling exemplars are incorporated into the rollout process, and the reward of retrieval-guided responses is used to update the policy. In RAG inference, the optimized policy uses the same retrieval-augmented generation process to produce the final response, but no parameter update is performed. Thus, retrieval serves as a learning signal during training and as generation guidance during inference. Algorithm 1 Retrieval-Augmented GRPO and RAG Inference of ESCRAG-R1 1: Training set train=ℋii=1MD_train=\H_i\_i=1^M, test set test=ℋii=1M′D_test=\H_i\_i=1^M , policy model πθ _θ, old policy πθold _ _old, reward model RϕR_φ, retriever ℛR, group size N 2: Optimized policy parameters θ and final responses cifinali=1M′\c_i^final\_i=1^M 3: Retrieval-Augmented GRPO 4: for each training step do 5: Sample a dialogue state ℋiH_i from trainD_train 6: for j=1j=1 to N do 7: Generate an initial response c^ij∼πθ(⋅∣ℋi) c_i^j _θ(· _i) 8: Retrieve relevant exemplars ℰij←ℛ(c^ij)E_i^j ( c_i^j) 9: Generate a retrieval-augmented response cij∼πθ(⋅∣ℋi,c^ij,ℰij)c_i^j _θ(· _i, c_i^j,E_i^j) 10: Compute reward rij←Rϕ(ℋi,cij)r_i^j← R_φ(H_i,c_i^j) 11: end for 12: Compute group-relative advantages Aijj=1N\A_i^j\_j=1^N from rewards rijj=1N\r_i^j\_j=1^N 13: Compute importance sampling ratios ρijj=1N\ _i^j\_j=1^N with respect to πθold _ _old 14: Compute the clipped GRPO loss ℒGRPO(θ)L_GRPO(θ) 15: Update policy parameters θ by optimizing ℒGRPO(θ)L_GRPO(θ) 16: end for 17: Obtain the optimized policy πθ _θ 18: RAG Inference 19: for each dialogue state ℋiH_i in testD_test do 20: Generate an initial response c^i∼πθ(⋅∣ℋi) c_i _θ(· _i) 21: Retrieve relevant exemplars ℰi←ℛ(c^i)E_i ( c_i) 22: Generate the final response cifinal∼πθ(⋅∣ℋi,c^i,ℰi)c_i^final _θ(· _i, c_i,E_i) 23: end for 24: return θ and cifinali=1M′\c_i^final\_i=1^M Appendix E Human Evaluation To complement automatic evaluation, we conduct a human evaluation to assess the quality of generated responses. We recruit three students with backgrounds in psychology as human annotators and provide them with training before the annotation process. During training, annotators are introduced to the evaluation criteria and the scoring prompt used in our study, ensuring that they understand the meaning of each evaluation dimension and apply the criteria consistently. For each dialogue context, annotators are presented with a blind pair of responses generated by different models, where the model identities are hidden to reduce potential bias. Following the evaluation prompt, annotators score each response from the Client–Counselor–Judge perspectives, covering emotional support quality, therapeutic coherence, and contextual appropriateness. After annotation, we compare the scores assigned to each response in the blind pair and determine model preference according to the higher overall score.