Paper deep dive
Ask to Be Sure: Informative Interactions for Confident Multi-Turn LLM Recommendation
Cedar Site Bai, Duanshun Li, Zhenyu Liao, Sheikh Sarwar, Huiyuan Chen, Yuan Chen, Changhe Yuan, Haiyang Zhang, Qilin Qi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 8/22/2026, 3:21:12 AM
Summary
This paper proposes a method to improve multi-turn Large Language Model (LLM) recommendation by using entropy reduction over recommendations as a reward signal for fine-tuning. Unlike previous methods that rely on ground-truth data or LLM-judged interactivity, this approach quantifies the information gain from user interactions by measuring the decrease in the assistant's uncertainty (Shannon entropy) about the recommended items. The method is validated using Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) on the INSPIRED and ReDial datasets, demonstrating improvements in both recommendation accuracy (Hit@1, Hit@5) and conversational efficiency (number of turns).
Entities (10)
Relation Signals (8)
Ask to Be Sure → evaluatedon → Inspired
confidence 99% · Empirical results with supervised fine-tuning (SFT) and direct preference optimization (DPO) on the INSPIRED and ReDial datasets show that our method improves both recommendation quality and conversational efficiency.
Ask to Be Sure → evaluatedon → ReDial
confidence 99% · Empirical results with supervised fine-tuning (SFT) and direct preference optimization (DPO) on the INSPIRED and ReDial datasets show that our method improves both recommendation quality and conversational efficiency.
Ask to Be Sure → uses → Entropy Reduction
confidence 98% · We propose a new approach that quantifies the effectiveness of each interaction by the reduction in the assistant's uncertainty, measured via entropy over recommendations.
Entropy Reduction → measures → Uncertainty
confidence 97% · We define uncertainty as the Shannon entropy over the distribution of recommendations... We measure this information gain as the reduction in the assistant’s recommendation uncertainty
Ask to Be Sure → finetunedwith → Supervised Fine-Tuning
confidence 96% · Empirical results with supervised fine-tuning (SFT) and direct preference optimization (DPO) on the INSPIRED and ReDial datasets show that our method improves both recommendation quality and conversational efficiency.
Ask to Be Sure → finetunedwith → Direct Preference Optimization
confidence 96% · Empirical results with supervised fine-tuning (SFT) and direct preference optimization (DPO) on the INSPIRED and ReDial datasets show that our method improves both recommendation quality and conversational efficiency.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advances in large language models (LLMs) have enabled their use as conversational recommender systems (CRS), demonstrating strong recommendation accuracy and natural dialogue. However, guiding multi-turn interactions to elicit user preferences effectively remains challenging. Existing approaches either use separate reinforcement learning agents with templated interactions or optimize for interactivity judged by another LLM, without measuring how much useful information is actually gained. We propose a new approach that quantifies the effectiveness of each interaction by the reduction in the assistant's uncertainty, measured via entropy over recommendations. We apply this entropy reduction as a reward---without relying on ground-truth recommendations, which are often unavailable in real-world scenarios---to fine-tune the LLM, enabling strategic interaction generation. Empirical results with supervised fine-tuning (SFT) and direct preference optimization (DPO) on the INSPIRED and ReDial datasets show that our method improves both recommendation quality and conversational efficiency.
Tags
Links
- Source: https://arxiv.org/abs/2608.15949v1
- Canonical: https://arxiv.org/abs/2608.15949v1
Trouble viewing inline? Open PDF directly →
Full Text
35,979 characters extracted from source content.
Expand or collapse full text
Ask to Be Sure: Informative Interactions for Confident Multi-Turn LLM RecommendationConference: Proceedings of the 35th ACM International Conference on Information and Knowledge Management; November 07–11, 2026; Rome, ItalyProceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM ’26), November 07–11, 2026, Rome, ItalyDOI: 10.1145/3799682.3839868ISBN: 979-8-4007-2539-5/2026/11CCS: Information systems Recommender systems Cedar Site Bai Affiliation: Amazon , Sunnyvale , CA , USA email: cedarbai@amazon.com , Duanshun Li Affiliation: Amazon , Seattle , WA , USA email: duanshun@amazon.com , Zhenyu Liao Affiliation: Amazon , Sunnyvale , CA , USA email: zyliao@amazon.com , Sheikh Sarwar Affiliation: Amazon , Sunnyvale , CA , USA email: smsarwar@amazon.com , Huiyuan Chen Affiliation: Amazon , Sunnyvale , CA , USA email: huiyuach@amazon.com , Yuan Chen Affiliation: Amazon , Sunnyvale , CA , USA email: yuanchn@amazon.com , Changhe Yuan Affiliation: Amazon , New York , NY , USA email: ychanghe@amazon.com , Haiyang Zhang Affiliation: Amazon , Sunnyvale , CA , USA email: hhaiz@amazon.com and Qilin Qi Affiliation: Amazon , Sunnyvale , CA , USA email: qilinqi@amazon.com 2026; © c Abstract. Recent advances in large language models (LLMs) have enabled their use as conversational recommender systems (CRS), demonstrating strong recommendation accuracy and natural dialogue. However, guiding multi-turn interactions to elicit user preferences effectively remains challenging. Existing approaches either use separate reinforcement learning agents with templated interactions or optimize for interactivity judged by another LLM, without measuring how much useful information is actually gained. We propose a new approach that quantifies the effectiveness of each interaction by the reduction in the assistant’s uncertainty, measured via entropy over recommendations. We apply this entropy reduction as a reward—without relying on ground-truth recommendations, which are often unavailable in real-world scenarios—to fine-tune the LLM, enabling strategic interaction generation. Empirical results with supervised fine-tuning (SFT) and direct preference optimization (DPO) on the INSPIRED and ReDial datasets show that our method improves both recommendation quality and conversational efficiency. Keywords: Conversational Recommender Systems, Multi-turn Recommendation, Interactive Recommendation, Large Language Models, Uncertainty Estimation, Entropy Reduction, AI Assistant †c-license: by 1. Introduction Conversational recommendation systems (CRS) (19; 14; 4; 25; 20) have recently garnered increasing attention for their ability to interpret user intent and provide personalized recommendations through natural language interaction. By engaging users in multi-turn dialogues, CRS can elicit preferences, clarify ambiguous requests, and refine recommendations from real-time feedback, transforming recommendation from a one-shot prediction task into an interactive preference discovery process. With the rapid development of large language models (LLMs), there has been a noticeable shift toward employing LLMs directly as conversational recommenders (10; 28; 27; 22; 11). These systems have shown superior performance over conventional systems (14; 4; 25) in both recommendation accuracy and user-aware dialogue. Despite this progress, key challenges remain, including preference elicitation, the integration of user history and collaborative filtering, up-to-date candidate retrieval, and grounding in real-world knowledge. We focus on the first challenge: understanding and eliciting user preferences through strategic, multi-turn interactions, while leaving the others for future exploration. We specifically examine how to design interactions that most effectively solicit user preferences. In an ideal setting, conversational recommendation should be a collaborative process between the user and the AI assistant, rather than a one-way delivery of suggestions. Existing approaches that use multi-turn dialogue to capture user preferences typically train a separate reinforcement learning (RL) agent to decide when and what to ask or recommend (5; 6). In such settings, LLMs are used only for dialogue generation, not recommendation, and interaction styles are restricted to rigid formats such as multiple-choice questions or Yes/No responses, limiting natural conversational flow. In parallel, CollabLLM encourages assistant interactivity through LLM-judge rewards (21); while effective in making LLMs more proactive in general-purpose tasks, in recommendation contexts it remains unclear how much useful preference information is gained and how these interactions benefit recommendation. To tackle this challenge, we draw inspiration from recent findings on the correlation between recommendation reliability and uncertainty (12). That work estimates predictive uncertainty over ranking lists from candidate top-1 probabilities using a Plackett–Luce model and finds that lower uncertainty is associated with better recommendation performance. We propose a method to quantify the effectiveness of each assistant interaction by the information provided in the corresponding user response. We measure this information gain as the reduction in the assistant’s recommendation uncertainty, computed via entropy over recommendations. This entropy reduction is then used as a reward—without relying on ground-truth recommendations, which are often unavailable in real-world scenarios—to fine-tune the LLM for strategic interaction generation. Empirical results on the INSPIRED and ReDial datasets, using both supervised fine-tuning (SFT) and direct preference optimization (DPO), demonstrate that our method improves both recommendation quality and conversational efficiency. Contributions. (1) We introduce an uncertainty-reduction reward that improves conversation strategy and recommendation accuracy. (2) The metric provides a practical solution when ground-truth recommendations are unavailable. (3) We contribute two turn-level completion-pair datasets from INSPIRED and ReDial for DPO fine-tuning. 2. Related Work Recent work shows that LLMs remain brittle in multi-turn interaction (13), motivating methods for clarification, preference following, and multi-turn RLHF (3; 23; 18; 26; 1; 7). CollabLLM encourages assistants to be proactive through collaboratively generated data and LLM-judge rewards (21); however, in recommendation contexts, a response can appear interactive while still revealing little useful preference information. Recommendation-oriented LLM work improves item control, tokenization, retrieval, or collaborative-signal integration (15; 11; 27; 24; 22), but these methods mainly improve what is recommended given a fixed conversation state rather than how the assistant should change that state through strategic elicitation. Orthogonally, Bayesian teaching studies how models update beliefs under controlled preference-elicitation settings (16). Our focus is targeted and recommendation-specific: we measure which assistant turns reduce uncertainty over recommendations, and use that information gain as a reward for open-ended conversational recommendation. 3. Proposed Method In conversational recommendation, the recommender (or AI assistant) engages users in multi-turn dialogues to dynamically elicit preferences, often by asking targeted questions. With each turn, the user provides more information, enabling the assistant to better understand their preferences and become more certain about what to recommend. For example, at the very beginning, the assistant has no prior knowledge of the user and is highly uncertain, resorting to essentially random recommendations. As the dialogue unfolds, the user reveals likes and dislikes, recently watched movies, and other relevant details. To encourage the assistant to ask the most informative questions, we aim for each interaction to significantly reduce recommendation uncertainty. We now introduce our measure of uncertainty. 3.1. Uncertainty Measurement We define uncertainty as the Shannon entropy over the distribution of recommendations induced by repeated sampling of a fixed prompt given a conversation C. Following 12, we use entropy over a recommendation distribution as an uncertainty estimate. Whereas they estimate a distribution over ranking permutations from candidate logits at a fixed conversation state, we empirically estimate an item distribution from repeated top-m list generation, incorporate rank position through weighted entropy, and track how uncertainty changes across conversational interactions. The prompt is as follows: Given the following conversation history: conversation C Generate a list of the top m movies the user would like to watch based on this conversation. Format your response as a numbered list with no extra sentences: 1. [First movie] 2. [Second movie] 3. [Third movie] Make sure each recommendation is unique and plausible given the conversation context. 3.1.1. Entropy over recommendations We sample n lists L(i)i=1n\L^(i)\_i=1^n generated by the AI assistant based on a conversation C, where each list L(i)=[rj(i)]j=1mL^(i)=[\,r^(i)_j\,]_j=1^m contains the top-m movie recommendations. Let R=Set(L(i)i=1n)R=Set (\L^(i)\_i=1^n ) denote the set of unique recommendations appearing across all lists. For each item r∈Rr∈ R, define its count and empirical frequency as (1) c(r) c(r) =∑i=1n∑j=1mrj(i)=r, = _i=1^n _j=1^m1\\,r^(i)_j=r\,\, (2) p(r) p(r) =c(r)mn, = c(r)m\,n, which serves as an empirical estimate of the recommendation distribution. Here, ⋅1\·\ is the indicator function that equals 1 if rj(i)=r_j^(i)=r and 0 otherwise. The entropy over recommendations is calculated as (3) H(C)=−∑r∈Rp(r)log2p(r).H(C)\;=\;- _r∈ Rp(r)\, _2p(r). 3.1.2. Weighted entropy over recommendations To further incorporate the difference in ranking order in each recommendation list L(i)=[rj(i)]j=1mL^(i)=[\,r^(i)_j\,]_j=1^m, we apply the following weighted entropy with logarithmic decay. For each rank position j∈1,2,…,mj∈\1,2,...,m\, the weight is (4) wj=1log2(j+1),j∈1,2,…,m.w_j\;=\; 1 _2(j+1), j∈\1,2,…,m\. For each item r∈Rr∈ R, define the weighted count as (5) cw(r)=∑i=1n∑j=1mwj 1rj(i)=r,c_w(r)\;=\; _i=1^n _j=1^mw_j\,1\\,r^(i)_j=r\,\, and the corresponding weighted empirical distribution is calculated as (6) pw(r)=cw(r)∑r′∈Rcw(r′)=cw(r)n∑j=1mwj.p_w(r)\;=\; c_w(r) _r ∈ Rc_w(r )\;=\; c_w(r)\,n _j=1^mw_j\,. The weighted entropy is then (7) Hw(C)=−∑r∈Rpw(r)log2pw(r).H_w(C)\;=\;- _r∈ Rp_w(r)\, _2p_w(r). 3.2. Information Gain as Uncertainty Reduction With uncertainty defined, we measure the reduction in uncertainty after an interaction, which we regard as recommendation-relevant information gain. Building on this uncertainty estimator, our contribution is to use its reduction after an assistant turn and the user response it elicits as a reward. This assigns credit to interactions according to recommendation-relevant information gain rather than an LLM judge’s general assessment of interactivity (21). We introduce two entropy-reduction measures. 3.2.1. Turn-level entropy reduction Turn-level entropy reduction measures the reduction in entropy attributable to one conversational turn. Specifically, given a partial conversation C1=[U1]C_1=[U_1], compute the uncertainty Hw(C1)H_w(C_1). An assistant interaction A1A_1 is generated (e.g., an initial recommendation or a clarifying question). The user responds with U2U_2 (e.g., expressing likes/dislikes or providing preferences). The extended conversation is C2=[U1,A1,U2]C_2=[U_1,A_1,U_2]; compute Hw(C2)H_w(C_2). The turn-level information gain is (8) IT(A1)=Hw(C1)−Hw(C2).I_T(A_1)\;=\;H_w(C_1)\;-\;H_w(C_2). 3.2.2. Conversation-level entropy reduction We also evaluate a turn by the total reduction achieved over the full conversation unrolled from that turn. We call this conversation-level entropy reduction. Specifically, given C1=[U1]C_1=[U_1], compute Hw(C1)H_w(C_1). Generate an assistant interaction A1A_1. Unroll to a terminal conversation C=[U1,A1,U2,A2,…,Aτ,Uτ]C=[U_1,A_1,U_2,A_2,…,A_τ,U_τ] and compute Hw(C)H_w(C). The conversation-level information gain is (9) IC(A1)=Hw(C1)−Hw(C).I_C(A_1)\;=\;H_w(C_1)\;-\;H_w(C). 3.3. An Illustrative Example We now give an example of the trend in entropy reduction in conversational recommendation. We use an INSPIRED conversation written by human workers (9) and provided in the Supplement (2). After each user response, we sample n=5n=5 lists of top-m=5m=5 movie recommendations and calculate the corresponding entropy. As shown in Figure 1, entropy exhibits a decreasing trend as the conversation progresses and the assistant gathers more information from the user. To verify that this pattern extends beyond one example, we repeat the same experiment over 300 conversations from INSPIRED and average the recommendation entropy at each turn. Figure 2 shows that mean entropy generally decreases as conversations progress, with a weighted trend of −0.040-0.040 bits per turn. Figure 1. Entropy trend for an INSPIRED conversation; see Supplement (2). Line chart showing recommendation entropy generally decreasing as the conversation progresses. Figure 2. Mean recommendation entropy by turn over 300 INSPIRED conversations. The shaded region shows the approximate 95% confidence interval. Line chart showing mean recommendation entropy across 300 INSPIRED conversations. Entropy generally decreases from the first through the tenth turn, with a weighted trend of negative 0.040 bits per turn; a shaded band shows the approximate 95 percent confidence interval. 3.4. Fine-tuning with Entropy Reduction We fine-tune LLMs using the proposed entropy-reduction score as the reward. We use standard SFT and DPO (17) as optimization mechanisms; the proposed component is the entropy-reduction reward used to select SFT examples or define DPO preferences. For SFT, we select training examples with high reward scores. For DPO, among two generations sampled from the recommender, the higher-reward one is preferred. 4. Experiments 4.1. Baselines We begin by introducing the baselines used in the experiments. We do not directly compare with the RL-based methods in (5; 6): they learn separate dialogue policies over restricted multiple-choice or Yes/No action spaces, whereas our setting uses a single LLM for open-ended dialogue and recommendation, making their interfaces and reported metrics not directly comparable. Vanilla. We compare with the vanilla zero-shot recommender (10), i.e., directly using the pre-trained LLM as the recommender. SFT. We also compare with an LLM fine-tuned via supervised fine-tuning (SFT) on the raw training data from the INSPIRED and ReDial datasets. CollabLLM. For each fine-tuning approach, we additionally adopt the CollabLLM framework (21) as a baseline. Its reward combines task-specific reward (RtaskR_task; ground-truth hit for movie recommendation), interactivity (RinteractR_interact; LLM-judge score using the prompt in the Supplement (2)), and token efficiency (RtokR_tok; generated tokens divided by the maximum allowed token budget): (10) R=Rtask+Rinteract− 0.1⋅Rtok,R\;=\;R_task\;+\;R_interact\;-\;0.1\,·\,R_tok, where Rtok=#tokens generatedmax tokens allowedR_tok= \#tokens generatedmax tokens allowed. Table 1. Results on the INSPIRED and ReDial datasets (best per column in bold). INSPIRED Dataset ReDial Dataset Method Hit@1 ↑ Hit@5 ↑ Simulated Conversation Hit ↑ # Turns to Ground Truth ↓ Hit@1 ↑ Hit@5 ↑ Simulated Conversation Hit ↑ # Turns to Ground Truth ↓ Vanilla 1.60±0.21%1.60± 0.21\% 1.94±0.18%1.94± 0.18\% 21.54±1.26%21.54± 1.26\% 3.37 0.78±0.08%0.78± 0.08\% 1.73±0.15%1.73± 0.15\% 25.35±0.38%25.35± 0.38\% 2.94 SFT (Raw) 2.06±0.22%2.06± 0.22\% 3.32±0.14%3.32± 0.14\% 23.90±0.48%23.90± 0.48\% 4.12 1.56±0.19%1.56± 0.19\% 2.96±0.26%2.96± 0.26\% 23.98±0.24%23.98± 0.24\% 2.87 SFT (CollabLLM) 2.68±0.22%2.68± 0.22\% 4.26±0.14%4.26± 0.14\% 25.25±1.65%25.25± 1.65\% 3.36 2.14±0.04%2.14± 0.04\% 5.00±0.09%5.00± 0.09\% 28.70±0.68%28.70± 0.68\% 2.85 SFT (Turn Entropy) 3.00±0.22%3.00± 0.22\% 5.05±0.47%5.05± 0.47\% 26.60±0.48%26.60± 0.48\% 3.12 2.17±0.06%2.17± 0.06\% 5.05±0.08%5.05± 0.08\% 28.94±0.30%28.94± 0.30\% 2.80 SFT (Conv Entropy) 3.00±0.26%3.00± 0.26\% 5.21±0.39%5.21± 0.39\% 26.93±0.48%26.93± 0.48\% 2.94 2.16±0.05%2.16± 0.05\% 5.19±0.07%5.19± 0.07\% 29.31±0.49%29.31± 0.49\% 2.81 DPO (CollabLLM) 3.15±0.34%3.15± 0.34\% 5.09±0.10%5.09± 0.10\% 26.60±1.20%26.60± 1.20\% 3.12 2.22±0.03%2.22± 0.03\% 5.12±0.04%5.12± 0.04\% 30.03±0.35%30.03± 0.35\% 2.86 DPO (Turn Entropy) 3.32±0.12%3.32± 0.12\% 5.21±0.22%5.21± 0.22\% 27.94±0.73%27.94± 0.73\% 3.07 2.24±0.01%2.24± 0.01\% 5.21±0.04%5.21± 0.04\% 31.62±0.36%31.62± 0.36\% 2.75 DPO (Conv Entropy) 3.15±0.12%3.15± 0.12\% 5.05±0.13%5.05± 0.13\% 27.02±0.55%27.02± 0.55\% 3.05 2.18±0.03%2.18± 0.03\% 5.14±0.02%5.14± 0.02\% 32.83±0.31%32.83± 0.31\% 2.74 4.2. Experimental Setup Datasets. We use the INSPIRED (9) and ReDial (14) conversational recommendation datasets, both collected by human workers on Amazon Mechanical Turk. Besides direct supervised fine-tuning, we use their conversations as references for user simulation. INSPIRED contains 801 training and 99 test conversations, each labeled with a ground-truth recommendation that the user accepts. In ReDial, movies are labeled by whether the user likes and has seen them. We use liked but unseen movies as ground truth and filter examples with no ground truth, yielding 8631 training and 1036 test conversations. Simulation. We adapt CollabLLM’s data generation approach (21), in which a user simulator generates conversations with the AI assistant. In this paper, we generate simulated conversations based on INSPIRED and ReDial: Claude Sonnet 4 is prompted with a reference conversation and asked to role-play the user; the partially adapted prompt is in the Supplement (2). The user terminates when (a) the ground-truth movie is recommended, (b) a satisfactory answer is obtained, (c) the assistant is no longer helpful, or (d) a maximum of 5 turns is reached. We generate datasets and test all methods using Llama-3.2-1B-Instruct (8). Turn-level DPO pairs. Following CollabLLM (21), we generate turn-level completion pairs for DPO with Algorithm 1. Algorithm 1 Simulated Pairwise Selection for Conversation Generation 1: User simulator U; AI assistant A; conversation generator G (based on U and A); reward evaluator R; reference conversation CrefC_ref (INSPIRED/ReDial); number of turns T 2: Generated conversation CgenC_gen 3: Cgen←[]C_gen←[\,] ⊳ initial empty conversation 4: for t=1t=1 to T do 5: Ut←U(Cgen,Cref)U_t← U(C_gen,C_ref) ⊳ user simulator emits a query 6: Cgen←Cgen|[Ut]C_gen← C_gen\,\|\,[U_t] ⊳ append 7: At(1)←A(Cgen)A_t^(1)← A(C_gen) ⊳ sample two assistant responses 8: At(2)←A(Cgen)A_t^(2)← A(C_gen) 9: Ct(1)←(Cgen∥[At(1)])C_t^(1) \! (C_gen\,\|\,[A_t^(1)] ) ⊳ full rollout 10: Ct(2)←(Cgen∥[At(2)])C_t^(2) \! (C_gen\,\|\,[A_t^(2)] ) 11: Rt(1)←R(Ct(1))R_t^(1)← R(C_t^(1)); Rt(2)←R(Ct(2))R_t^(2)← R(C_t^(2)) ⊳ evaluation 12: if Rt(1)≥Rt(2)R_t^(1)≥ R_t^(2) then 13: choose At(1)A_t^(1) and reject At(2)A_t^(2) ⊳ for prompt CgenC_gen 14: Cgen←Cgen|[At(1)]C_gen← C_gen\,\|\,[A_t^(1)] 15: else 16: choose At(2)A_t^(2) and reject At(1)A_t^(1) 17: Cgen←Cgen|[At(2)]C_gen← C_gen\,\|\,[A_t^(2)] 18: end if 19: end for Dataset generation uses the CollabLLM reward. The entropy-reduction datasets are generated by relabeling the resulting preference pairs with our entropy-reduction reward. SFT datasets. SFT (Raw) uses the raw conversation data provided in the training set of INSPIRED (801 conversations) and ReDial (8631 conversations). SFT (CollabLLM) selects conversations with high CollabLLM reward scores from the generated DPO dataset. SFT (Turn Entropy) selects conversations with high entropy-reduction reward scores from the generated DPO dataset. The entropy reduction is measured by the difference between the entropy values before and after the current assistant response and the next user response it elicits. SFT (Conv Entropy) also filters by entropy reduction, measured by the difference between the entropy values before the assistant response and at the end of the whole generated conversation. For fair comparison, we keep the top 801 INSPIRED and 8631 ReDial conversations for each filtered SFT variant. Across reward variants, we use the same generated preference pairs for relabeling and match the SFT data budget. Thus, within each training paradigm, the comparisons isolate the effect of the reward used for data selection or preference labeling. Evaluation. For all metrics and baselines, training uses either the INSPIRED/ReDial training set or the generated training set, and testing uses the corresponding test set. Hit@1/Hit@5 follow the tests from 10: each conversation is cut before a ground-truth recommendation, and the recommender is asked to generate a list of 5 movies given the partial conversation; we then calculate and report Hit@1 and Hit@5. A total of 228 INSPIRED and 3552 ReDial partial conversations are evaluated. We also report simulated conversation hit rate by passing each test conversation to a user simulator and checking whether the ground-truth movie is recommended, along with the average number of turns needed to reach that recommendation. This number-of-turns metric reflects whether the recommender generates strategic interactions and efficiently solicits enough information for an accurate recommendation. 4.3. Example of Baseline and Proposed Reward We provide the full generated training example in the Supplement (2). For the same initial comedy-movie query, two assistant responses received similar CollabLLM-style interactivity scores (0.9 vs. 0.8), but turn-entropy reduction revealed a larger gap (1.728 vs. 0.733). While one might intuitively assume that general questions such as asking about slapstick, witty one-liners, or heartfelt comedy would reveal more about a user’s preferences, giving initial recommendations and asking for the user’s opinion can also be effective. Most importantly, we do not impose a predefined, subjective notion of what types of questions are good for information gain; our reward leaves this to the observed entropy reduction rather than a manually defined question style. 4.4. Results Table 1 reports the experimental results for each evaluation metric. Models are fine-tuned in the LoRA setting. Throughout testing, the temperature for the assistant is set to 0.1, and we report the average over three runs for each setting. The entropy-reduction variants show the strongest gains on the main conversational metrics: DPO (Turn Entropy) obtains the best INSPIRED Hit@1, Hit@5, and simulated conversation hit rate, while DPO (Conv Entropy) obtains the best ReDial simulated conversation hit rate and the lowest ReDial turn count. From Table 1, we observe that (1) Direct SFT on the INSPIRED and ReDial datasets can improve hit rate when the assistant is asked to directly generate recommendation lists rather than engage in conversational recommendation, but it does not substantially enhance conversational recommendation ability. This is likely because these datasets are created by human annotators who are not professional recommenders, and their language is often less engaging or proactive than that generated by LLMs. (2) Fine-tuning on simulated data selected via entropy reduction also improves recommendation accuracy, consistent with the finding in (12) that lower recommendation uncertainty is associated with better performance. (3) In the context of conversational recommendation (rather than recommendation list generation) evaluated by simulated conversation hit rate, our proposed entropy-reduction reward outperforms CollabLLM’s reward design—which combines ground-truth hit rate and LLM-judged interactivity—even without access to ground-truth recommendations. (4) Models fine-tuned with the entropy-reduction reward achieve higher conversational efficiency, requiring fewer turns to recommend the ground truth—reflecting effective strategic interaction generation and information gathering. 4.5. Example Conversations Before and After Fine-Tuning We provide example conversations before and after our proposed fine-tuning in the Supplement (2). From these examples, we observe that the fine-tuned model tends to go beyond simply giving recommendations and instead actively asks for user preferences based on the conversation context, thereby improving conversational efficiency. 4.6. Limitations Our evaluation uses automatic recommendation metrics and an LLM-based user simulator rather than human judgments, and we fix m=n=5m=n=5; conversational usefulness and sensitivity to these sampling parameters remain to be validated. We also assume that the base LLM can name candidate items. Dynamic or specialized catalogs would require retrieval, with entropy measured over retrieved rankings. 5. Conclusion and Future Work We introduced an uncertainty-driven approach for multi-turn conversational recommendation that quantifies the assistant’s uncertainty as entropy over sampled recommendation lists and rewards turns that reduce this entropy. This objective aligns interaction design with information gain, avoids reliance on ground-truth targets, and integrates cleanly with SFT and DPO. Empirically, on INSPIRED and ReDial, our fine-tuned models improve recommendation quality and conversational efficiency compared to strong baselines. Future work includes evaluating larger models and datasets, integrating the reward into on-policy RLHF methods such as PPO and GRPO, and combining it with CollabLLM’s reward when ground-truth recommendations are available. The framework can also be extended to retrieval-based conversational recommendation by measuring entropy over rankings within a retrieved recommendation list (12). GenAI Usage Disclosure The authors used generative AI tools for assistance with language polishing and LaTeX drafting. The authors remain responsible for all content, experiments, claims, and citations in this manuscript. References Abdulhai et al. (2023) M. Abdulhai, I. White, C. Snell, C. Sun, J. Hong, Y. Zhai, K. Xu, and S. Levine LMRL gym: benchmarks for multi-turn reinforcement learning with language models. arXiv preprint arXiv:2311.18232. External Links: Link Cited by: §2. Bai et al. (2026) C. S. Bai, D. Li, Z. Liao, S. Sarwar, H. Chen, Y. Chen, C. Yuan, H. Zhang, and Q. Qi Supplement for “ask to be sure: informative interactions for confident multi-turn llm recommendation”. External Links: Link Cited by: Figure 1, Figure 1, §3.3, §4.1, §4.2, §4.3, §4.5. Chen et al. (2025) M. Chen, R. Sun, T. Pfister, and S. Ö. Arık Learning to clarify: multi-turn conversations with action-based contrastive self-training. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Chen et al. (2019) Q. Chen, J. Lin, Y. Zhang, M. Ding, Y. Cen, H. Yang, and J. Tang Towards knowledge-based recommender dialog system. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, p. 1803–1813. External Links: Document, Link Cited by: §1, §1. Deng et al. (2021) Y. Deng, Y. Li, F. Sun, B. Ding, and W. Lam Unified conversational recommendation policy learning via graph-based reinforcement learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, NY, USA, p. 1431–1441. External Links: Document, Link Cited by: §1, §4.1. Du et al. (2025) H. Du, B. Peng, and X. Ning SAPIENT: mastering multi-turn conversational recommendation with strategic planning and Monte Carlo tree search. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, p. 2629–2648. External Links: Link, Document, ISBN 979-8-89176-189-6 Cited by: §1, §4.1. Gao et al. (2025) Z. Gao, W. Zhan, J. D. Chang, G. Swamy, K. Brantley, J. D. Lee, and W. Sun Regressing the relative future: efficient policy optimization for multi-turn rlhf. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Grattafiori et al. (2024) A. Grattafiori et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. External Links: Link Cited by: §4.2. Hayati et al. (2020) S. A. Hayati, D. Kang, Q. Zhu, W. Shi, and Z. Yu INSPIRED: toward sociable recommendation dialog systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, p. 8142–8152. External Links: Document, Link Cited by: §3.3, §4.2. He et al. (2023) Z. He, Z. Xie, R. Jha, H. Steck, D. Liang, Y. Feng, B. P. Majumder, N. Kallus, and J. McAuley Large language models as zero-shot conversational recommenders. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM ’23, New York, NY, USA, p. 720–730. External Links: ISBN 9798400701245, Link, Document Cited by: §1, §4.1, §4.2. He et al. (2025) Z. He, Z. Xie, H. Steck, D. Liang, R. Jha, N. Kallus, and J. McAuley Reindex-then-adapt: improving large language models for conversational recommendation. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, WSDM ’25, New York, NY, USA, p. 866–875. External Links: ISBN 9798400713293, Link, Document Cited by: §1, §2. Kweon et al. (2025) W. Kweon, S. Jang, S. Kang, and H. Yu Uncertainty quantification and decomposition for LLM-based recommendation. In Proceedings of the ACM on Web Conference 2025, New York, NY, USA, p. 4889–4901. External Links: Document, Link Cited by: §1, §3.1, item (2), §5. Laban et al. (2026) P. Laban, H. Hayashi, Y. Zhou, and J. Neville LLMs get lost in multi-turn conversation. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Li et al. (2018) R. Li, S. Ebrahimi Kahou, H. Schulz, V. Michalski, L. Charlin, and C. Pal Towards deep conversational recommendations. In Advances in Neural Information Processing Systems, Vol. 31. External Links: Link Cited by: §1, §1, §4.2. Liang et al. (2025) Y. Liang, L. Yang, C. Wang, X. Xu, P. S. Yu, and K. Shu Taxonomy-guided zero-shot recommendations with llms. In Proceedings of the 31st International Conference on Computational Linguistics (COLING), Abu Dhabi, UAE, p. 1520–1530. External Links: Link Cited by: §2. Qiu et al. (2026) L. Qiu, F. Sha, K. Allen, Y. Kim, T. Linzen, and S. van Steenkiste Bayesian teaching enables probabilistic reasoning in large language models. Nature Communications 17 (1), p. 1238. External Links: Document, Link Cited by: §2. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, Vol. 36, p. 53728–53741. External Links: Document, Link Cited by: §3.4. Shani et al. (2024) L. Shani, A. Rosenberg, A. Cassel, O. Lang, D. Calandriello, A. Zipori, H. Noga, O. Keller, B. Piot, I. Szpektor, A. Hassidim, Y. Matias, and R. Munos Multi-turn reinforcement learning with preference human feedback. In Advances in Neural Information Processing Systems, Vol. 37, p. 118953–118993. External Links: Document, Link Cited by: §2. Sun and Zhang (2018) Y. Sun and Y. Zhang Conversational recommender system. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, New York, NY, USA, p. 235–244. External Links: ISBN 9781450356572, Link, Document Cited by: §1. Wang et al. (2022) X. Wang, K. Zhou, J. Wen, and W. X. Zhao Towards unified conversational recommender systems via knowledge-enhanced prompt learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, New York, NY, USA, p. 1929–1937. External Links: ISBN 9781450393850, Link, Document Cited by: §1. Wu et al. (2025) S. Wu, M. Galley, B. Peng, H. Cheng, G. Li, Y. Dou, W. Cai, J. Zou, J. Leskovec, and J. Gao CollabLLM: from passive responders to active collaborators. In Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, p. 67260–67283. External Links: Link Cited by: §1, §2, §3.2, §4.1, §4.2, §4.2. Zhang et al. (2025) Y. Zhang, F. Feng, J. Zhang, K. Bao, Q. Wang, and X. He CoLLM: integrating collaborative embeddings into large language models for recommendation. IEEE Transactions on Knowledge and Data Engineering 37 (5), p. 2329–2340. External Links: Document, Link Cited by: §1, §2. Zhao et al. (2025) S. Zhao, M. Hong, Y. Liu, D. Hazarika, and K. Lin Do llms recognize your preferences? evaluating personalized preference following in llms. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Zheng et al. (2024) B. Zheng, Y. Hou, H. Lu, Y. Chen, W. X. Zhao, M. Chen, and J. Wen Adapting large language models by integrating collaborative semantics for recommendation. In Proceedings of the IEEE International Conference on Data Engineering (ICDE), p. 1435–1448. External Links: Document, Link Cited by: §2. Zhou et al. (2020) K. Zhou, W. X. Zhao, S. Bian, Y. Zhou, J. Wen, and J. Yu Improving conversational recommender systems via knowledge graph based semantic fusion. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, New York, NY, USA, p. 1006–1014. External Links: Document, Link Cited by: §1, §1. Zhou et al. (2024) Y. Zhou, A. Zanette, J. Pan, S. Levine, and A. Kumar ArCHer: training language model agents via hierarchical multi-turn rl. In Proceedings of the 41st International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 235, p. 62178–62209. External Links: Link Cited by: §2. Zhu et al. (2025) Y. Zhu, C. Wan, H. Steck, D. Liang, Y. Feng, N. Kallus, and J. Li Collaborative retrieval for large language model-based conversational recommender systems. In Proceedings of the ACM on Web Conference 2025, New York, NY, USA, p. 3323–3334. External Links: Document, Link Cited by: §1, §2. Zhu et al. (2024) Y. Zhu, L. Wu, Q. Guo, L. Hong, and J. Li Collaborative large language model for recommender systems. In Proceedings of the ACM Web Conference 2024, New York, NY, USA, p. 3162–3172. External Links: Document, Link Cited by: §1.