Paper deep dive
SimInter: Evidence-Grounded User Simulation with Interview-Based Profile Enrichment for Conversational Recommender Systems
Erfan Shahabi, Amirhossein Roshandel, Mohammad Sadegh Sadeghi, Azadeh Shakery
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Conversational Recommender Systems (CRSs) require reliable user simulators for cost-effective evaluation. LLM-based simulators have made realistic simulation possible but typically condition on profiles derived from explicit ratings, demographics, and static sources capturing surface preferences while omitting implicit behavioral characteristics such as decision-making style, aspect-level preferences, and domain expertise. We propose SimInter, a pipeline that elicits these attributes via synthetic interviews between two LLM agents, grounded in the user's statistical profile and historical conversational evidence. Before profile enrichment, an audit agent explicitly flags potential fabrication and stereotyping at the turn level. We instantiate and evaluate SimInter on movie recommendation. We simulate 300 ReDial users and, on a matched subset of 100 users, compare SimInter with a statistics-only baseline: SimInter significantly increases the diversity of expressed preference aspects (filtered aspect entropy: 1.619 vs. 0.859; paired d = 0.96, Holm-adjusted p < 0.001). This gain persists when the simulator converses with two trained recommenders, KBRD and UniCRS, under two independent evaluation judges (+0.28 to +0.56 nats; all p < 0.001). Human evaluation finds 94.3% of enriched-attribute claims supported or partially supported, while human judgments agree with 95.2% of the auditor's binary filtering decisions on a stratified sample.
Tags
Links
- Source: https://arxiv.org/abs/2610.18308v1
- Canonical: https://arxiv.org/abs/2610.18308v1
Trouble viewing inline? Open PDF directly â
Full Text
103,963 characters extracted from source content.
Expand or collapse full text
SimInter: Evidence-Grounded User Simulation with Interview-Based Profile Enrichment for Conversational Recommender Systems Erfan Shahabi Email: erfanshahabi@ut.ac.ir Amirhossein Roshandel Email: roshandel.amir@ut.ac.ir Mohammad Sadegh Sadeghi Affiliation: School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran Email: mhmssadeghi74@ut.ac.ir Azadeh Shakery Affiliation: School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran Affiliation: Institute for Research in Fundamental Sciences (IPM), Tehran, Iran Email: shakery@ut.ac.ir Abstract Conversational Recommender Systems (CRSs) require reliable user simulators for cost-effective evaluation. LLM-based simulators have made realistic simulation possible but typically condition on profiles derived from explicit ratings, demographics, and static sources capturing surface preferences while omitting implicit behavioral characteristics such as decision-making style, aspect-level preferences, and domain expertise. We propose SimInter, a pipeline that elicits these attributes via synthetic interviews between two LLM agents, grounded in the userâs statistical profile and historical conversational evidence. Before profile enrichment, an audit agent explicitly flags potential fabrication and stereotyping at the turn level. We instantiate and evaluate SimInter on movie recommendation. We simulate 300300 ReDial users and, on a matched subset of 100100 users, compare SimInter with a statistics-only baseline: SimInter significantly increases the diversity of expressed preference aspects (filtered aspect entropy: 1.6191.619 vs. 0.8590.859; paired d=0.96d=0.96, Holm-adjusted p<0.001p<0.001). This gain persists when the simulator converses with two trained recommenders, KBRD and UniCRS, under two independent evaluation judges (+0.28+0.28 to +0.56+0.56 nats; all p<0.001p<0.001). Human evaluation finds 94.3%94.3\% of enriched-attribute claims supported or partially supported, while human judgments agree with 95.2%95.2\% of the auditorâs binary filtering decisions on a stratified sample. **footnotetext: Equal contribution, ordered randomly. 1 Introduction Conversational Recommender Systems (CRSs) have become a primary interface for content discovery Fu et al. (2020); Kim et al. (2012), but their evaluation remains difficult: studies with real users are expensive, slow, and rarely reproducible, with outcomes depending on the specific sample recruited Zhu et al. (2024b). User simulation has emerged as a complementary methodology in which artificial users interact with a CRS in place of humans, enabling controlled, repeatable experimentation at scale Balog and Zhai (2025); Zhu et al. (2024a). Early simulators were rule-based or agenda-based RuĂ et al. (2012); Afzali et al. (2023), offering controllability at the cost of behavioral realism. Large language models (LLMs) have changed this picture, but LLM-based simulators still exhibit characteristic failure modes: repetitive responses, narrow choice ranges, binary reactions, and drift toward generic âaverage userâ behavior Yoon et al. (2024); Zhu et al. (2024b). We argue that an important source of these limitations is the user profile that conditions the simulator. The profile bottleneck. Most LLM-based simulators condition generation on profiles built from observable signals: ratings, interaction histories, and demographic attributes. Such profiles capture what a user has consumed but leave largely unspecified how the user makes choices: whether they satisfice or maximize, whether their choices are driven by mood or social context, and how willing they are to accept unfamiliar recommendations. These behavioral dispositions strongly shape conversational behavior yet are precisely what ratings-based profiles omit Tigunova et al. (2019); Yoon et al. (2024). A natural alternativeâprompting an LLM to infer behavioral attributes from a static profileâhas a known failure mode: LLMs tend to produce stereotyped inferences, inflating apparent profile richness while reducing individual fidelity. The same risk applies to self-play dialogue, where two agents can amplify rather than reveal latent attributes, fabricating plausible-sounding traits not grounded in actual behavior. Our approach. We propose SimInter, which derives behavioral attributes from synthetic dialogue while explicitly defending against fabrication via four design choices. (i) Statistical grounding: we first compute behavioral statistics (acceptance rate, reference style, genre distribution, exploration ratio) from the userâs interaction history; the synthetic dialogue is then conditioned on these statistics and the userâs historical conversational evidence. (i) Interview rather than peer dialogue: generation is cast as an asymmetric interview, with the interviewee anchored to the userâs profile and the interviewer as a neutral elicitor probing specific dimensions Tigunova et al. (2019), preventing mutual reinforcement of speculative content. (i) Turn-level auditing: an Audit Agent flags fabrication, history contradictions, and caricature (generic persona templates and invented rationalizations). Depending on severity, flagged turns are excluded from profile enrichment, while structurally compromised dialogues trigger regeneration. (iv) Evidence-linked enrichment: an LLM agent enriches the userâs statistical profile with attributes extracted from the audited interview, with each categorical or scored attribute paired with a supporting evidence quote; human annotators verify that 94%94\% of these enriched claims are supported by the interview. Contributions. (1) We identify the profile bottleneck in LLM-based user simulation. (2) We propose SimInter, combining statistical grounding, asymmetric interview dialogue, turn-level auditing, and evidence-linked profile enrichment; the audit uses a flag taxonomy that distinguishes unsupported claims from invented rationalizations. (3) We evaluate SimInter on 300300 ReDial users (1,2001,200 simulated conversations), with matched 100100-user comparisons against statistics-only and CSHI Zhu et al. (2024a) baselines across four complementary tracks (item diversity, population-level preference alignment, aspect richness, and linguistic variety). We find a significant increase in aspect richness over statistics-only profiles (T3, d=+0.96d=+0.96, Holm-adjusted p<0.001p<0.001). 2 Related Work User simulation provides a scalable alternative to human evaluation of conversational recommender systems (CRSs) Balog and Zhai (2025); Zhu et al. (2024a); Zhang and Balog (2020); Bernard and Balog (2025). Earlier approaches relied on agenda-based control, later extended with neural generation in frameworks such as GenTUS Afzali et al. (2023); Lin et al. (2022). LLM-based frameworks, including iEvaLM, enable more flexible interactions but raise concerns about behavioral realism, data leakage, and controllability Wang et al. (2023); Yoon et al. (2024); Zhu et al. (2024b). Recent approaches incorporate richer user models. RecUserSim combines bounded-rationality principles with memory tracking Chen et al. (2025a); PerCRS uses assigned personality traits Zhao et al. (2025), while PUB infers Big-Five traits from behavioral logs Ma et al. (2025). PEARL derives persona knowledge from user reviews Kim et al. (2024), and CSHI offers controllable, target-conditioned simulation Zhu et al. (2024a). Other work investigates LLM-based and user-centric CRS evaluation and parameterized behavioral traits Chen et al. (2025b); Jin et al. (2024); SekuliÄ et al. (2024). SimInter differs by eliciting implicit behavioral attributes through synthetic interviews grounded in historical conversational evidence. Unlike predefined trait configurations or review-derived personas, SimInter uses user-specific dialogue and audits potentially fabricated or stereotyped content before profile enrichment. Table 1 compares these frameworks in terms of datasets, profile sources, available ground truth, target conditioning, reliability mechanisms, evaluation objectives, and data requirements. Notably, while CSHI addresses target-item leakage and PerCRS uses knowledge-graph grounding, SimInter audits interview-derived claims against historical user evidence before profile enrichment. Figure 1: Overview of the SimInter pipeline. Dimension CSHI Zhu et al. (2024a) PerCRS Zhao et al. (2025) PUB Ma et al. (2025) SimInter (ours) Dataset ReDial, OpenDialKG, MovieLens DuRecDial 2.0 Amazon Reviews ReDial Conversational Yes Yes No Yes Profile source History; generated or editable profiles History; injected Big Five Behavioral logs; inferred Big Five Statistics; audited interviews Profile ground truth Target items; held-out interactions Injected personality traits Held-out real interactions Historical preferences; no trait labels Target-item conditioning Yes Yes N/A No Fabrication control Target-leakage prevention KG-based factual grounding Not reported Evidence-based turn-level auditing Evaluation target CRS recommendation performance Trait consistency; CRS outcomes Behavioral fidelity; RS ranking Simulator behavior; human evaluation Data requirements Interaction history; item information Preferences; traits; KG Ratings; reviews; timestamps; metadata Dialogues; historical preferences Table 1: Conceptual comparison of user simulation frameworks. Ground truth denotes available evaluation references, not necessarily verified personality traits. Fabrication-control mechanisms differ in scope; N/A indicates not applicable. 3 Method Figure 1 illustrates the SimInter pipeline. The method consists of six stages: (1) constructing basic user profiles from historical interactions, (2) extracting behavioral statistics from these profiles, (3) generating synthetic interviews between two LLM-based agents, (4) auditing the generated conversations with an audit agent, (5) extracting implicit user attributes and enriching the profiles, and (6) simulating users interacting with a CRS using their enriched profiles. We instantiate the LLM-based components with Gemini 2.5 Flash Comanici et al. (2025) for interview generation and user simulation, Claude Haiku 4.5 Anthropic (2025) for auditing, Tencent Hy3 Tencent Hy Team (2026) for enrichment, and GPT-4o-mini Achiam et al. (2023) as the CRS the simulated user interacts with. Models were chosen for availability and cost; where possible, we drew components from different model families to reduce shared bias between generator, auditor, and judges. All model choices are flexible. 3.1 Basic User Profiles and Behavioral Statistics Stage 1 (Basic User Profiles). We construct basic user profiles from ReDial Li et al. (2018), treating each crowdworker as a user across both seeker and recommender roles. Of the 1,112 workers in the 11,34811,348 ReDial dialogues, we retain the 618 with at least two conversations. Each profile contains observed movie preferences (liked, disliked, and implicitly disliked movies, derived from ReDialâs post-conversation questionnaires) and the userâs historical conversations. Stage 2 (Behavioral Statistic Extraction). We augment these profiles with rule-based behavioral statistics derived from historical interactions: acceptance and rejection rates, reference style (genre, mood, actor, director, year), genre and decade distributions, exploration ratio, and popularity score. Together these form a behavioral fingerprint that grounds the synthetic interview. Figure 2 shows an example. Of the 618 profiles, 543 meet our eligibility criteria, requiring at least one seeker-role conversation and a nonzero acceptance or implicit-rejection signal. We randomly sample 300 users from this eligible set for the full SimInter pipeline, and randomly select a subset of 100 of these users for matched ablation experiments and comparison with CSHI (§4.1). Figure 2: Example ReDial user profile after Stage 2, combining historical preferences with derived behavioral statistics. 3.2 Synthetic Interview Generation Stage 3 (Synthetic Interview Generation). Conversations often contain implicit information that provides deeper insights into users Tigunova et al. (2019); Torbati et al. (2021). We exploit LLMsâ role-playing capabilities Wang et al. (2024); Chen et al. (2024) to simulate interviews between two agents: an interviewee anchored to the userâs profile and an interviewer that elicits it. For each of the 300 sampled users, the interviewee agent receives the userâs statistical profile and one historical ReDial conversation in which the user acts as a seeker. The interviewer, in contrast, has no access to the profile or the ReDial history; it sees only the ongoing interview, so that it cannot steer the interviewee toward known preferences. It opens with one of four predefined open-ended questions, chosen at random (e.g., âWalk me through the last movie you really enjoyed â what was it and what stuck with you?â; all four are listed in Appendix J.1), and dynamically adapts its follow-up questions to the intervieweeâs responses. Each interview comprises an initial six-turn elicitation phase followed by four clarification turns, for a total of 10 turns. Figure 3 shows an example; full system prompts are in Appendix J. Figure 3: Example simulated interview between interviewer and interviewee agents. 3.3 Conversation Auditing Stage 4 (Conversation Auditing). An Audit Agent checks each interviewee turn against the userâs own evidence, namely the statistical profile and the historical ReDial conversation, for three types of issues. Fabrication covers specific claims that are neither stated in nor plausibly inferable from this evidence, such as invented viewing experiences or personal anecdotes. Caricature covers stereotyped persona performances and invented rationalizations: plausible-sounding justifications for preferences that the profile does not attest, or recognizable persona archetypes it does not support. History contradiction covers statements that conflict with the userâs liked and disliked movies or with their earlier ReDial conversation. In addition, deterministic structural checks verify the number of turns, the alternation of speaker roles, and the absence of empty turns. The auditor assigns each flag a severity (low, medium, or high), and a deterministic rule derives the dialogue-level decision. Any structural error or â„3â„ 3 high-severity flags trigger regenerate (up to two retries, with a corrective nudge appended to the interviewee prompt). At least one high-severity or â„3â„ 3 medium-severity flags trigger partial: the interview is retained, but its high- and medium-severity turns are excluded from enrichment. All other interviews are accepted (accept). For example, in one interview, the interviewee claimed to have enjoyed Blue Streak, although the userâs ReDial conversation indicated that they had not seen it. In another, the interviewee dismissed the action in Justice League as hard to care about, although the film is on the userâs liked list. The auditor flagged both turns, so they could be excluded before profile enrichment. Full auditor prompts, flag definitions, and representative examples are in Appendix B. 3.4 Profile Enrichment Stage 5 (Profile Enrichment). From audited interviews, we extract six implicit user attributes grounded in prior user-modeling literature: aspect-level preferences Radlinski et al. (2019), pickiness Yoon et al. (2024), decision-making style Scott and Bruce (1995); Hayati et al. (2020), viewing motivation Oliver and Raney (2011), novelty preference Tang and Liao (2022), and domain expertise Kostric et al. (2025). Each attribute reveals behavior not observable in the static profile: aspect preferences organize preferences along Movie/Genre/Person/Other anchors; pickiness captures selectivity; decision-making style ranges across rational, intuitive, dependent, avoidant, and spontaneous; viewing motivation combines a hedonic-eudaimonic score (1â5) with a primary category (escapism, hedonic, eudaimonic, social, catharsis, arousal); novelty preference scores openness to unfamiliar content (1â5); and domain expertise classifies users as novice, intermediate, or cinephile. Full attribute schemas, evidence-extraction prompts, and an example enriched profile are in Appendix H. The inferred attributes and derived pickiness are combined with the Stage 2 statistical profile to form the enriched profile that conditions user simulation in Stage 6. 3.5 User Simulation Stage 6 (User Simulation). The enriched profile of each user conditions Gemini 2.5 Flash Comanici et al. (2025) to simulate that userâs behavior in conversation with a CRS powered by GPT-4o-mini Achiam et al. (2023). Each user is simulated in four conversations varying the opening strategy across aspect, movie, mood, and expertise prompts. After each CRS turn, starting from the fourth, an LLM termination judge classifies the last exchange as true acceptance (the user explicitly commits to a specific recommendation), resigned acceptance, explicit give-up, other, or continue; any label other than continue ends the conversation, which is otherwise capped at eight CRS turns. 4 Evaluation We evaluate SimInter on ReDial using 300 users in the full pipeline and a matched subset of 100 users for comparisons against two baselines: StatisticalOnly and CSHI. We ask four questions: do enriched users (T1) mention a more diverse set of items? (T2) express preferences correlated with population-level movie ratings? (T3) discuss movies along richer behavioral aspects? (T4) differ in surface linguistic variety? We additionally conduct a targeted human evaluation (§4.6) and report a separate cross-CRS robustness experiment with two trained ReDial recommenders, KBRD Chen et al. (2019) and UniCRS Wang et al. (2022) (Appendix E). Our evaluation focuses on profile-conditioned simulated user behavior, rather than the recommendation accuracy of the underlying CRS models. 4.1 Experimental Setup Full represents the complete SimInter pipeline, combining behavioral statistics, audited interviews, and enriched user attributes. StatisticalOnly uses only the behavioral statistics, without interview-based profile enrichment. We additionally compare against CSHI, an existing user-simulation baseline that follows a target-conditioned dialogue protocol. We run Full on 300 ReDial users, generating four conversations per user (1,200 conversations). For matched comparisons, we select a subset of 100 users and evaluate StatisticalOnly and CSHI on the same users, generating 400 conversations per baseline. All comparisons use the corresponding 100-user subset of Full. We report descriptive metrics for the full 300-user cohort and conduct paired comparisons on the matched 100-user subset. For each comparison, we compute user-level mean differences with bootstrap 95% confidence intervals (20,000 paired resamples), two-sided paired sign-flip tests (50,000 randomizations), and paired Cohenâs dzd_z Cohen (2013). We apply Holmâs correction Holm (1979) across the original family of 24 paired metric tests, using a significance level of α=0.05α=0.05. T2 population-level correlations are reported descriptively. 4.2 Evaluation Tracks T1 â Item diversity. For each user, Shannon entropy of the movie-mention distribution across their four conversations. Higher entropy indicates broader coverage, reflecting whether the simulator escapes the âaverage userâ attractor Zhu et al. (2024b). We also report JensenâShannon divergence (JSD) against the movie-mention distribution of ReDial seekers. T2 â Popularity alignment. For movies the user rates or reacts to in conversation, we correlate the simulatorâs expressed preference with the movieâs mean MovieLens-25M rating Harper and Konstan (2015) (Pearson and Spearman). This tests whether stated preferences track population-level signal. T3 â Aspect richness. Using Claude Haiku 4.5 Anthropic (2025) as an aspect-level judge Liu et al. (2023); Zheng et al. (2023), we extract (aspect, sentiment) pairs from the userâs sentences about each movie discussed in the conversation. After filtering generic placeholders (e.g., movie, film, story), we compute per-user Shannon entropy over remaining aspects and over the sentiment distribution. Higher aspect entropy indicates the simulator articulates preferences along a wider range of behavioral dimensions rather than collapsing to a single template. T4 â Linguistic surface variety. Per-user metrics over user turns: corpus-level type-token ratio (TTR), n-gram monotony (mean Jaccard overlap of bigrams/trigrams between turn pairs), and SBERT semantic diversity Reimers and Gurevych (2019). TTR and n-gram monotony measure lexical variety, SBERT diversity measures the semantic spread of requests, and T3 captures semantic-content variety; these can move oppositely. 4.3 Results Table 2 summarizes the matched 100-user comparisons. Relative to StatisticalOnly, Full achieves higher T3 aspect entropy (Î=+0.760 =+0.760) but lower T1 item diversity (Î=â0.375 =-0.375), both significant after Holm correction. Full also exceeds CSHI on T1, T3, and T4. T2 correlations are reported descriptively. Track Metric Full StatOnly CSHI ÎFâS _F-S ÎFâC _F-C T1 Item entropy 2.034 2.409 1.454 â0.375â-0.375 +0.588â+0.588 T1 JSD vs. ReDial 0.681 0.676 0.682 +0.005â+0.005 â0.001-0.001 T2 Population Pearson r 0.171 0.133 0.074 â â T3 Aspect entropy (filtered) 1.619 0.859 0.194 +0.760â+0.760 +1.425â+1.425 T3 Aspect entropy (unfiltered) 1.857 1.490 0.694 +0.366â+0.366 +1.163â+1.163 T4 SBERT diversity (pairwise) 0.532 0.500 0.345 +0.032+0.032 +0.186â+0.186 T4 SBERT diversity (centroid) 0.226 0.211 0.140 +0.016+0.016 +0.086â+0.086 Table 2: Evaluation results on the matched 100-user subset. StatOnly denotes StatisticalOnly. ÎFâS _F-S and ÎFâC _F-C denote paired mean differences between Full and StatisticalOnly and CSHI, respectively. â indicates significance after Holm correction (padj<0.05p_adj<0.05). CSHI T1 is defined for 94 users (six mention a single movie). T2 correlations are descriptive and computed over condition-specific movie sets. T1 â Item diversity. Full exhibits a more concentrated movie-mention distribution than StatisticalOnly (padj<0.001p_adj<0.001), and a marginally higher but significant JSD from ReDial (0.6810.681 vs. 0.6760.676). These results characterize the breadth of movie mentions rather than the fidelity of simulated preferences. T2 â Popularity alignment. In the full 300-user evaluation, SimInter exhibits a moderate positive correlation between simulated movie preferences and MovieLens-25M mean ratings (Pearson r=0.347r=0.347, n=275n=275 movies). This suggests that the preferences expressed in simulated dialogue capture meaningful population-level rating patterns. On the matched 100-user subset, correlations remain positive for Full (r=0.171r=0.171) and StatisticalOnly (r=0.133r=0.133). As the movie sets differ across conditions, these estimates are descriptive rather than paired comparisons and do not establish individual-level preference fidelity. Further diagnostics are provided in Appendix A. T3 â Aspect richness. One of the strongest findings is the substantial increase in the diversity of movie-related aspects expressed by enriched users. Across the full 300-user cohort, Full achieves a mean aspect entropy of 1.6061.606. On the matched 100-user subset, Full substantially exceeds StatisticalOnly (1.6191.619 vs. 0.8590.859; Î=+0.760 =+0.760, dz=0.955d_z=0.955, padj<0.001p_adj<0.001). These findings highlight the contribution of interview-based profile enrichment to richer aspect-level preference expression. Figure 4: Effect of interview-based profile enrichment on filtered T3 aspect entropy across 100 matched users. (a) Per-user paired differences between Full and StatisticalOnly; the dashed line indicates the mean difference and the solid line indicates zero. (b) Percentage of users with zero aspect entropy under both conditions. Figure 4 shows that Full achieves higher filtered aspect entropy than StatisticalOnly for 81 of the 100 matched users (15 lower, 4 unchanged). Moreover, only 5% of users under Full exhibit zero aspect entropy, compared with 41% under StatisticalOnly. A concrete example illustrates how enrichment shapes preference expression. User 468âs enriched profile captures preferences for relentless pacing and action choreography. In a matched mood-strategy opening, the Full simulator requests âan action thriller with relentless pacing and some genuinely tight choreography,â whereas StatisticalOnly asks for âsome intense actionâ with âa good crime or thriller plot.â References to pacing and choreography recur across all four Full openings, illustrating how enriched profile attributes translate into more specific movie-related requests. Additional qualitative examples and aspect distributions are provided in Appendix D. T4 â Linguistic surface variety. Full shows slightly higher SBERT semantic diversity than StatisticalOnly (0.5320.532 vs. 0.5000.500; dz=+0.25d_z=+0.25), but the difference is not significant (padj=0.11p_adj=0.11). Conversely, Full exhibits lower corpus TTR (0.6740.674 vs. 0.7280.728; dz=â0.68d_z=-0.68) and higher n-gram monotony (0.0280.028 vs. 0.0190.019; both uncorrected p<0.001p<0.001). This lexical trade-off is consistent with the repetition of profile-specific expressions across conversations, a known issue in profile-conditioned generation SekuliÄ et al. (2024). For example, User 468 repeatedly uses ârelentlessâ across three of four conversations, reflecting vocabulary from the enriched profile. Thus, T3 and T4 capture complementary properties: enrichment can increase aspect richness and semantic diversity while narrowing lexical expression. These distinctions matter when diagnosing CRS breakdowns Bernard and Balog (2024), particularly when varied user phrasing is important. Comparison with CSHI. Under its target-conditioned protocol with the same GPT-4o-mini CRS, CSHI produces markedly less diverse simulated behavior than Full: aspect entropy is 0.1940.194 vs. 1.6191.619 (dz=1.85d_z=1.85), item entropy 1.4541.454 vs. 2.0342.034 (dz=1.13d_z=1.13), and SBERT diversity 0.3450.345 vs. 0.5320.532 (dz=1.30d_z=1.30; all padj<0.001p_adj<0.001), while JSD from ReDial does not differ. CSHI requests are also lexically repetitive (n-gram monotony 0.3000.300 vs. 0.0280.028), and its population correlation is weak (r=0.074r=0.074, 8888 movies). Because CSHI steers each conversation toward a predefined target item, these differences reflect both profile content and protocol, not profile enrichment alone. 4.4 Auditor Behavior The Audit Agent processed 300 interviews: accept (209209, 70%70\%), partial (9191, 30%30\%), regenerate (00). Only one interview reached the regenerate threshold; the regenerated interview received partial. All other corrective action was therefore turn-level exclusion under partial, removing 196196 interviewee turns from enrichment. The auditor raised 676676 flags (372372 caricature, 173173 history contradiction, 131131 fabrication); the full taxonomy and example dialogues are in Appendix B. Three observations follow. First, the dominant issue is not fabrication but caricature (372/676372/676 flags, 55%55\%), mostly invented rationalizations. For example, one interviewee explained a dislike of The Hangover and Superbad, both on the userâs disliked list, by calling them âloud and obnoxious,â although the user likes comparably broad comedies; the dislike is attested, but the reason is not. Second, the human evaluation shows a matching pattern (§4.6): decision-making style, which has no quantitative anchor in the profile, is the least supported attribute (21.4%21.4\% Unsupported; 28.6%28.6\% Evidence-grounded), whereas pickiness, derived from the rejection rate, is among the best supported (97.0%97.0\%); this correspondence offers limited, indirect evidence of audit calibration. Third, the absence of final regenerate decisions indicates that gross structural failure is rare with capable LLMs at our interview scale; the dominant failure mode is local, which makes turn-level exclusion an appropriate response. 4.5 Human Evaluation of Audit Decisions We evaluated Stage 4 auditing on 105105 synthetic interviewee turns, with one turn per historical ReDial user, sampled equally from the 3535 excluded, 3535 flagged-but-retained, and 3535 unflagged audit outcomes. Three co-author annotators independently examined each synthetic response alongside evidence from the corresponding userâs original ReDial conversations, without seeing the auditorâs decisions, flags, or rationales. They assigned one of three labels: Excluded (a serious, evidence-supported issue warrants removing the turn), FlaggedâRetained (a meaningful issue exists but does not warrant removal), or Unflagged (no meaningful issue is identified). The annotators agreed unanimously on 70/10570/105 turns (66.7%66.7\%; Fleissâ Îș=0.664Îș=0.664). Using majority human judgments as the reference, the auditor achieved 86.7% three-class accuracy (91/10591/105), with agreement on 31/3531/35 excluded, 31/3531/35 flagged-but-retained, and 29/3529/35 unflagged turns. Collapsing both retained classes yielded 95.2% binary filtering accuracy (100/105100/105): four auditor-excluded turns were judged retainable and one auditor-retained turn was judged excludable. These results suggest that Stage 4 can distinguish concerns that warrant removal from those that need not discard an otherwise usable response. The figures are human-referenced accuracies on this balanced sample, not prevalence-adjusted estimates over all audited turns; annotation protocol, disagreements, and limitations are detailed in Appendix G. 4.6 Human Evaluation of Enriched Attributes Three annotators independently evaluated enriched- attribute claims from Full profiles. We report 105 claims from 45 users rated by all three annotators (315 annotations). Using behavioral profiles and interview transcripts, annotators assessed support, evidence grounding, consistency, usefulness, overreach, and confidence (Appendix F). Results. By majority vote, 94.3% of claims were Supported or Partially supported (41.0% strictly Supported), and 5.7% were Unsupported or Contradicted. At the annotation level, 1.3% were Contradicted; mean usefulness and overreach were 3.61/5 and 1.70/5. Strict support was highest for pickiness (72.7%) and liked aspects (75.0%) while decision-making style had the highest Unsupported/Contradicted rate (21.4%; Table 3). Inter-annotator agreement. Pairwise quadratic-weighted Cohenâs Îș ranged from 0.40 to 0.52, with 97â98% within-one-rank agreement. For binary support, Gwetâs AC1 was 0.89, while Fleissâ Îș was lower under label imbalance. Disagreements mainly involved Supported versus Partially supported (Appendix F). Attribute n Support Grounded Viewing motivation 51 100.0% 47.1% Aspect prefs. (liked) 60 100.0% 81.7% Aspect prefs. (descr.) 24 100.0% 91.7% Pickiness 66 97.0% 65.2% Domain expertise 36 88.9% 41.7% Novelty preference 15 86.7% 46.7% Aspect prefs. (disliked) 21 81.0% 76.2% Decision-making style 42 78.6% 28.6% All 315 93.3% 59.7% Table 3: Human-evaluation support rates by attribute. Support = Supported + Partially supported; Grounded = Evidence-grounded. n is number of annotations (each claim has three). 4.7 Cross-CRS Robustness We evaluated T3 robustness on 120 users with KBRD and UniCRS under two independent judges. After audit filtering, the paired samples contained 111 KBRD and 80 UniCRS users. The aspect-entropy improvement persisted for KBRD (Î=+0.265 =+0.265 and +0.280+0.280) and UniCRS (Î=+0.330 =+0.330 and +0.369+0.369) under GPT-4o-mini and DeepSeek-chat, respectively (all p<0.001p<0.001). Including all technically completed conversations also yielded positive effects (+0.275+0.275 to +0.560+0.560; all p<0.001p<0.001). These results support T3 robustness across backends and judges, but do not establish overall simulator fidelity. Full details appear in Appendix E. 5 Conclusion We presented SimInter, a pipeline for LLM-based user simulation that elicits qualitative behavioral attributes through structured synthetic interviews and guards against fabrication via explicit turn-level auditing. On 300300 ReDial users, simulators conditioned on enriched profiles discuss movies along substantially richer behavioral aspects than a statistics-only baseline (dz=0.96d_z=0.96, padj<0.001p_adj<0.001; zero-aspect users drop from 41%41\% to 5%5\%); pooled popularity alignment with MovieLens-25M reaches r=0.347r=0.347; and 94%94\% of enriched-attribute claims are human-supported. Enrichment also concentrates item mentions and reduces lexical variety, consistent with profile-conditioned generation narrowing expression toward profile vocabulary. User-simulation quality in CRS is thus multidimensional: the goal is not to maximize every diversity metric, but to gain more on dimensions that matter for CRS evaluation than is lost on those that do not. Limitations Evidence grounding and audit reliability. SimInter grounds enriched attributes in historical user evidence, while independent validation against real user behavior remains an important direction for future work. Human evaluation assesses evidence support rather than direct behavioral fidelity. Moreover, audit decisions involve uncertainty, particularly when distinguishing unsupported rationalizations from legitimate preference elaboration. Automatically retrieved evidence and differences in annotator judgments further limit conclusions about audit reliability. Evaluation scope and generalizability. Our ablations do not include direct LLM-based attribute inference without interviewing, leaving the interviewâs independent contribution unresolved. Sparse and imbalanced preference evidence limits individual-level alignment evaluation. Enrichment also involves trade-offs between aspect richness, item diversity, and lexical variety. Although cross-CRS experiments support the robustness of aspect-level findings, they rely on automated evaluation and do not establish agreement with real-user assessments of recommenders. Generalization beyond English movie recommendation remains to be investigated. Ethical Considerations ReDial is released under C BY 4.0, MovieLens-25M is available for non-commercial research use, and the iEvaLM-CRS implementations of KBRD and UniCRS Wang et al. (2023) are under the MIT License; we use all of them solely for research. LLMs are accessed through their official APIs under the providersâ terms. References Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §3.5, §3. Afzali et al. (2023) J. Afzali, A. M. Drzewiecki, K. Balog, and S. Zhang UserSimCRS: a user simulation toolkit for evaluating conversational recommender systems. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM â23, p. 1160â1163. External Links: Link, Document Cited by: §1, §2. Anthropic (2025) Anthropic Claude haiku 4.5 system card. Technical report External Links: Link Cited by: §3, §4.2. Balog and Zhai (2025) K. Balog and C. Zhai User simulation in the era of generative ai: user modeling, synthetic data generation, and system evaluation. ArXiv abs/2501.04410. External Links: Link Cited by: §1, §2. Bernard and Balog (2024) N. Bernard and K. Balog Identifying breakdowns in conversational recommender systems using user simulation. Proceedings of the 6th ACM Conference on Conversational User Interfaces. External Links: Link Cited by: §4.3. Bernard and Balog (2025) N. Bernard and K. Balog Limitations of current evaluation practices for conversational recommender systems and the potential of user simulation. In Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, p. 261â271. Cited by: §A.3, §2. Chen et al. (2024) J. Chen, X. Wang, R. Xu, S. Yuan, Y. Zhang, W. Shi, J. Xie, S. Li, R. Yang, T. Zhu, A. Chen, N. Li, L. Chen, C. Hu, S. Wu, S. Ren, Z. Fu, and Y. Xiao From persona to personalization: a survey on role-playing language agents. External Links: 2404.18231, Link Cited by: §3.2. Chen et al. (2025a) L. Chen, Q. Dai, Z. Zhang, X. Feng, M. Zhang, P. Tang, X. Chen, Y. Zhu, and Z. Dong RecUserSim: a realistic and diverse user simulator for evaluating conversational recommender systems. Companion Proceedings of the ACM on Web Conference 2025. External Links: Link Cited by: §2. Chen et al. (2025b) N. Chen, Q. Dai, X. Dong, P. Wang, Q. Jia, Z. Du, Z. Dong, and X. Wu Evaluating conversational recommender systems via large language models: a user-centric framework. External Links: Link Cited by: §2. Chen et al. (2019) Q. Chen, J. Lin, Y. Zhang, M. Ding, Y. Cen, H. Yang, and J. Tang Towards knowledge-based recommender dialog system. External Links: 1908.05391, Link Cited by: §4. Cohen (1968) J. Cohen Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit.. Psychological bulletin 70 (4), p. 213. Cited by: §F.5. Cohen (2013) J. Cohen Statistical power analysis for the behavioral sciences. 2 edition, Routledge, London, England. Cited by: §4.1. Comanici et al. (2025) G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: §3.5, §3. Fu et al. (2020) Z. Fu, Y. Xian, Y. Zhang, and Y. Zhang Tutorial on conversational recommendation systems. Proceedings of the 14th ACM Conference on Recommender Systems. External Links: Link Cited by: §1. Gwet (2008) K. L. Gwet Computing inter-rater reliability and its variance in the presence of high agreement.. The British journal of mathematical and statistical psychology 61 Pt 1, p. 29â48. External Links: Link Cited by: §F.5. Harper and Konstan (2015) F. M. Harper and J. A. Konstan The movielens datasets: history and context. Acm transactions on interactive intelligent systems (tiis) 5 (4), p. 1â19. Cited by: §4.2. Hayati et al. (2020) S. A. Hayati, D. Kang, Q. Zhu, W. Shi, and Z. Yu INSPIRED: toward sociable recommendation dialog systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), B. Webber, T. Cohn, Y. He, and Y. Liu (Eds.), Online, p. 8142â8152. External Links: Link, Document Cited by: §H.1, §3.4. Holm (1979) S. Holm A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6 (2), p. 65â70. External Links: ISSN 03036898, 14679469, Link Cited by: §4.1. Jin et al. (2024) Y. Jin, L. Chen, W. Cai, and X. Zhao CRS-que: a user-centric evaluation framework for conversational recommender systems. ACM Trans. Recomm. Syst. 2 (1). External Links: Link, Document Cited by: §2. Kim et al. (2012) H. Kim, Y. U. Ryu, Y. H. Cho, and J. K. Kim Customer-driven content recommendation over a network of customers. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 42, p. 48â56. External Links: Link Cited by: §1. Kim et al. (2024) M. Kim, M. Kim, H. Kim, B. Kwak, S. Kang, Y. Yu, J. Yeo, and D. Lee Pearl: a review-driven persona-knowledge grounded conversational recommendation dataset. Findings of the Association for Computational Linguistics: ACL 2024, p. 1105â1120. Cited by: §2. Kostric et al. (2025) I. Kostric, U. Gadiraju, and K. Balog Know your users! estimating user domain knowledge in conversational recommenders. arXiv preprint arXiv:2512.13173. Cited by: §H.1, §3.4. Li et al. (2018) R. Li, S. Ebrahimi Kahou, H. Schulz, V. Michalski, L. Charlin, and C. Pal Towards deep conversational recommendations. Advances in neural information processing systems 31. Cited by: §3.1. Lin et al. (2022) H. Lin, C. Geishauser, S. Feng, N. Lubis, C. van Niekerk, M. Heck, and M. Gasic GenTUS: simulating user behaviour and language in task-oriented dialogues with generative transformers. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, O. Lemon, D. Hakkani-Tur, J. J. Li, A. Ashrafzadeh, D. H. Garcia, M. Alikhani, D. Vandyke, and O. DuĆĄek (Eds.), Edinburgh, UK, p. 270â282. External Links: Link, Document Cited by: §2. Liu et al. (2023) Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ProcessingProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Stroudsburg, PA, USA, p. 2511â2522. Cited by: §4.2. Ma et al. (2025) C. Ma, Z. Xu, Y. Ren, D. Hettiachchi, and J. Chan PUB: an llm-enhanced personality-driven user behaviour simulator for recommender system evaluation. Cited by: Table 1, §2. Oliver and Raney (2011) M. B. Oliver and A. A. Raney Entertainment as pleasurable and meaningful: identifying hedonic and eudaimonic motivations for entertainment consumption. Journal of Communication 61 (5), p. 984â1004. External Links: Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1460-2466.2011.01585.x Cited by: §H.1, §3.4. Radlinski et al. (2019) F. Radlinski, K. Balog, B. Byrne, and K. Krishnamoorthi Coached conversational preference elicitation: a case study in understanding movie preferences. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, S. Nakamura, M. Gasic, I. Zukerman, G. Skantze, M. Nakano, A. Papangelis, S. Ultes, and K. Yoshino (Eds.), Stockholm, Sweden, p. 353â360. External Links: Link, Document Cited by: §H.1, §3.4. Reimers and Gurevych (2019) N. Reimers and I. Gurevych Sentence-BERT: sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Stroudsburg, PA, USA. Cited by: §4.2. RuĂ et al. (2012) A. RuĂ, M. Quade, M. Kruppa, and M. Runge Rule-based approach for simulating age-related usability problems. External Links: Link Cited by: §1. Scott and Bruce (1995) S. G. Scott and R. A. Bruce Decision-making style: the development and assessment of a new measure. Educational and Psychological Measurement 55, p. 818 â 831. External Links: Link Cited by: §H.1, §3.4. SekuliÄ et al. (2024) I. SekuliÄ, L. Lu, N. S. Bedi, and F. Crestani Simulating conversational search users with parameterized behavior. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, SIGIR-AP 2024, New York, NY, USA, p. 72â81. External Links: ISBN 9798400707247, Link, Document Cited by: §2, §4.3. Tang and Liao (2022) M. Tang and I. Liao Preference diversity and openness to novelty: scales construction from the perspective of movie recommendation. Journal of the Association for Information Science & Technology 73 (9), p. 1222â1235. External Links: Document, Link Cited by: §H.1, §3.4. Tencent Hy Team (2026) Tencent Hy Team Hy3: a mixture-of-experts language model. Note: https://github.com/Tencent-Hunyuan/Hy3Official model repository Cited by: §3. Tigunova et al. (2019) A. Tigunova, A. Yates, P. Mirza, and G. Weikum Listening between the lines: learning personal attributes from conversations. In The World Wide Web Conference, p. 1818â1828. Cited by: §1, §1, §3.2. Torbati et al. (2021) G. H. Torbati, A. Yates, and G. Weikum You get what you chat: using conversations to personalize search-based recommendations. In European Conference on Information Retrieval, p. 207â223. Cited by: §3.2. Wang et al. (2024) N. Wang, Z.y. Peng, H. Que, J. Liu, W. Zhou, Y. Wu, H. Guo, R. Gan, Z. Ni, J. Yang, M. Zhang, Z. Zhang, W. Ouyang, K. Xu, W. Huang, J. Fu, and J. Peng RoleLLM: benchmarking, eliciting, and enhancing role-playing abilities of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 14743â14777. External Links: Link, Document Cited by: §3.2. Wang et al. (2023) X. Wang, X. Tang, X. Zhao, J. Wang, and J. Wen Rethinking the evaluation for conversational recommendation in the era of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 10052â10065. External Links: Link, Document Cited by: §2, Ethical Considerations. Wang et al. (2022) X. Wang, K. Zhou, J. Wen, and W. X. Zhao Towards unified conversational recommender systems via knowledge-enhanced prompt learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD â22, p. 1929â1937. External Links: Link, Document Cited by: §4. Yoon et al. (2024) S. Yoon, Z. He, J. Echterhoff, and J. McAuley Evaluating large language models as generative user simulators for conversational recommendation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 1490â1504. Cited by: §A.3, §H.1, §1, §1, §2, §3.4. Zhang and Balog (2020) S. Zhang and K. Balog Evaluating conversational recommender systems via user simulation. In Proceedings of the 26th acm sigkdd international conference on knowledge discovery & data mining, p. 1512â1520. Cited by: §2. Zhao et al. (2025) X. Zhao, Y. Deng, W. Wang, H. Cheng, R. Zhang, S. Ng, T. Chua, et al. Exploring the impact of personality traits on conversational recommender systems: a simulation with large language models. arXiv preprint arXiv:2504.12313. Cited by: Table 1, §2. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging LLM-as-a-judge with MT-bench and chatbot arena. External Links: 2306.05685 Cited by: §4.2. Zhu et al. (2024a) L. Zhu, X. Huang, and J. Sang A llm-based controllable, scalable, human-involved user simulator framework for conversational recommender systems. Proceedings of the ACM on Web Conference 2025. External Links: Link Cited by: §1, §1, Table 1, §2, §2. Zhu et al. (2024b) L. Zhu, X. Huang, and J. Sang How reliable is your simulator? analysis on the limitations of current llm-based user simulators for conversational recommendation. In Companion Proceedings of the ACM Web Conference 2024, W â24, New York, NY, USA, p. 1726â1732. External Links: ISBN 9798400701726, Link, Document Cited by: §1, §1, §2, §4.2. Appendix A T2 Degeneracy Diagnostics The main text reports only the pooled T2 popularity correlation (§4.3). This appendix documents why per-user and persona-anchored variants are not reportable. A.1 Per-User Popularity Correlation For each user, the per-user popularity Pearson r is the correlation between the simulatorâs rating and the MovieLens-25M mean rating across the movies that user mentioned. This requires a minimum sample of âŒ30 30 (user, movie) pairs per user for a reliable estimate. Only 6/3006/300 Full users reach this threshold (median 77 pairs per user, maximum 4747); on the matched subset, only 22 users in each of Full and StatisticalOnly reach this threshold. A paired test across users is therefore not feasible. The pooled estimate in Table 2 aggregates all (user, movie) pairs across the cohort, treating each pair as an independent observation, a weaker statistical claim than per-user testing would provide, but the only one available at our sample size. A.2 Persona-Anchored T2 (Liked vs. Disliked) A second variant of T2 correlates the simulatorâs in-conversation rating with the userâs ReDial liked/disliked label (binary ground truth). Table 4 reports the diagnostics. In every condition, 9494â97%97\% of high-confidence ground-truth labels are liked, giving varianceredial<0.1variance_redial<0.1, a near-constant variable. Pearson r against a near-constant variable is driven by a handful of disliked pairs and is not a reliable discriminative metric, so the apparent ordering across conditions is not interpreted. Full StatOnly Persona r (pooled) 0.603 0.261 n pairs 52 92 Var.redial_redial 0.078 0.074 % liked (GT) 94.0% 96.3% n high-conf. pairs 50 81 n users, both classes 2 2 Accuracy (high-conf.) 94.0% 96.3% Table 4: T2 persona-anchored diagnostics on the matched 100-user subset. Ground-truth labels are near-constant in every condition, making Pearson r unreliable as a discriminative metric. A.3 Balanced Accuracy as an Alternative A natural alternative is balanced accuracy on liked/disliked classification. Computing this requires users with both classes in ground truth. As shown above, only 22 users per condition satisfy this, too few to estimate a meaningful per-user balanced accuracy distribution. Pooled simple accuracy (9494â96%96\%) equals the majority-class rate and is uninformative. Recommendation for future work. Per-user behavioral alignment metrics in CRS user simulation require either (a) a larger sample where most users have â„30â„ 30 rated items in conversation, or (b) controlled prompting that elicits a balanced acceptance/ rejection ratio per user (e.g., 50/5050/50 targeted CRS recommendations). Both would address the per-user consistency gap noted by Yoon et al. (2024) and Bernard and Balog (2025). Appendix B Auditor Flag Taxonomy and Examples B.1 Flag Taxonomy Issue type High Med. Low Tot. fabrication 12 47 72 131 caricature 12 255 105 372 history_contradiction 55 68 50 173 structural_error 0 0 0 0 Total 79 370 227 676 Table 5: Distribution of auditor flags by type and severity across 300300 interviews (final attempts). Dialogue-level decision distribution: accept 209209, partial 9191, regenerate 00 (one interview regenerated once before receiving partial). Table 5 summarizes the full flag distribution across the 300300 interviews. We make three observations: 1. Invented rationalization is the dominant pattern (372/676372/676 flags are caricature, 69%69\% of them medium severity): the simulator frequently produces reasons for preferences that sound plausible but are not attested in the profile. 2. Flags grounded in the userâs own evidence are the most reliable. Contradictions with the liked and disliked lists or with the ReDial conversation can be checked directly (Examples 2â4). Fabrication flags about film facts rely on the auditorâs parametric knowledge and are less reliable: in several cases the auditor flagged accurate scene descriptions as fabricated. Such errors are conservative: they can remove grounded content from the enrichment input, but they do not introduce unsupported claims into the profile. 3. regenerate is rare. The auditor reserves regenerate for structural failure or â„3â„ 3 high-severity flags. This occurred in one of 300300 interviews, and the regenerated interview received partial, indicating that gross structural failure is rare with capable LLMs (Gemini 2.5 Flash) at our interview scale. The dominant failure mode is local rather than global. B.2 Representative Example 1: accept User 78. The auditor raised no flags; every film the interviewee mentions appears in the userâs liked or disliked lists. Q1: If a friend asked you to recommend a movie that captures your taste, what would you pick and why? A1: Iâd probably point them toward something like Juno or The Perks of Being a Wallflower. I really lean toward comedies and dramas that have a sharp, witty sense of humor while still hitting you with some genuine, deeper moments. Q4: [âŠ] When does that attempt at honesty in a drama or comedy feel unearned or fall flat for you? A4: [âŠ] Like when I watched White Noise, it just didnât land for me at allâit missed that grounded, witty balance and felt a bit too forced. B.3 Representative Example 2: partial (history contradiction) User 1. One high-severity history_contradiction flag (turn 6) and one low-severity caricature flag (turn 10); turn 6 was excluded from enrichment. A (turn 6, excluded): I just really enjoy that buddy-cop or mismatched-duo comedy style [âŠ]. Itâs the same kind of fun, fast-paced energy I liked in Blue Streak with Martin Lawrence [âŠ] Auditor explanation: âThe user claims to have liked Blue Streakâs energy in the recent dialogue, but in the earlier conversation explicitly stated they had not seen Blue Streak and expressed intent to watch it in the future.â B.4 Representative Example 3: partial (caricature) User 730. One high-severity caricature flag (turn 8), which was excluded from enrichment. A (turn 8, excluded): Honestly, a lot of big-budget stuff just gets way too chaotic and messy without really earning those stakes, like some of the massive CGI battles in Justice League where itâs hard to actually care about whatâs happening. Auditor explanation: âThe user claims Justice Leagueâs CGI battles donât earn stakes and are hard to care about, but Justice League is in their liked movies list with no negative commentary in the profile.â This illustrates an invented rationalization: a plausible reason for a dislike that the userâs history does not support. B.5 Representative Example 4: partial (invented rationalization) User 66. One high-severity caricature flag (turn 8) and one medium-severity caricature flag (turn 2); both turns were excluded from enrichment. A (turn 8, excluded): I think it fails when a comedy tries way too hard to be funny without caring about the characters, like in The Hangover or Superbad. It just feels loud and obnoxious rather than having any real situation driving the humor. Auditor explanation: âThis response performs a snobby cinephile persona dismissing comedies as âloud and obnoxious,â but the profile shows the user actually likes comedies like Super Troopers, Beerfest, and Tommy Boy.â Both films are on the userâs disliked list, so the dislike itself is attested; the stated reason is not. The simulator invents a plausible justification that is inconsistent with the userâs liking for comparably broad comediesâan invented rationalization in the sense of Stage 4. Appendix C Per-User Metric Distributions The main text (§4.3) shows the per-user T3 aspect entropy distribution. This appendix shows the corresponding distributions for all other primary metrics on the matched 100-user subset. Throughout, Full is dark blue and StatisticalOnly is light blue, with dashed lines marking condition means. Figure 5: T1 per-user item entropy. Full is shifted left of StatisticalOnly (2.0342.034 vs. 2.4092.409; Î=â0.375 =-0.375, dz=â0.95d_z=-0.95, padj<0.001p_adj<0.001), reflecting a more concentrated movie-mention distribution. Figure 6: T1 per-user JSD against the ReDial reference distribution. Full is marginally higher (0.6810.681 vs. 0.6760.676; dz=0.47d_z=0.47, padj<0.001p_adj<0.001); the absolute difference is small. Figure 7: T3 per-user sentiment entropy. Full is slightly lower (0.1530.153 vs. 0.2010.201; dz=â0.26d_z=-0.26, uncorrected p=0.011p=0.011). Both conditions cluster at zero (7272 and 6464 users), reflecting a strong positive-sentiment imbalance. Figure 8: T4 per-user SBERT semantic diversity. Full is slightly higher (0.5320.532 vs. 0.5000.500; dz=0.25d_z=0.25, padj=0.11p_adj=0.11); 1010 Full and 1515 StatisticalOnly users fall below 0.40.4. Figure 9: T4 per-user corpus TTR. Full TTR is systematically lower (0.6740.674 vs. 0.7280.728; dz=â0.68d_z=-0.68, uncorrected p<0.001p<0.001); enriched users repeat tokens more. Figure 10: T4 per-user n-gram monotony (Jaccard overlap of bigrams/trigrams between turn pairs; higher = more repetitive). Full is skewed right of StatisticalOnly (0.0280.028 vs. 0.0190.019; dz=0.36d_z=0.36, uncorrected p<0.001p<0.001), reflecting profile-vocabulary reuse. Appendix D T3 Aspect Vocabulary Rank Full aspect Count StatOnly aspect Count 1 mood 164 movie overall 407 2 pacing 103 plot 58 3 movie overall 81 mood 57 4 atmosphere 78 premise 23 5 overall vibe 50 action 22 6 plot 43 overall concept 21 7 vibe 37 humor 20 8 overall experience 27 genre 20 9 humor 26 comedy 18 10 mood/atmosphere 20 intensity 12 11 energy 18 overall fit 10 12 action 17 drama 9 13 overall quality 14 genre/mood 8 14 overall concept 14 vibe 8 15 tone 11 tone 7 16 overall appeal 11 atmosphere 7 17 premise 11 soundtrack 7 18 mood/vibe 11 pacing 6 19 intensity 11 thriller 6 20 style 10 overall quality 6 Distinct aspects (filtered) 267 174 Table 6: Top-20 most frequent aspects on the matched 100-user subset (after filtering generic placeholders). Fullâs vocabulary skews toward experiential dimensions (mood, pacing, atmosphere, energy); StatisticalOnly is dominated by the near-generic movie overall (43%43\% of its aspect mentions vs. 7%7\% for Full) and broad genre labels. Total distinct non-generic aspects: 267267 vs. 174174. Table 6 shows the top-20 aspects in each condition after filtering generic placeholders. Beyond the aggregate counts, the qualitative difference is informative: Full introduces experiential vocabulary (pacing, overall vibe, energy) that is rare or absent in StatisticalOnly, whose mentions concentrate on a single holistic judgment (movie overall). Appendix E Cross-CRS Robustness with Trained Recommenders We conducted a separate paired robustness experiment using the frozen ABLATION-120 cohort, sampled from the original FULL-300 population before inspecting the results. Each user interacted with KBRD and UniCRS under two conditions: Full, containing the statistical profile and semantic enrichment, and StatisticalOnly, containing the same statistical profile and pickiness score but no semantic enrichment. For every user, condition, and recommender, we generated four conversations corresponding to the aspect, movie, mood, and expertise opening strategies. This produced 120Ă2Ă2Ă4=1,920120Ă 2Ă 2Ă 4=1,920 conversations. User identity, behavioral statistics, opening-strategy schedule, conversation budget, and recommender configuration were held fixed across paired conditions. Recommender implementations and generation. We used the iEvaLM-CRS implementations and released ReDial checkpoints. KBRD combines a relational-graph recommender with a BART-base conversation model (checkpoint files kbrd_rec_redial/model.pt and kbrd_conv_redial/pytorch_model.bin; SHA-256 prefixes d62f7548 and d85488a). UniCRS combines knowledge-graph prompts with DialoGPT-small and a RoBERTa-base text encoder (unicrs_rec_redial/model.pt and unicrs_conv_redial/model.pt; SHA-256 prefixes 9cd55118 and 23599951). For UniCRS, the released checkpointâs padding-token index was mapped to the <movie> placeholder before movie-title substitution. Both recommenders used a maximum response length of 128 tokens; UniCRS additionally used a no-repeat 3-gram constraint. The user simulator was Gemini 2.5 Flash with temperature 0.70.7, a 256-token output limit, and reasoning disabled. Each conversation was generated once without a fixed sampling seed. Unlike the primary experiment, which used Claude Haiku 4.5 for evaluation, this robustness experiment evaluated the same stored conversations with separate GPT-4o-mini and DeepSeek-chat calls, both at temperature 0.00.0. Each evaluator used its own extraction outputs and T3 reference baseline (baseline fingerprints 1afc994d78c1045 and 82a23731b46cf48d, respectively). We therefore interpret within-judge paired differences and do not compare absolute entropy values across judges or with the primary experiment. Post-hoc CRS-dialogue audit. This Phase-2 audit is distinct from the Stage 4 Audit Agent that audits interviews before enrichment. GPT-4o-mini (temperature 0.20.2) checked the simulated userâs messages for fabrication, caricature, role confusion, and persona drift, with a 512-token output limit. It assigned Accept when there was no major issue (at most a minor issue in one turn), Partial when one issue type was present but the conversation remained mostly usable, and Reject when multiple issue types or one severe issue were present; role confusion across multiple turns was explicitly severe. Model, prompt, temperature, and criteria were fixed across recommenders and conditions; only the condition-specific profile and generated conversation differed. Audit decisions were produced before T3 extraction and without access to extraction outputs. The audit affected inclusion only and did not regenerate conversations. Although the auditor and one T3 judge both used GPT-4o-mini, they were separate calls with different prompts; the DeepSeek-chat evaluation used a different underlying model. Audit filtering and paired-sample construction. All four CRSâcondition arms contained 480 technically completed conversations (120 users Ă four opening strategies), and no conversation terminated with a transport or generation error. In the primary analysis, individual conversations assigned a Phase-2 Reject decision were excluded, whereas Accept and Partial conversations were retained. Rejecting one conversation did not remove the entire user: a user remained evaluable in an arm whenever at least one of their four conversations was retained. The paired analysis then used the intersection of users evaluable in both conditions. This produced 111 paired users for KBRD (112 evaluable in Full and 118 in StatisticalOnly) and 80 for UniCRS (94 and 99, respectively). The conversation-level rejection rates were 39.6% for Full and 36.5% for StatisticalOnly with KBRD, and 68.5% and 61.7%, respectively, with UniCRS. A post-hoc paired Wilcoxon diagnostic over per-user rejection rates found no significant condition difference for KBRD (p=0.262p=0.262), but found a higher rejection rate for Full with UniCRS (p=0.019p=0.019; two-sided). We therefore report the audit-filtered analysis together with an all-conversation sensitivity analysis. Statistical inference. For each recommenderâjudge combination, inference was performed on paired user-level differences. We report percentile 95% confidence intervals from 10,000 paired bootstrap resamples (seed 00), two-sided Wilcoxon signed-rank tests with zero differences removed and SciPyâs automatic exact/asymptotic method selection, and paired Cohenâs dzd_z. The table reports unadjusted p-values for these four T3 comparisons. As a robustness check, two-sided paired sign-flip tests with 100,000 randomizations (seed 11) gave the same significance conclusions. CRS Evaluation judge nun_u Full StatOnly Î T3 95% CI pWp_W dzd_z KBRD GPT-4o-mini 111 0.392 0.126 +0.265 [0.144, 0.389] 0.0002 0.40 KBRD DeepSeek-chat 111 0.350 0.069 +0.280 [0.170, 0.394] <0.0001<0.0001 0.46 UniCRS GPT-4o-mini 80 0.488 0.157 +0.330 [0.179, 0.487] 0.0001 0.47 UniCRS DeepSeek-chat 80 0.498 0.129 +0.369 [0.214, 0.527] <0.0001<0.0001 0.52 Table 7: Primary audit-filtered Cross-CRS results for per-user filtered T3 aspect entropy. Here, nun_u is the number of users with at least one Accept or Partial conversation in both conditions. For each user and condition, aspectâsentiment pairs were pooled across all retained conversations before computing entropy. Differences are Fullâ-StatisticalOnly; pWp_W denotes the two-sided paired Wilcoxon signed-rank test. All bootstrap confidence intervals exclude zero. Because means and differences are computed from unrounded user-level values, a displayed difference need not equal the subtraction of the two rounded means. Absolute entropy values should not be compared across evaluation judges. Figure 11: Paired effect of semantic enrichment on filtered T3 aspect entropy. Points represent mean user-level differences and horizontal bars show 95% bootstrap confidence intervals. The vertical line marks the null effect. Circles denote GPT-4o-mini and squares denote DeepSeek-chat; dark and light blue denote KBRD and UniCRS. Per-user T3 aggregation. For each user and condition, the evaluator extracted aspectâsentiment pairs from every retained conversation and merged them into a single user-level aspect-frequency distribution. Filtered T3 aspect entropy was then computed once from this merged distribution. We did not compute a separate entropy for each opening strategy and then average the four values. The paired effect was calculated as Îu=HuFullâHuStatisticalOnly _u=H_u Full-H_u StatisticalOnly, and statistical inference was performed over these user-level differences. Because retained conversation counts could differ between conditions, the filtered entropy estimates may be sensitive to unequal exposure. Among paired users, the mean numbers of retained conversations were 2.60 for Full and 2.62 for StatisticalOnly with KBRD, and 1.66 and 1.80, respectively, with UniCRS. Pooling more conversations can expose more aspect categories and thereby affect entropy, so the filtered estimates may reflect both condition effects and retained-exposure differences. We therefore use the all-conversation analysis below, which retains all four opening strategies in both conditions for all 120 users, as a sensitivity check. Table 7 and Figure 11 show a positive primary T3 effect for both recommenders and both evaluation judges. The all-conversation sensitivity analysis yielded effects of +0.275+0.275 and +0.371+0.371 for KBRD, and +0.560+0.560 and +0.534+0.534 for UniCRS, under GPT-4o-mini and DeepSeek-chat, respectively (all p<0.001p<0.001). Here, âtechnically completedâ means that a complete conversation record was produced without a transport or generation error; it does not require natural acceptance and includes max-turn terminations. Per-user T3 scores were strongly correlated between the two evaluation judges: Spearmanâs Ï=0.77Ï=0.77 in the primary analysis and Ï=0.72Ï=0.72 in the all-conversation sensitivity analysis. The consistent direction makes unequal exposure an unlikely sole explanation of the T3 result, but does not establish overall simulator quality or fidelity to individual real users. Appendix F Human Evaluation Protocol F.1 Annotation Task Each annotation task presented an annotator with a single claim (one of the six attribute values from a Full-enriched user profile), the userâs behavioral profile (Stage 1â2 statistics), and the userâs interview transcript (Stage 3 output, including any turns the auditor had flagged). Annotators were asked to judge whether the claim was supported by the interview evidence. F.2 Rating Scales Support (4-class): âą Supported: the claim is directly justified by specific content in the interview. âą Partially supported: the interview contains relevant content but does not fully justify the specific claim (e.g., the claim is more specific or stronger than the evidence). âą Unsupported: the interview does not contain evidence relevant to the claim, even though the claim is consistent with the profile. âą Contradicted: the interview contains evidence inconsistent with the claim. Grounding (3-class): âą Evidence-grounded: the claim is anchored to a specific quotation or behavioral pattern in the interview. âą Generic: the claim is consistent with the interview but could apply to many users; the evidence is weak or non-specific. âą Cliche/stereotype: the claim performs a recognizable persona archetype not specifically attested in the userâs data. Numeric ratings (1â5): âą Consistency: how consistent the claim is with the userâs behavioral profile (1 = inconsistent, 5 = highly consistent). âą Usefulness: how useful the claim would be for conditioning a simulator (1 = not useful, 5 = highly useful). âą Overreach: how much the claim overshoots the available evidence (1 = no overreach, 5 = severe overreach). âą Confidence: the annotatorâs confidence in their judgment (1 = uncertain, 5 = highly confident). Mean ratings across all 315315 annotations are 3.773.77 (consistency), 3.613.61 (usefulness), 1.701.70 (overreach), and 3.003.00 (confidence). F.3 Annotator Pool and Procedure Three annotators (graduate students with prior NLP annotation experience) independently rated claims sampled at random from the 582582 enriched-attribute claims of the Full profiles. We report the 105105 claims, drawn from 4545 users and covering all six attribute types, that were rated by all three annotators (315315 annotations). Annotators worked independently without discussion of disagreements; no calibration round was performed prior to annotation, a methodological choice we identify as a limitation (see Limitations). F.4 Per-Attribute Detail Attribute Sup. Part. Uns. Contr. Viewing motivation 27.5% 72.5% 0.0% 0.0% Aspect prefs. (liked) 75.0% 25.0% 0.0% 0.0% Aspect prefs. (descr.) 33.3% 66.7% 0.0% 0.0% Pickiness 72.7% 24.2% 0.0% 3.0% Domain expertise 27.8% 61.1% 11.1% 0.0% Novelty preference 26.7% 60.0% 13.3% 0.0% Aspect prefs. (disliked) 38.1% 42.9% 9.5% 9.5% Decision-making style 11.9% 66.7% 21.4% 0.0% All 45.1% 48.3% 5.4% 1.3% Table 8: Full per-attribute breakdown of support ratings (annotation level). The 1.3%1.3\% Contradicted rate concentrates in disliked aspects (9.5%9.5\%) and pickiness (3.0%3.0\%); decision-making style has the highest Unsupported rate (21.4%21.4\%), consistent with cognitive-style claims being the most amenable to invented rationalization. F.5 Inter-Annotator Agreement Details We report multiple agreement metrics. Since the Support label is ordinal (Supported â» Partially supported â» Unsupported â» Contradicted), quadratic-weighted Îș is the methodologically appropriate primary metric (Cohen, 1968): it penalizes large ordinal disagreements (Supported vs. Contradicted) more than small ones (Supported vs. Partially supported). Support label (4-class, ordinal). Pairwise agreement across the 105105 triply-annotated claims: Pair Unwgt. Îș Wgt. Îș Exact ±1± 1-rank A vs. B 0.30 0.43 60.0% 98.1% A vs. C 0.19 0.40 45.7% 97.1% B vs. C 0.43 0.52 69.5% 98.1% Table 9: Pairwise inter-annotator agreement on the Support label. Weighted Îș is quadratic-weighted; â±1± 1-rankâ is the rate at which two annotators fell in the same or adjacent ordinal category. Annotator C applies the strictest interpretation of Supported. Grounding label (3-class, nominal). For the Grounding label (Evidence-grounded / Generic / Cliche-stereotype), Îș values are: A vs. B =0.12=0.12, A vs. C =0.27=0.27, B vs. C =0.37=0.37. Agreement on grounding is fair, mainly reflecting the boundary between Generic and Evidence-grounded; only one of 315315 annotations used Cliche/stereotype. Binary support. Collapsing the 4-class label into positive (Supported ++ Partially supported) vs. negative (Unsupported ++ Contradicted) gives a coarser agreement signal: âą A vs. B: Îș=0.20Îș=0.20, raw agreement 93.3%93.3\% âą A vs. C: Îș=0.37Îș=0.37, raw agreement 89.5%89.5\% âą B vs. C: Îș=0.11Îș=0.11, raw agreement 88.6%88.6\% All three annotators agree on 85.7%85.7\% of claims under this split. Binary Îș is depressed by the high prevalence of positive labels (the kappa paradox); Gwetâs AC1 (Gwet, 2008), which is robust to prevalence, is 0.890.89 for this split, and 0.990.99 for Contradicted versus all other labels. Disagreement pattern. The most consistent disagreement is between Supported and Partially supported: in 32%32\% of annotator pairs per claim, one annotator marked the claim Supported while the other marked it Partially supported. This boundary was under-specified in our protocol: the task statement did not operationalize how much explicit textual evidence was required to justify the stronger rating. Annotator C as strictest rater. Annotator C marks claims Supported far less often than A and B (24.8%24.8\% vs. 68.6%68.6\% and 41.9%41.9\%), while all three judge most claims at least partially supported (88.6%88.6\%, 93.3%93.3\%, and 98.1%98.1\% for C, A, and B). Manual inspection suggests C required near-verbatim evidence for Supported, whereas A and B accepted paraphrased evidence. We retain all three annotators in the main results. What the agreement numbers mean for the paperâs claims. The 94%94\% aggregate support rate quoted in §4.6 is robust under several reasonable agreement assumptions: (a) it remains â„91%â„ 91\% under any single-annotator-out leave-one-out; (b) within-one-rank agreement of 9797â98%98\% across all annotator pairs indicates strong convergence on the direction of support, which is the quantity the headline claim relies on; (c) the 1.3%1.3\% Contradicted rate, where chance-corrected agreement is highest (AC1 =0.99=0.99), is the single most claim-critical number for paper integrity (it bounds the fabrication rate the auditor failed to catch). A revised protocol with sharper category definitions, an agreement-calibration pre-annotation phase, and a larger pool of annotators is the natural next step for human-eval methodology in this domain. Appendix G Human Evaluation Protocol for Audit Decisions Evaluation material and sampling. The evaluation unit was a single synthetic interviewee response produced by the SimInter user-simulation pipeline and processed by Stage 4 in the 300300-user experiment. These are generated interview responses, not utterances taken from ReDial. Each response was linked to the historical ReDial user whose interactions informed the simulation. We selected 105105 turns from distinct users, equally stratified by the auditorâs final outcome: 3535 Excluded, 3535 FlaggedâRetained, and 3535 Unflagged. The three groups were presented as consecutive 3535-case blocks in a fixed order shared by all annotators, but the group boundaries and auditor outputs were concealed. Annotation procedure and human reference. Three co-author annotators independently reviewed the same 105105 turns (315315 judgments in total). They saw the target synthetic response, optional preceding synthetic-interview context, automatically retrieved statements from the matching historical ReDial userâs original conversations, and optional derived statistical-profile information. They did not see the auditorâs action, issue flags, or explanations. The three judgment categories were interpreted as follows: Excluded, an evidence-grounded problem warrants removing the entire synthetic turn; FlaggedâRetained, a meaningful concern exists but does not warrant removal; and Unflagged, no meaningful issue is identified. We used the majority vote of the three annotators as the human reference for each turn. The evaluation unit and reference are at the whole-turn level; they do not validate the correctness of individual auditor flag types. Inter-annotator agreement. All 105105 cases received three labels. Agreement was unanimous on 70/10570/105 (66.7%66.7\%) and 22â11 on the remaining 3535; there were no three-way splits. Nominal Fleissâ Îș was 0.6640.664 (95% stratified case-bootstrap interval [0.572,0.751][0.572,0.751]; 2,5002,500 replicates, resampling within the three auditor strata). Within-stratum Fleissâ Îș values were 0.3370.337 (excluded), 0.3080.308 (flagged-but-retained), and 0.2680.268 (unflagged). These lower within-stratum values suggest that aggregate agreement is influenced by the differences in label distributions across strata. Auditor outcome n Human Human Human Excluded Flaggedâret. Unflagged Excluded 35 31 4 0 Flaggedâretained 35 0 31 4 Unflagged 35 1 5 29 Total 105 32 40 33 Table 10: Auditor outcomes versus the majority judgments of three annotators for 105105 synthetic interviewee turns. The diagonal corresponds to 91/10591/105 (86.7%86.7\%) three-class human-referenced accuracy on the balanced sample. Three-class and filtering accuracy. Three-class accuracy against the human-majority reference was 91/10591/105 (86.7%86.7\%): 31/3531/35 (88.6%88.6\%) in the excluded stratum, 31/3531/35 (88.6%88.6\%) in the flagged-but-retained stratum, and 29/3529/35 (82.9%82.9\%) in the unflagged stratum (Table 10). Fourteen cases disagreed at the three-class level: four auditor-excluded turns were judged flagged-but-retained; four flagged-but-retained turns were judged unflagged; and among auditor-unflagged turns, five were judged flagged-but-retained and one excluded. Thus, nine of the fourteen disagreements concerned the distinction between flagged-but-retained and unflagged outcomes, without changing the final filtering action. When both retained categories were collapsed to a single Retain action, the auditor agreed with the majority human filtering action on 100/105100/105 turns (95.2%95.2\%): 31/3531/35 auditor-excluded turns and 69/7069/70 auditor-retained turns. On the same evaluation set, an always-retain baseline would agree with 73/10573/105 (69.5%69.5\%) human binary judgments. This comparison is descriptive for the selected, audit-stratified sample; its class balance differs from the full Stage 4 population. Illustrative disagreements and supported exclusions. In CASE-066, the auditor excluded a synthetic response expressing appreciation for Get Out based partly on an alleged aversion to scary films. However, the corresponding historical ReDial user requested horror and a combination of comedy and scares (ReDial conversation 3189); two annotators favored retention and one exclusion. This illustrates the risk of treating genre preferences as exclusive. Conversely, in CASE-071 all three annotators endorsed the auditorâs exclusion of a detailed recollection from Arrival; the historical user had earlier stated that they did not think they had seen the film and would add it to a watch list (ReDial conversation 12034). Such temporal evidence is suggestive rather than conclusive: a user might watch the film subsequently. Finally, in CASE-082, all three annotators favored excluding a synthetic response that the auditor left unflagged. The historical user had previously stated that they had not seen The Greatest Showman (ReDial conversation 14457). However, expressing a positive opinion about a film does not necessarily imply having watched it. We therefore treat this case as a disagreement requiring further examination, rather than a confirmed auditor error. These examples illustrate turn-level judgments, not independent validation of individual audit flags. Scope and limitations. The equal 35/35/3535/35/35 allocation, together with at most one sampled turn per historical ReDial user, means that the reported sample accuracies are not unbiased estimates of accuracy over all Stage 4 turns. Consecutive presentation blocks may confound audit stratum with order, learning, or fatigue effects. The ReDial excerpts were retrieved automatically; although matched to historical users, their relevance and completeness were not exhaustively adjudicated, and missing context may affect individual decisions. In addition, the annotators were co-authors and majority vote is an operational human reference rather than an independently established ground truth. The evaluation therefore characterizes human agreement with Stage 4 turn-level outcomes, not the validity of each issue flag or population-level recall. The 1414 three-class disagreements, particularly the five affecting the remove-versus-keep action, remain priorities for further evidence-based adjudication. Appendix H Six Enriched Attribute Schemas This appendix provides the formal schema for each enriched attribute and an example enriched profile. H.1 Schema Definitions Aspect-level preferences (Radlinski et al., 2019). A nested object with three sub-fields. Each entry is an (anchor, text) pair where anchor â Movie, Genre, Person, Other: âą liked: aspects/items the user expressed positive sentiment toward âą disliked: aspects/items the user expressed negative sentiment toward âą descriptions: factual or descriptive statements the user made about specific items Pickiness (Yoon et al., 2024). Deterministically computed from the rejection rate r, not LLM-extracted: pickiness_score =5=5 if râ„0.40râ„ 0.40, 44 if râ„0.25râ„ 0.25, 33 if râ„0.10râ„ 0.10, 22 if râ„0.05râ„ 0.05, and 11 otherwise. Decision-making style (Scott and Bruce, 1995; Hayati et al., 2020). Object: style, evidence, where style â rational, intuitive, dependent, avoidant, spontaneous. Viewing motivation (Oliver and Raney, 2011). Object: hedonic_score, primary_motivation, evidence, where hedonic_score â[1,5]â[1,5] (11 = purely eudaimonic, 55 = purely hedonic) and primary_motivation â hedonic, eudaimonic, escapism, social, catharsis, arousal. Novelty preference (Tang and Liao, 2022). Object: novelty_score, evidence, where novelty_score â[1,5]â[1,5]. Domain expertise (Kostric et al., 2025). Object: level, evidence, where level â novice, intermediate, cinephile. H.2 Example Enriched Profile (User 468) User 468âs enriched profile after Stage 5 (interview decision accept), with values shortened for readability. user_id: 468, aspect_preferences: liked: [ Movie: âJohn Wickâ, Genre: âcrime thrillerâ, Genre: âaction movieâ, Other: ânon-stop action and style of choreographyâ, Other: âhigh-energy pace and relentless energyâ, Other: âphysical set pieces and choreography unfoldâ ], disliked: [], descriptions: [ Movie: âCrouching Tiger, Hidden Dragon rooftop chase lays out history and skill through movementâ, Other: âaction can do storytelling through style and movementâ, [...] ] , pickiness_score: 1, // from rejection rate decision_making_style: style: ârationalâ, evidence: âthe steady momentum of a good crime thriller or action movie where the pacing never really lets upâ , viewing_motivation: hedonic_score: 4, primary_motivation: âarousalâ, evidence: ârelentless energy keeps you hooked from the start without any wasted momentsâ , novelty_preference: novelty_score: 4, [...] , domain_expertise: level: âintermediateâ, evidence: âthe action itself is doing the storytelling through the style and movementâ The basic profile that conditions StatisticalOnly contains only the Stage 1â2 fields (liked/disliked movie lists, acceptance rate, reference style, genre distribution, decade, exploration ratio, popularity, and pickiness score). The six enriched attributes above are absent in StatisticalOnly: aspect_preferences, decision_making_style, viewing_motivation, novelty_preference, and domain_expertise are all null. Appendix I Example End-to-End Simulated Conversations This appendix shows conversations between the simulator and the GPT-4o-mini CRS for the same user (468) under Full and StatisticalOnly, both with the mood opening strategy. I.1 Full (opening = mood) User: Iâm chasing a real adrenaline rush tonight, so I need an action thriller with relentless pacing and some genuinely tight choreography. CRS: You might enjoy âJohn Wick (2014)â [âŠ]. If youâre looking for something a bit different, âThe Raid: Redemption (2011)â delivers incredible martial arts and relentless pacing. User: Iâve already seen John Wick, but The Raid: Redemption sounds right up my alley with that martial arts choreography and pacing. Letâs go with that one. [âŠ] User: That was fantastic, exactly the kind of relentless pacing and fight choreography I was looking for. Got anything else along those lines with high energy? I.2 StatisticalOnly (same user, opening = mood) User: Iâm in the mood for some intense action tonight. Something with a good crime or thriller plot. CRS: How about âHeat (1995)â? [âŠ] User: Heat sounds good. Iâl watch that one. [âŠ] User: Die Hard with a Vengeance is a good one. I like those kinds of action films. I.3 Observations The Full simulator voices the aspects recorded in its enriched profile (pacing, choreography), which the T3 judge extracts as distinct aspects; the StatisticalOnly simulator refers mainly to genres and titles. Across this userâs four conversations, T3 retains eight distinct aspects for Full (e.g., pacing, martial arts choreography, intensity; filtered entropy 1.751.75) but only two for StatisticalOnly (plot and a genre label; entropy 0.690.69), where 1111 of 1313 extracted aspects were generic and filtered out. The same profile vocabulary also explains the T4 pattern: ârelentlessâ appears in three of the four Full opening requests. Appendix J Full System Prompts This appendix reproduces the prompts of all LLM-based components in condensed form. Wording is kept verbatim where shown; â[âŠ]â marks omitted examples or formatting instructions (full versions are in the released codebase). Text in braces (e.g., liked_movies) denotes slots filled at run time. J.1 Interviewer Agent (Stage 3) Each interview has ten messages: an Account phase (six messages) followed by a Clarify phase (four messages). The first interviewer message is not generated: it is one of four predefined open-ended questions, selected uniformly at random: 1. âTell me about your relationship with movies â when you sit down to watch something, what kind of experience are you usually looking for?â 2. âWalk me through the last movie you really enjoyed â what was it and what stuck with you?â 3. âIf a friend asked you to recommend a movie that captures your taste, what would you pick and why?â 4. âWhen youâre choosing what to watch on a typical Friday night, what goes through your mind?â All subsequent interviewer messages are generated (temperature 0.70.7) from the interview so far only; the interviewer never receives the userâs profile or ReDial history. Account phase. You are an interviewer conducting a conversation about movie preferences. Your goal is to understand how this person thinks about and chooses movies --- their taste, their reasoning, the patterns in what they like. CRITICAL CONSTRAINTS: - You do NOT know anything about this person beforehand. You have no profile, no history, no list of their favorite movies. - You must NOT make assumptions about what kind of movies they like based on stereotypes or popular taste patterns. - Every question should follow naturally from what they have ACTUALLY told you in this conversation. QUESTIONING STYLE FOR THIS STAGE (ACCOUNT): - This is the OPENING phase. Let them talk freely. - Ask broad, inviting questions that let them set the direction. - When they mention a specific movie, genre, or feeling, follow up on THAT thing rather than pivoting to a new topic. - Do NOT challenge or contradict. Avoid leading questions. - Do NOT introduce movies, directors, or genres they did not mention. CONVERSATION SO FAR: conversation_history Output ONLY your next question. Clarify phase. You are continuing the interview from earlier. [...] Your job now is different: you are NOT collecting more topics --- you are pushing for the WHY behind what they have already said. PREVIOUS DIALOGUE (Stage 1): stage_1_dialogue WHAT TO ASK IN THIS STAGE: Pick ONE or TWO things the user said that have a specific texture --- a particular movie title, a phrase they repeated, a feeling they named. For each, ask a question that pushes for one of these: - The MECHANISM. They said they liked X --- what specifically about X works for them? - A CONTRAST. They expressed a general preference --- when does that preference NOT hold? - The PERSONAL MEANING. They used a word or phrase --- what does it mean to them concretely? WHAT NOT TO DO: - Do NOT ask the same kind of question Stage 1 already asked. - Do NOT zoom out to abstractions. - Do NOT introduce movies, directors, or genres they did not mention. - Do NOT challenge or contradict. - Use THEIR specific vocabulary when probing. [... good and bad probe examples ...] CONVERSATION SO FAR: full_conversation_history Output ONLY your next question. J.2 Interviewee Agent (Stage 3) The interviewee (temperature 0.60.6) receives the userâs statistical profile and one historical ReDial conversation in which the user acted as a seeker. You are participating in a conversation about your movie preferences. Someone is interviewing you to understand your taste in movies. Respond naturally, as yourself. STATISTICAL PROFILE: - Rejection rate: rejection_rate (you accept about accept_pct% of recommendations on average) - Movies youâve liked: liked_movies - Movies youâve disliked: disliked_movies - You tend to talk about movies in terms of: dominant_ref_style - You prefer movies of popularity level: popularity_score - Your genre preferences: genre_preferences YOUR TASTE PATTERN: taste_summary YOUR ACTUAL CONVERSATION HISTORY: redial_conversation GROUNDING RULES (most important): 1. Everything you say must be CONSISTENT with the profile and history above. 2. When you mention specific movies, prefer movies from your liked/disliked lists or movies that came up in your past conversation. 3. When you give reasoning, the reasoning should be consistent with your profile patterns. 4. Do NOT invent specific personal stories [...] that contradict or go beyond your profile. 5. It is OK and encouraged to say âIâm not sureâ [...] when a question pushes beyond what your profile and history can plausibly answer. 6. Do NOT recite your statistical profile out loud. SPECIFICITY RULES: - Avoid generic emotional phrases like âemotional connectionâ [...]. - When asked WHY you like a film, name a specific element: a scene, a line, a moment, a character choice, a visual style. - Anchor comparisons in concrete examples from YOUR liked or disliked lists rather than abstract categories. CONVERSATION STYLE: first person; 2--4 sentences; avoid absolute statements unless your profile genuinely supports them. [...] CONVERSATION SO FAR: conversation_history When the audit decision is regenerate, the interview is regenerated (up to two retries) with the following nudge appended to the interviewee prompt: ADDITIONAL GUIDANCE FROM A PREVIOUS ATTEMPT: In your previous attempt at this conversation, you made claims that were not fully grounded in your profile or history. Be particularly careful to: - avoid absolute statements unless your profile supports them - avoid inventing specific personal anecdotes - prefer âIâm not sureâ over fabricated detail J.3 Audit Agent (Stage 4) The Audit Agent combines a deterministic structural check with three separate LLM calls (Claude Haiku 4.5, temperature 0.20.2). Only interviewee turns are audited. (a) Structural check (deterministic). Verifies that the interview has ten turns, that interviewer and interviewee strictly alternate, and that no turn is empty. Any violation yields a high-severity structural_error, and the LLM checks are skipped. (b) Movie mentions vs. profile. Extract all movie titles mentioned by the USER (not the interviewer) in this dialogue. For each movie, determine the sentiment expressed about it. [...] Classify sentiment: âpositiveâ, ânegativeâ, or âneutralâ. A movie mentioned positively but listed as disliked in the profile, or negatively but listed as liked, is flagged deterministically as a high-severity history_contradiction. (c) Contradiction with the ReDial conversation. You are checking whether a recent dialogue contradicts an earlier conversation by the same user. EARLIER CONVERSATION: redial_conversation RECENT DIALOGUE TO CHECK: dialogue_text TASK: For each statement made by the USER [...], check whether it directly contradicts something they said in the earlier conversation. Distinguish carefully: - CONTRADICTION: directly opposing statements - REPHRASING / ELABORATION / UNRELATED: NOT a contradiction Only flag CONTRADICTION. Output: turn_index, user_statement, redial_reference, explanation, confidence (low|medium|high). Each contradiction becomes a history_contradiction flag whose severity equals the reported confidence. (d) Fabrication and caricature. You are auditing a dialogue where a user discussed their movie preferences. The user was supposed to respond consistently with the profile and prior conversation history below. [...] CATEGORY 1 --- FABRICATION A specific factual claim that is neither in the profile nor plausibly inferable from it. The bar for flagging is HIGH --- only flag specific, ungrounded claims. DO flag: personal anecdotes with specific people, places, or dates; claims about working in the film industry, attending specific festivals, knowing directors; specific verifiable facts the user could not plausibly know. DO NOT flag: plausible elaboration on a movie in the profile; general emotional reactions; vague time references; hedged statements. CATEGORY 2 --- CARICATURE A response that performs a stereotype rather than reflecting the actual profile. The test: would someone with THIS specific profile plausibly say this, or is this what an LLM imagines a âtype of personâ would say? DO flag: absolute statements that contradict profile nuance; reciting tropes about a âtype of personâ; extreme rejection or acceptance beyond what the rejection rate supports; a snob/cinephile persona without profile support. DO NOT flag: proportionate strong opinions; specific personal preferences; self-aware statements; mixed feelings. Output: for each issue, turn_index, claim or quote, reason, and severity (low|medium|high). Decision rule (deterministic). Any structural_error or â„3â„ 3 high-severity flags yield regenerate; at least one high-severity or â„3â„ 3 medium-severity flags yield partial, in which case all high- and medium-severity interviewee turns are removed from the enrichment input; otherwise accept. J.4 Profile Enrichment (Stage 5) A single Tencent Hy3 call (temperature 0.10.1) receives the audited interview and the userâs Stage 2 profile as background context. You are an expert in user modeling for movie recommender systems. [...] You are given two inputs: 1. The userâs existing profile data [...] 2. A movie interview dialogue with that same user. Use the profile data as background context to better interpret what the user says in the dialogue. Extract exactly 6 profile fields based on what the user says in the dialogue. USER PROFILE (background context): profile_context INTERVIEW DIALOGUE: dialogue_text FIELD DEFINITIONS: 1. aspect_preferences: items of anchor: Movie|Genre|Person|Other, text, grouped into liked, disliked, and descriptions (neutral observations). Extract only from what the user says. 2. pickiness: DO NOT EXTRACT from dialogue [...]. Set to null. 3. decision_making_style: style: rational|intuitive|dependent|avoidant|spontaneous, evidence 4. viewing_motivation: hedonic_score: 1--5 (eudaimonic â hedonic), primary_motivation: hedonic|eudaimonic|escapism|social|catharsis|arousal, evidence 5. novelty_preference: novelty_score: 1--5 (conservative â explorer), evidence 6. domain_expertise: level: novice|intermediate|cinephile, evidence [... definitions of each category and score level ...] RULES: - Extract only from the DIALOGUE, not from the profile context - Use the profile context only to resolve ambiguity or calibrate scores - If a field cannot be determined from the dialogue, set it to null - Do not invent information not present in the dialogue Return ONLY this JSON [...] The pickiness score (11â55) is assigned deterministically from the userâs rejection rate r: 55 if râ„0.40râ„ 0.40, 44 if râ„0.25râ„ 0.25, 33 if râ„0.10râ„ 0.10, 22 if râ„0.05râ„ 0.05, and 11 otherwise. J.5 User Simulator (Stage 6) Opening message. For each of the four opening strategies, the simulator first generates the opening request (temperature 0.70.7): You are generating the OPENING MESSAGE for a user about to start a conversation with a movie recommender. USER PROFILE: profile_context OPENING STRATEGY: strategy Generate ONE opening message (1--3 sentences) that: - Uses the strategy specified above - Reflects this userâs decision style (decision_style) and expertise level (expertise_level) - Is specific enough to give the recommender something to work with [...] - Sounds natural, like a real person typing Strategies: aspect (signal one or two specific aspects from your liked-aspects list); movie (anchor on a movie from your liked list and ask for something similar); mood (describe the mood you want the movie to match tonight); expertise (signal domain knowledge before asking for something specific). [... example phrasings per strategy; do not copy any example verbatim ...] Simulator. Subsequent user turns (temperature 0.70.7) are generated with: You are simulating a real person seeking movie recommendations from a recommender system. profile_context CRITICAL BEHAVIORAL RULES: 1. You are the USER. NEVER take the role of the recommender. [...] 2. Do NOT prefix your messages with any role label. 3. Do NOT recite your profile aloud. Your preferences emerge through your reactions, not declarations. 4. Stay consistent with your profile across all turns. 5. Keep responses 1--3 sentences typically. 6. pickiness_rule_block 7. Do not invent new movies youâve watched beyond your profile. The pickiness block depends on the pickiness score: agreeable for scores â€2†2 (âDefault to accepting recommendations unless they obviously contradict your stated preferencesâ), moderate for 33 (âAccept recommendations that match at least TWO of your stated preferencesâ), and picky for â„4â„ 4 (âReject recommendations that fail to match your stated preferences in specific ways [âŠ] state the specific reasonâ). Under StatisticalOnly, profile_context omits the enriched attributes. CRS (GPT-4o-mini, temperature 0.50.5). You are a conversational movie recommender system. [...] Ask clarifying questions when the userâs request is vague; recommend 1--3 specific movies with brief reasoning when you have enough information; provide titles in the format âTitle (Year)â; adapt to user feedback; keep responses concise (2--4 sentences typically). After each CRS turn, starting from the fourth, the termination judge (§J.6) classifies the last four messages; any label other than continue ends the conversation, which is otherwise capped at eight CRS turns. J.6 Evaluation Judges In the primary experiment, all judges use Claude Haiku 4.5 at temperature 0.00.0. The termination judge operates during simulation; the others operate on the stored conversations. In the separate cross-CRS experiment, GPT-4o-mini serves as the termination judge and performs the post-hoc conversation audit before T3 extraction, without access to the extraction outputs; separate GPT-4o-mini and DeepSeek-chat calls then evaluate T3 (Appendix E). Termination judge (Stage 6). Classify whether the following conversation has reached a natural endpoint. [...] Output EXACTLY ONE of these labels: - true_acceptance: user explicitly endorses a SPECIFIC recommendation with positive language - resigned_acceptance: user acknowledges a recommendation without enthusiasm or specificity - explicit_giveup: user explicitly disengages or expresses frustration - other: conversation has ended but fits none of the above - CONTINUE: user is still exploring or asking RULES: true_acceptance requires BOTH a specific title AND clearly positive language; do NOT output it if the user asks a follow-up question. CONVERSATION (last 4 messages): last_messages Movie mention extraction (T1, T2, T3). Identify every distinct movie that appears in the conversation. For each one, return: title (canonical); year; introduced_by (âuserâ if the USER first mentioned this movie, âcrsâ if the CRS introduced it); first_user_mention_turn_index; all_turn_indices. Do NOT include person names, countries, or generic words. Do NOT include hypothetical references like âa comedyâ. Deduplicate surface forms. [...] Rating inference (T2). Infer how much the user would rate a specific movie. Assign a rating ONLY when the user expresses an actual opinion or evaluative reaction about this movie itself. REFERENCE-ONLY RULE: if the user mentions this movie ONLY as a reference point or comparison anchor (e.g., âsomething like Xâ), the rating MUST be null. Scale: 5.0 loved / accepted enthusiastically; 4.0 clear positive interest; 3.0 neutral; 2.0 mild disinterest; 1.0 explicitly rejected (half points allowed). Output: rating, confidence: high|medium|low, evidence. Aspectâsentiment extraction (T3). You are extracting aspect-sentiment pairs from a userâs opinion about a movie. [...] Extract ALL aspect-sentiment pairs the user mentions. Aspects are concrete elements of the movie: plot, acting, pacing, cast, soundtrack, cinematography, dialogue, ending, characters, mood, atmosphere, etc. For each pair output: aspect (short noun phrase); sentiment (positive|neutral|negative); evidence (short quote from the user). If user expressed no specific aspect opinions, output no pairs. A post-processing step removes generic placeholders (movie(s), film(s), overall movie, overall film, story/stories, thing(s), stuff, experience) before computing aspect entropy. Acceptance/rejection classification. For each movie the CRS explicitly recommended, classify how the USER responded to it. ONLY classify movies the CRS recommended. - accept: clear positive intent to watch, enthusiasm, or agreement to try it (âsounds greatâ, âIâl watch thatâ) - reject: explicit decline, clear disinterest, or dismissal (ânot really my thingâ, âno thanksâ) - neutral: ambiguous, hedging, or absent reaction (âmaybeâ, no direct response) Appendix K Model Configuration and API Cost Table 11 lists the model, decoding temperature, and output-token limit of every LLM component in the primary experiment. Reasoning was disabled for Gemini 2.5 Flash and enabled (effort high) for Tencent Hy3; all other models were used without reasoning. Component Model Temp. Max tok. Interviewer (S3) Gemini 2.5 Flash 0.7 512 Interviewee (S3) Gemini 2.5 Flash 0.6 512 Audit: movie check (S4) Claude Haiku 4.5 0.2 512 Audit: contradiction (S4) Claude Haiku 4.5 0.2 512 Audit: fabr./caric. (S4) Claude Haiku 4.5 0.2 1,024 Enrichment (S5) Tencent Hy3 0.1 8,192 Opening message (S6) Gemini 2.5 Flash 0.7 256 User simulator (S6) Gemini 2.5 Flash 0.7 256 CRS (S6) GPT-4o-mini 0.5 256 Termination judge (S6) Claude Haiku 4.5 0.0 20 Movie extraction (T1âT3) Claude Haiku 4.5 0.0 2,000 Rating inference (T2) Claude Haiku 4.5 0.0 256 Aspectâsentiment (T3) Claude Haiku 4.5 0.0 2,048 Accept/reject Claude Haiku 4.5 0.0 1,000 CSHI simulator Gemini 2.5 Flash 0.0 2,048 Table 11: LLM components of the primary experiment. S3âS6 denote pipeline stages. The CSHI baseline uses the same CRS configuration. API cost. Table 12 reports the provider-reported API cost of each run, split into generation (Stages 3â6, including the CRS) and evaluation (all judges). Generating the full Full cohort cost $8.30, i.e., about $0.028 per user including four simulated conversations; profile enrichment alone (Stages 3â5) cost $4.29 ($0.014 per user). Evaluation dominates the total cost, chiefly through per-movie rating inference for T2. StatisticalOnly and CSHI involve no interviews or enrichment, so their generation cost covers only user simulation and the CRS. Run Calls Tokens Gen. Eval. Total in/out (M) ($) ($) ($) Full (300) 32,907 28.7 / 4.4 8.30 23.85 32.15 StatOnly (100) 11,149 8.5 / 0.9 1.41 8.93 10.34 CSHI (100) 16,589 14.8 / 1.4 1.37 16.90 18.27 Total 60,645 52.1 / 6.8 11.08 49.68 60.76 Table 12: API usage and cost (USD) per run of the primary experiment. Hy3 output tokens include reasoning tokens. Numbers in parentheses give the number of users.