Paper deep dive
From Memory to Behavior: A Behavior-Aware Role-Playing Framework for Social Media Influencers
Ji-Lun Peng, Yi-Zhen Zhang, Chun-Nan Chou, Yun-Nung Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 9/21/2026, 4:18:36 AM
Summary
The paper introduces SIBPersona, a behavior-aware role-playing framework for social media influencers that models situation-dependent behavioral strategies using a Situation-Internal state-Behavior (SIB) triplet. It addresses gaps in existing in-context learning methods by incorporating reaction processes and proposes a reference-augmented LLM evaluation protocol to handle obscure individuals, demonstrating superior performance over baselines on a newly constructed dataset.
Entities (30)
Relation Signals (19)
SIBPersona → basedontheory → Cognitive-Affective Processing System
confidence 95% · Inspired by the Cognitive-Affective Processing System (Mischel and Shoda, 1995), we propose Situation–Internal state–Behavior Persona (SIBPersona)
Reaction Process → composedof → SIB Triplet
confidence 95% · We call this three-tuple, namely⟨s,i,b⟩, an SIB triplet.
SIBPersona → models → Reaction Process
confidence 95% · we propose to incorporate so-called reaction process, the third dimension, into the data-variant persona.
SIBPersona → outperforms → Vanilla RAG
confidence 95% · Experimental results show that our proposed method outperforms state-of-the-art ICL-based baselines
SIBPersona → outperforms → IMPersOna
confidence 95% · Experimental results show that our proposed method outperforms state-of-the-art ICL-based baselines
SIBPersona → addressesgap → LLM Familiarity Gap
confidence 90% · LLM Familiarity Gap: ...direct prompting strategy is less applicable to obscure real individuals
SIBPersona → addressesgap → Behavioral Modeling Gap
confidence 90% · Behavioral Modeling Gap: Existing in-context learning (ICL)- based role-playing methods... fail to capture how an individual tends to react
SIB Triplet → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models have shown strong potential as role-playing agents for real individuals, yet faithful impersonating remains challenging. Existing in-context learning-based methods fail to capture how individuals react under different situations. In addition, LLM-based evaluation is difficult for obscure individuals. To address these challenges, we propose Situation--Internal state--Behavior Persona method to incorporate situation-dependent behavioral strategies. We further design an evaluation protocol that provides LLM evaluators with references about the impersonated individual. We evaluate our approach on a newly constructed dataset for the task of generating replies on social media. Experimental results show that our proposed method outperforms state-of-the-art ICL-based baselines, while our evaluation protocol achieves moderate correlation with human judgment. Besides, experiments on fictional-character benchmarks demonstrate that our proposed method is applicable beyond the social media setting. These findings suggest that incorporating behavioral information broadly improves the fidelity of role-playing for real individuals on social media or fictional characters.
Tags
Links
- Source: https://arxiv.org/abs/2609.21349v1
- Canonical: https://arxiv.org/abs/2609.21349v1
Trouble viewing inline? Open PDF directly →
Full Text
127,815 characters extracted from source content.
Expand or collapse full text
From Memory to Behavior: A Behavior-Aware Role-Playing Framework for Social Media Influencers Ji-Lun Peng ⋆† Yi-Zhen Zhang † Chun-Nan Chou † Yun-Nung Chen ⋆ ⋆ National Taiwan University, Taipei, Taiwan † CMoney Technology Corporation, Taiwan r13946006@ntu.edu.tw jason_zhang,jason_chou@cmoney.com.tw y.v.chen@ieee.org Abstract Large language models have shown strong po- tential as role-playing agents for real individu- als, yet faithful impersonating remains chal- lenging. Existing in-context learning-based methods fail to capture how individuals react under different situations. In addition, LLM- based evaluation is difficult for obscure individ- uals. To address these challenges, we propose Situation–Internal state–Behavior Persona method to incorporate situation-dependent be- havioral strategies. We further design an eval- uation protocol that provides LLM evaluators with references about the impersonated indi- vidual. We evaluate our approach on a newly constructed dataset for the task of generating replies on social media. Experimental results show that our proposed method outperforms state-of-the-art ICL-based baselines, while our evaluation protocol achieves moderate corre- lation with human judgment. Besides, ex- periments on fictional-character benchmarks demonstrate that our proposed method is appli- cable beyond the social media setting. These findings suggest that incorporating behavioral information broadly improves the fidelity of role-playing for real individuals on social me- dia or fictional characters. 1 1 Introduction Recent advances in large language models (LLMs) have accelerated the development of role-playing agents (RPAs) capable of impersonating both fic- tional characters (Yu et al., 2025; Zhang et al., 2025) and real individuals (Chuang et al., 2025; Wang et al., 2025a). Recent evidence further sug- gests that RPAs for real individuals can produce convincing impersonations (Shi et al., 2025). Toward faithful role-playing of specific real indi- viduals, two major challenges remain. Behavioral Modeling Gap: Existing in-context learning (ICL)- based role-playing methods mainly provide RPAs 1 Code: https://github.com/MiuLab/SIBPersona with factual information about impersonated indi- viduals extracted from historical conversations (Liu et al., 2024; Shi et al., 2025; Gao et al., 2025). How- ever, such information does not explicitly capture how an individual tends to react under different sit- uations. As illustrated in Figure 1, in social media comment-replying scenarios between influencers and followers, replies are often brief or knowledge- light, making faithful impersonation depends more on behavioral strategies than on factual informa- tion. LLM Familiarity Gap: Existing LLM-based evaluation protocols for role-playing are largely de- veloped for fictional characters, whose information are often represented in LLMs’ knowledge (Peng and Chen, 2026). As a result, directly prompting an LLM can provide a reasonable evaluation (Wang et al., 2024; Yu et al., 2025). By contrast, this direct prompting strategy is less applicable to ob- scure real individuals, whose information may be sparsely represented in LLMs’ pre-training data. This limited familiarity makes it difficult to directly use LLMs for automated evaluation of role-playing real-individual. To address these challenges, we develop a behavior-aware framework for role-playing real individuals. Inspired by the Cognitive-Affective Processing System (Mischel and Shoda, 1995), we propose Situation–Internal state–Behavior Per- sona (SIBPersona), a method that models indi- viduals’ situation-dependent behavioral strategies. In addition, we design an automated LLM-based evaluation protocol tailored to role-playing real- individual, which provides the LLM evaluator with reference about the impersonated individual to alle- viate its limited familiarity with the individual. We apply this framework to the domain of social media by constructing a dataset of influencer–follower in- teractions. Using this dataset, we evaluate RPAs on their ability to impersonate influencers, specifically in generating replies to followers’ comments. The contributions of this paper are three-fold: arXiv:2609.21349v1 [cs.CL] 18 Sep 2026 Post In my previous post, I specifically highlighted "Stock-A". If you had seen it, you wouldn't have entered the market only to be trapped until now. ... I clearly posted a tutorial and a warning, yet there are still so many retail investors acting like moths to a flame. Comment Monday-morning quarterback. Follower Then just don’t read my posts. The influencer typically responds to criticism from detractors by offering direct rebuttals. Influencer Why do you feel that way? RPA The reply rarely requires factual knowledge about the influencer; instead, it depends on the influencer’s behavioral tendencies when dealing with detractors. An incorrect reply strategy leads to low fidelity. Reply Figure 1: In the social media comment-replying scenar- ios between influencers and followers, replies involve short texts where behavioral strategy is more important for determining the fidelity of impersonation. •We present a structured view of persona for RPAs by distinguishing data-invariant and data-variant personae. Building on this view, we propose SIBPersona, which introduces re- action process as a novel dimension of data- variant persona to capture situation-dependent behavioral strategies. •We establish an reference-augmented evalua- tion protocol for role-playing of obscure in- dividuals. The protocol achieves moderate correlation with human judgment. •We conduct comprehensive experiments across multiple influencers, model families, and fictional-character benchmarks, show- ing that SIBPersona consistently outperforms state-of-the-art ICL-based baselines. 2 Related Work Approaches of LLM Role-Playing Two main- stream approaches enable LLMs to impersonate specific targets: Fine-Tuning (FT) (Yang et al., 2026; Ye et al., 2025) and ICL. We focus on ICL-based methods because persona information is explicitly represented in prompts, making the role-playing process easier to interpret and mod- ify. Existing ICL-based methods utilize retrieval- augmented generation (RAG) (Lewis et al., 2020) to incorporate context-relevant information into prompts (Shi et al., 2025; Gao et al., 2025). How- ever, the retrieved content primarily consists of raw conversational data or factual knowledge. Such approaches largely emphasize what an individual knows rather than how an individual tends to be- have across different interaction situations. To address this gap, we draw on the Cognitive- Affective Processing System (CAPS) (Mischel and Shoda, 1995), which models behavior as a dynamic reaction process. According to CAPS, individuals perform behaviors by processing situational inputs through internal states. This perspective suggests that RPAs should model the situation-dependent reaction process. Nonetheless, how to explicitly model such a dynamic reaction process in ICL- based method remains underexplored. Datasets for Role-Playing of Real Individuals Existing role-playing datasets primarily focus on fictional characters (Xu et al., 2025, 2026), while datasets for role-playing real individuals remain relatively limited.To study role-playing real- individual, we turn to social media, which provides rich conversational records for observing how an individual responds across diverse situations (Li et al., 2025). To support our research goal of im- personating influencers, the dataset must satisfy two requirements: (1) it should contain abundant influencer-centered interactions from a single iden- tifiable individual, allowing LLMs to obtain com- prehensive information about the impersonated in- dividual; and (2) it should preserve the comment- reply structure under the influencer’s posts, which is essential for observing how the individual re- sponds to different situations. As summarized in Table 1, no existing dataset satisfies both requirements simultaneously. Among the public datasets, BluePrint (Bück-Kaeffer et al., 2025) SYNTHIA (Rahimzadeh et al., 2026), Per- sonalDialog (Zheng et al., 2019), and Persona-Chat (Zhang et al., 2018) do not preserve an identifiable individual’s history: BluePrint anonymizes users and aggregates interactions into clusters, while SYNTHIA provides only synthesized persona at- tributes rather than raw, individual-level social me- dia interactions. PER-CHAT (Wu et al., 2021) does contain large-scale, identifiable individual histories (314, 749individuals), but lacks the comment-reply structure needed to observe situational responses. Conversely, PersonaConvBench (Li et al., 2025) preserves this structure but provides a limited num- ber of samples per individual (6.87posts and37.05 comment-reply pairs on average), making it insuf- ficient for LLMs to obtain comprehensive infor- mation about the impersonated individual. These limitations motivate us to construct a new dataset tailored for the task of generating replies on social media. DatasetPublic Identifiable Individual History #Individual Comment- Reply Structure Avg. Posts per Individual Avg. Comment-Reply Pairs per Individual Max Posts per Individual Max Comment-Reply Pairs per Individual BuzzFace (2018)✗------- TWON (2026)✗------- BluePrint (2025)✓✗------ SYNTHIA (2026)✓✗------ PersonalDialog (2019)✓✗------ Persona-Chat (2018)✓✗------ PER-CHAT (2021)✓314749✗---- PersonaConvBench (2025)✓2796✓6.8737.05111414 Ours✓6✓18115554290315240 Table 1: Feasibility of existing datasets for studying role-playing of real individuals on social media reply generation. ”-” indicates that the corresponding information is not applicable under our comparison criteria. Since our research goal is to impersonate influencers across diverse interaction situations, we construct a dataset centered on a small number of influencers with rich historical interactions. Evaluation of RPAs LLM evaluation have be- come widely adopted for evaluating role-playing of fictional character (Gu et al., 2026), as they enable fine-grained assessment of performance (Wang et al., 2025b, 2024; Shao et al., 2023). These ap- proaches typically assume that the evaluator model possesses sufficient knowledge about the imperson- ated character (Peng and Chen, 2026). Evaluating role-playing of real individuals re- mains challenging because LLM evaluators have limited knowledge about obscure people. Conse- quently, prior studies on role-playing real individu- als often rely on human evaluation (Shi et al., 2025; Tu et al., 2023), while automated evaluation pro- tocols for role-playing of obscure real individuals remain underdeveloped. 3 SIBPersona In this section, we present SIBPersona method designed for role-playing of influencers on social media. We first introduce our multifaceted view of persona and define the reaction process in §3.1, fol- lowed by formalizing the task of generating replies on social media in §3.2. Subsequently, we elabo- rate how data-variant persona is constructed from historical data in §3.3. Finally, we explain how the constructed data-variant persona is retrieved and used for reply generation in §3.4. 3.1 Multifaceted Nature of Persona In RPA, a "persona" is always referred to the infor- mation describing fictional characters or real indi- viduals. We argue that personae can be divided into two types: data-invariant and data-variant. The data-invariant persona, as the name implies, is pro- vided in advance and is not derived from the data. For example, we can create a narrative to describe the demographic information of the influencer or the fictional traits of the fictional character, as far as we know. On the other hand, the data-variant persona is derived from the data and designated to capture the dynamics of RPA. Apart from deriving from the data or not, the prior methods mainly modeled the persona by two dimensions: 1) memory (Shi et al., 2025) capturing the factual information and 2) speaking style (Tu et al., 2024) capturing expression habits. However, we contend that these two dimensions are insuffi- cient for RPAs. As suggested in CAPS, responses of real people are shaped not only by what they know and how they speak but also by the reaction process they tend to follow in a given situation. Reaction Process In order to resolve this issue, we propose to incorporate so-called reaction pro- cess, the third dimension, into the data-variant per- sona. To this end, we model reaction process as the mapping of a given situation to an internal state and subsequently to a behavioral strategy. Formally, re- action process is defined by⟨s,i,b⟩∈ S×I×B | s = LLM(x),i = LLM(x),b = LLM(x), whereScontains situations,Irepresents the set of internal states,Bis the set of behavioral strategies, andxdenotes a text input. We call this three-tuple, namely⟨s,i,b⟩, an SIB triplet. In the following, we utilize the example that influencers reply fol- lowers’ comments on social media to illustrate the idea of reaction process. When influencers reply to followers’ comments, they first interpret the com- ments into high-level situations, e.g., praise, in- quiry, or hostility. This emerging situation then triggers a specific internal state, such as happy or angry, which leads to a certain behavioral strategy, such as expressing gratitude or defensive rebuffing. Grounded in CAPS, this three-dimensional data- variant persona provides a more psychologically faithful modeling of how real individuals respond. 3.2 Problem Formulation After illustrating the three-dimensional data-variant persona, we provide the formal settings of the prob- lem for easy exposition. In general, RPAs are de- signed to engage in conversations with users by emulating specific characters or impersonating spe- cific individuals. In this work, we focus on the generation of single-turn replies for influencers on social media. To be specific, we generate the re- plyraccording to the influencer’s postip, the fol- lower’s commentfc, the influencer’s data-invariant profilep, and the influencer’s data-variant persona PS dv , which can be represented as: r = RPA(ip,fc,p,PS dv ). In particular,PS dv contains information about the three aforementioned dimensions. 3.3 Constructing Data-Variant Persona In this subsection, we elaborate on how each dimension is extracted and represented.We first view a data sample associated with a spe- cific influencer asds j = ⟨ip j ,CRP j ⟩, where ip j is the influencer’s post, andCRP j = ⟨fc j1 ,r j1 ⟩,⟨fc j2 ,r j2 ⟩,...,⟨fc jn ,r jn ⟩denotes the set of all comment-reply pairs associated with ip j post. As a consequence, the historical data of this influencer is a collection of the data samples, denoted asD =ds 1 ,ds 2 ,...,ds n . In accordance with the previous works of ex- tracting data-variant personae by LLMs (Shi et al., 2025; Liu et al., 2024), we employ a two-stage pipeline to produce data-variant personae from raw social media histories. Our entire pipeline is de- picted in the upper part of Figure 2 and is explained in detail below. Stage 1: Dimensional Relevance FilteringDue to data noise on social media, we employ a rec- ognizing LLM to filter irrelevant information in the first stage to increase the efficacy of the data pipeline. Specifically, the recognizing LLM acts like a multi-label classifier that assigns each data sample to the set of dimensional labelsL = l M ,l T ,l RP . That is,∀j h(ds j ) = A j A j ⊆ L, wherehis a mapping function from a data sam- ple to the subset ofL. As a shorthand, we use the notationsM,T,RPto represent memory, speak- ing style, and reaction process, respectively. If the recognizing LLM assigns the data sampleds j to the empty set, i.e.h(ds j ) = ∅,ds j is discarded. Otherwise, ds j proceeds to the next stage. Stage 2: Dimensional Document Extraction Given a data sampleds j =⟨ip j ,CRP j ⟩such that h(ds j )̸=∅, we use an extracting LLM to generate the dimensional information ofds j for each label l∈ h(ds j ). We introduce dimensional documents whose details are elaborated later as a countable surrogate for the generated dimensional informa- tion. Givends j and a labell∈ h(ds j ), the extract- ing LLM may generate more than one dimensional document, and we usekto index the generated dimensional documents. To equip our approach with interpretability, we define a subset ofCRP j as a set of evidence pairs denotedEP jk , andEP jk provides the grounding evidence for the generated dimensional informa- tion. If the labellisl M , the extracting LLM gener- ates a memory documentd (l M ) jk = ⟨mp jk ,EP jk ⟩, wheremp jk denotes a memory point capturing factual information. When the labellisl T , the extracting LLM produces a speaking style docu- mentd (l T ) jk =⟨sp jk ,EP jk ⟩, wheresp jk represents a speaking style point reflecting a expression habit. Otherwise, the extracting LLM yields a reaction process documentd (l RP ) jk =⟨s jk ,i jk ,b jk ⟩,EP jk ⟩, where⟨s jk ,i jk ,b jk ⟩constitutes an SIB triplet com- prising the situation, internal state, and behavioral strategy, respectively. Depending on the type of dimensions, we store the extracted dimensional document in the corre- sponding dimensional corpus, which is a collec- tion of dimensional documents. Thus, our data- variant personaPS dv has three dimensional cor- pora: memory corpus, speaking style corpus, and reaction process corpus. These three dimensional corpora lay the foundation for the reply generation illustrated in the next subsection. 3.4 Reply Generation After the data-variant persona is established, one intuitive idea is to generate the reply by consider- ing all dimensional documents in these three di- mensional corpora as the demonstrations to LLMs. However, this idea is inherently infeasible since influencers on social media always have numer- Speaking Style Corpus Memory Corpus Reaction Process Corpus Data-variant persona Data-invariant persona Profile Dimensional Relevance Filtering Dimensional Document Extraction RPA I’l feel much better once you start losing money in the stock market. Influencer A responds to followers’ comments with brief text. Influencer A has hosted livestreams related to stock investing. Influencer Then there’s no need to read my posts. In my previous post, I specifically highlighted "Stock-A". If... Input Influencer Post Monday-morning quarterback Follower Historical data samples I woke up and saw this, haha it’s bounced back... Thank’s for sharing the information. Did you do all of the analysis yourself? I did it myself, thanks. Impressive. Of course. You have great taste too. Influencer A is a financial content creator... Situation: A follower criticized for... Internal State: Influencer A felt the criticism was unreasonable.... Behavior Strategy: Respond in a rebuttal tone... Top-3 Retrieval Dimensional Corpus Construction Reply Generation Impersonate Figure 2: Overview of the SIBPersona method. The upper part illustrates the process of extracting data-variant persona from historical data. The lower part presents reply generation process: retrieving the most relevant dimensional documents based on the comment enables the RPA to generate a high-fidelity reply. ous posts and comment-reply pairs; otherwise, they could hardly be called "influencers." Instead of usingPS dv directly, we utilize the approximate data-variant personaPS dv derived fromPS dv as an alternative. As depicted in the lower part of Figure 2, we utilize the concept of RAG (Lewis et al., 2020) to derivePS dv . Given the queryfc, we indepen- dently retrieve the top three dimensional documents which are most relevant to the queryfcfrom each dimensional corpus. The relevance measure is cal- culated as follows. We treatmp jk ,sp jk , ands jk in the SIB triplet as the retrieval keys for memory corpus, speaking style corpus, and reaction pro- cess corpus, respectively. For each dimensional corpus, we compute the cosine similarity between the embedding of queryfcand the embedding of the retrieval keys. Afterwards, we rank the dimen- sional documents by similarity scores and retrieve the top three corresponding dimensional documents for each dimensional corpus. Finally, the nine re- trieved dimensional documents constitutePS dv , serving as demonstrations to LLM. The problem we address becomes: r = RPA(ip,fc,p, PS dv ). Figure 11 shows an example of a retrieved dimen- sional document and the reply generated by SIBPer- sona. Additional implementation details are pro- vided in Appendix B. 4 Reference-Augmented Evaluation In this section, we propose a reference-augmented protocol for automatically evaluating role-playing agents. We first explain the challenges of this task, then detail our design, and conclude by empirically validating its alignment with human judgment. 4.1 Penalty-Based LLM Evaluator To enable scalability and facilitate future itera- tions, we adopt LLM evaluators to automatically assess role-playing agents for the task of generat- ing replies on social media. However, as addressed in §2, LLM evaluators may have limited knowl- edge of obscure individuals. To resolve this knowl- edge scarcity issue, we augment the LLM evaluator with some references called representative dimen- sional documents about the impersonated individ- ual. The representative dimensional documents are constructed as follows. We first apply the method described in §3.3 to construct the three dimensional corpora from the test samples and then select repre- sentative dimensional documents, denoted asrdd l , from the resulting dimensional corpus for each di- mensionl∈ L. Specifically, we apply K-medoids clustering to select p M/2representative dimen- sional documents from each dimensional corpus, whereMis the size of dimensional documents. Clustering is performed based on the embeddings ofs jk ,mp jk andsp jk . Crucially, the representative dimensional documents are constructed and sam- pled exclusively from the test samples rather than MethodsGPT-5.2Gemini-3-flashDeepSeek-V3.2Claude-Opus-4.6Average MTRPMTRPMTRPMTRPMTRP Naive97.50 46.40 ‡ 67.53 ‡ 94.70 42.47 ‡ 69.27 ‡ 95.13 48.93 ‡ 70.83 ‡ 96.77 46.93 ‡ 76.87 ‡ 96.03 46.18 ‡ 70.78 ‡ CoT 97.47 44.93 ‡ 70.73 ‡ 95.83 41.17 ‡ 69.53 ‡ 96.27 49.23 ‡ 71.47 ‡ 97.33 47.00 77.33 ‡ 96.73 45.58 ‡ 72.27 ‡ IMPersona96.07 47.50 ‡ 71.93 ‡ 94.90 44.60 ‡ 70.10 ‡ 96.00 50.80 ‡ 72.50 ‡ 97.43 48.67 ‡ 78.13 ‡ 96.10 47.89 ‡ 73.17 ‡ Vanilla RAG 97.33 49.43 ‡ 73.20 ‡ 93.77 53.20 71.50 ‡ 94.23 58.37 ‡ 73.4396.70 56.93 77.70 ‡ 95.51 54.48 ‡ 73.96 ‡ SIBPersona 96.90 54.33 77.6393.77 56.47 75.5094.83 63.67 76.4396.93 62.77 81.9395.61 59.31 77.88 Table 2: Performance on Influencer A. Bold: the best results;‡: significant degradation compared to SIBPersona (p < 0.05). MethodsDeepseek-V3.2Gemini-3-flash MTRPMTRP Naive92.10± 3.0442.03 ‡ ± 8.4368.48 ‡ ± 5.8692.50± 2.3243.33 ‡ ± 7.3570.64 ‡ ± 7.37 CoT 92.17± 3.0343.92 ‡ ± 8.1369.83 ‡ ± 6.1492.15± 2.5444.05 ‡ ± 7.0870.48 ‡ ± 6.40 IMPersona91.60± 2.2145.54 ‡ ± 5.8671.14 ‡ ± 6.0291.42± 1.9445.21 ‡ ± 6.4071.56 ‡ ± 7.19 Vanilla RAG91.39± 3.2357.05 ‡ ± 2.4372.43 ‡ ± 5.0692.19± 2.4653.43 ‡ ± 4.9272.59 ‡ ± 4.79 SIBPersona 92.14± 3.2661.09± 1.6675.84± 4.9092.21± 2.2057.34± 3.2076.48± 5.08 Table 3: Results on other five influencers. Bold indicates the better scores.‡: significant degradation compared to SIBPersona (p < 0.05). MethodsMTRP SIBPersona92.67± 3.18 61.60± 1.8275.96± 4.49 w/o I92.61± 2.49 60.30± 2.3775.44± 3.61 w/o RP92.83± 2.44 57.82 ‡ ± 2.51 73.36 ‡ ± 4.64 Table4:Dimensionablationstudyusing DeepSeek-V3.2on the six influencer.Bold: the best results;‡: significant degradation compared to SIBPersona (p < 0.05). the demonstration examples. This ensures that the LLM evaluator’s reference standard remains inde- pendent of the role-playing agent’s demonstration source, so that the evaluation reflects the agent’s true generalization ability rather than its capacity to reproduce seen demonstrations. For the clarity of presentation, we define a test sample asts j = ⟨ip j ,fc j ,r a j ⟩,wherer a j is the reply generated by an RPA. For each dimension l ∈ L, our LLM evaluator takests j ,p,rdd l , and the ground-truth replyr gt j written by the influencer as input. Following Wang et al. (2025c), our LLM evaluator is instructed to identify the flaws inr a j and to assign a severity level ranging from1to 5for each flaw at the same time. The final score for the dimensionlis computed by averaging the scores on all test samples whose size isN, which can be summarized as follows: Score l = 1 N N X j=1 max(0, 100− 5· X f∈F lj v f ), whereF lj is the set of the identified flaws for di- mensionlonr a j , andv f is the severity level of flawf. Additional details regarding the evaluation implementation are provided in Appendix C. 4.2 Alignment with Human judgment To assess whether our reference-augmented eval- uation protocol is well-aligned with human judg- ment, we examine the correlation between human and LLM evaluator scores. For evaluation scores averaged across the three dimensions, the align- ment between human and LLM evaluator yields a Pearson’srof0.561and a Spearman’sρof0.530, indicating a moderate correlation by conventional social-science standards (Cohen, 1988; Lovakov and Agadullina, 2021) and also consistent with the previous findings in role-playing evaluation (Zhou et al., 2025; He et al., 2025). These results demon- strate that our reference-augmented protocol serves as a reasonable proxy for human judgment. Ad- ditional details of the correlation analysis are pro- vided in Appendix D.2, and the advantages of our protocol over traditional NLP metrics are reported in Appendix E.2. 5 Experimental Evaluation In this section, we present the experimental evalua- tion of SIBPersona. We first describe the process of building the dataset in §5.1, followed by the exper- imental settings in §5.2. We then show the results of the experiments conducted on six influencers to demonstrate the effectiveness of SIBPersona in §5.3. To validate our SIBPersona, we perform a human evaluation to examine whether theRPdi- mension provides interpretable evidence in §5.4. Lastly, we show that SIBPersona is also applicable to fictional character benchmarks in §5.5. 5.1 Dataset Construction We construct our dataset from posts, comments, and replies associated with six influencers, denoted as Influencer A to F from CMoney Forum 2 . We collect the data span from January 1, 2024, to September 30, 2025. The data whose dates are before August 1, 2025 belong to the demonstration source, while the remaining data are preserved as the candidate pool for test samples. To have a fair comparison between the different influencers, we perform stratified sampling on the candidate pool and keep the size of the test samples for each influencer in the same order of magnitude. In addition, our preliminary analysis reveals that different comment–reply pairs exhibit distinct inter- action types. To ensure that the test samples thor- oughly cover these diverse interactions, we prompt gpt-4o-minito assign each comment–reply pair a score reflecting the degree of expressing personal characteristic. Based on these scores, the data are partitioned into three levels: high, medium, and low. From each level, we select50test samples at most. Additional details regarding dataset con- struction are provided in Appendix A. 5.2 Experimental Setting We compare our method with several ICL-based role-playing methods, all of which take the im- personated individual’s profilepas default input. We briefly describe each compared method below. Naive relies solely on the impersonated individ- ual’s profile. CoT builds upon the Naive by ad- ditionally incorporating Chain-of-Thought reason- ing (Wei et al., 2022). Vanilla RAG augments the prompt with nine retrieved raw comment–reply pairs to match the number of dimensional docu- ments utilized in SIBPersona. Finally, IMPersona (Shi et al., 2025) is a state-of-the-art ICL-based method for role-playing real individuals in con- versations, which uses the proposed hierarchical memory module to model its data-variant persona. 2 CMoney Forum is the largest online investor community in Taiwan, where members post Traditional Chinese messages under individual stock tickers and follow financial influencers whose posts routinely attract numerous replies. MethodsM T RPAvg. IMPersona2.92 ‡ 2.40 ‡ 2.67 ‡ 2.67 ‡ Vanilla RAG 3.683.723.623.67 SIBPersona3.783.643.623.68 Table 5: Human evaluation results comparing our frame- work with the baseline. Bold: better results; ‡ : signifi- cant degradation compared to SIBPersona (p < 0.05). ConditionsM T RP Uninterpretable3.033.002.87 Interpretable4.13 † 3.97 † 4.00 † Table 6: Human evaluation scores for Interpretable (n=20) and Uninterpretable (n=10) groups. Bold: better results; † : significant improvement with p < 0.05. We conduct two types of experiments to evalu- ate the robustness of SIBPersona along two axes: model choice and impersonated influencer. First, we examine model-agnostic robustness by ap- plying multiple LLMs to construct data-variant personas, includingGPT-5.2,Gemini-3-flash, DeepSeek-v3.2, andClaude-Opus-4.6. These experiments are conducted on Influencer A to as- sess whether the effectiveness of SIBPersona is robust to the choice of the LLM used for persona construction, rather than being tied to a specific model. Second, we examine the robustness of cross-influencers by applying SIBPersona to Influ- encer B through Influencer F. In these experiments, we useGemini-3-flashandDeepSeek-v3.2to evaluate whether SIBPersona remains effective across different influencers. All experiments are evaluated using the reference-augmented evalu- ation protocol introduced in §4 in all three di- mensions, withgpt-4.1as the LLM evaluator andtext-embedding-3-smallas the embedding model for retrieval. For statistical significance test- ing, we perform t-tests with Holm correction for multiple comparisons. 5.3 Main Results Table 2 shows the results of Influencer A in four LLMs. On average, SIBPersona outperforms all baselines by a statistically significant margin on both theTandRPdimensions. Across all meth- ods, performance on theMdimension remains uni- formly high, reflecting that replies on social media naturally demand less factual memory information and thus leave minimal room for penalties. In terms of the other two dimensions, Naive and CoT per- form the worst due to insufficient information re- garding the impersonated individual. While IMPer- sona incorporates rich memory information, it fails on speaking style and reaction process due to its single-dimensional design. Vanilla RAG emerges as the strongest baseline, indicating that simply mir- roring similar historical replies yields reasonable results. Nevertheless, Vanilla RAG still falls short of SIBPersona, showing that constructing struc- tured, multi-dimensional data-variant persona is crucial for faithful role-playing. Table 3 reports the results aggregated over In- fluencer B through Influencer F. SIBPersona con- sistently achieves the highest scores in theTand RPdimensions across all five influencers under both models. Nearly all baselines exhibit statisti- cally significant degradation compared to SIBPer- sona, and the advantage is consistent regardless of model family or impersonated influencer. These results show that SIBPersona is robust along two orthogonal axes: model choice and impersonated individual. From a qualitative perspective, Fig- ure 12 illustrates that SIBPersona better captures the behavioral strategy and speaking style of the impersonated individual facing both praise and crit- icism. Detailed experimental results and qualitative analysis are provided in Appendix E.1 and G. The ablation study further demonstrates the im- portance of theRPdimension. As shown in Ta- ble 4, removing theRPdimension significantly degrades the performance on theTandRPdi- mensions, confirming that theRPdimension con- tributes substantially to overall performance. In contrast, removing only the internal state com- ponent causes marginal, non-significant declines. This asymmetry suggests that while the full SIB triplet is beneficial, explicit behavioral strategies Bcontribute more critically than intermediate internal-stateIfor the social media reply gener- ation task, where replies tend to be short and direct. 5.4 Human Evaluations To validate the effectiveness of SIBPersona, we perform a human evaluation in which the anno- tators rate the replies generated by SIBPersona, Vanilla RAG and IMPersona along three dimen- sions using a five-point Likert scale (Joshi et al., 2015). As shown in Table 5, SIBPersona signifi- cantly outperforms IMPersona across all three di- mensions, while performing comparably to Vanilla RAG. Through qualitative analysis, we observe that Vanilla RAG frequently directly replicates the replies from the retrieved replies. This tendency of- ten leads to misaligned replies when discrepancies arise between the follower’s comment and the re- trieved context. In contrast, SIBPersona generates replies grounded in the structured points, allow- ing it to adaptively fit the current situation. While such fine-grained differences are less distinguish- able under the small-scale human evaluation, the adaptability of SIBPersona is more clearly reflected in our large-scale LLM-based evaluations. Addi- tional details and results of the human evaluation are provided in Appendix D. Interpretability of SIBPersona We further ex- amine the interpretability of SIBPersona. In this type of experiments, the annotators are asked to assess whether the SIB triplets inPS dv , especially the internal state and behavior strategy, can pro- vide reasonable explanations for the generated re- ply given a test sample. Based on these assess- ments, we divide the test samples into interpretable and uninterpretable groups. Additional details of the interpretability experiments are provided in Ap- pendix D.2. As shown in Table 6, the replies in the interpretable group significantly outperform those in the uninterpretable group across all three dimen- sions. This result suggests that the interpretability of the SIB triplets inPS dv influences the quality of the generated replies and subsequently contributes to improved individual fidelity. In addition, we provide a qualitative example in Appendix G.3, showing that SIBPersona can facilitate error diag- nosis. 5.5 Results on CharacterEval To examine whether SIBPersona is applicable be- yond the social media setting, we evaluate our method on CharacterEval (Tu et al., 2024), a fic- tional character benchmark formulated as a single- turn response generation task. To minimize con- founding effects from a model’s prior knowledge of well-known fictional characters, we follow the anonymized evaluation setting proposed by Peng and Chen (2026). We construct dimensional cor- pora usinggpt-4o-miniand generate replies using OLMo-3-7B-Instruct. As summarized in Table 7, SIBPersona signif- icantly outperforms IMPersona on three out of four evaluation dimensions. Remarkably, SIBPer- sona also achieves performance competitive with CoSER-Llama-3.1-8B(Wang et al., 2025c), a MethodsCARACKCP IMPersona2.94 2.43 2.25 2.38 SIBPersona3.28 † 2.66 † 2.32 † 2.69 † Fine-tuned CoSER3.35 2.61 2.28 2.68 Table 7: Results on CharacterEval. CA: Conversational Ability, RA: Role-Playing Attractiveness, CK: Charac- ter Consistency–Knowledge, and CP: Character Con- sistency–Persona. Bold: the best results;†: significant improvement with p < 0.05. MethodsCARACKCP SIBPersona3.282.662.322.69 w/o T3.332.692.312.74 w/o I 3.282.662.312.68 w/o RP3.16 ‡ 2.612.30 2.62 ‡ w/o RP & T 3.05 ‡ 2.54 ‡ 2.28 2.57 ‡ Table 8: Component ablation study on CharacterEval. Bold: the best results;‡: significant degradation com- pared to SIBPersona (p < 0.05). model specifically fine-tuned on fictional charac- ters’ conversational data, demonstrating that an ICL-based method augmented with behavioral modeling can approach the performance of task- specific fine-tuned models. The ablation study in Table 8 further corrobo- rates the findings in §5.3. Removing theRPdimen- sion leads to significant degradation across multi- ple evaluation dimensions, while removing only the internal state yields a smaller performance drop. Removing theTdimension does not lead to a sig- nificant performance drop, suggesting that reaction process may play a more important role than stylis- tic information in fictional character benchmarks. These consistent patterns across two distinct role- playing settings confirm that SIBPersona is broadly applicable beyond the domain for which it was orig- inally designed. Additional results and analysis are provided in Appendix E.4. 6 Conclusion In this work, we study the problem of role-playing real individuals on social media and propose a behavior-aware framework to address two ma- jor challenges. In our framework, we introduce SIBPersona, which extends data-variant persona by modeling the reaction process. We construct a dataset featuring real social media influencers and formulate the task of generating replies. Our ex- perimental results across multiple models demon- strate that SIBPersona improves role-playing fi- delity, and human evaluation further verifies the in- terpretability of the retrieved SIB triplets. We also find that reaction process modeling is applicable to fictional character role-playing benchmarks. More- over, our reference-augmented evaluation protocol for role-playing real-individual shows acceptable alignment with human judgment. These findings suggest that behavior-aware persona modeling can improve role-playing performance on social media setting. In future work, this framework could be ex- tended to broader domains and more complex gen- eration tasks. One such direction is extending SIBPersona to multi-turn dialogue, which would require handling evolving reaction processes. This can be achieved by dynamically formulating re- trieval queries from the accumulated conversational history instead of relying solely on the latest com- ment, enabling the model to fetch situation-relevant SIB triplets that capture accumulated context. Limitations Although our study demonstrates effectiveness on both our newly constructed social media dataset and a public open-source dataset, and shows con- sistent improvements across both large API-based models and smaller open-source models, several limitations remain. First, we observe that our method struggles with two types of comment situations (Figure 13): real- time information queries and sarcastic comments. In the former case, the LLM lacks up-to-date in- formation, which leads to hallucinated replies. In the latter case, the LLM may misinterpret sarcastic remarks from followers, resulting in inappropriate behavioral strategies. Second, our current study focuses on single- turn reply generation and does not evaluate per- formance in more challenging multi-turn dialogue settings. Since conversational contexts may evolve across turns, future work could investigate how the proposed framework performs under dynamically changing interaction scenarios. Third, the dimensional documents in SIBPer- sona are currently provided to the LLM through in-context learning rather than model training. We have not explored whether integrating dimensional documents into model training could further im- prove performance. Future works could investigate training models with dimensional documents and examine whether such models can achieve stronger performance than API-based models. Ethical Considerations This work involves the use of social media data and the impersonation of real individuals. We take several measures to address ethical concerns related to data usage, annotation practices, and potential societal impacts. First, this study was conducted as an industry- academia collaboration between National Taiwan University and CMoney, the company that owns and operates the platform from which the data were collected. All data access and processing were con- ducted solely by CMoney-affiliated authors within a secure, company-managed environment in full compliance with the platform’s Terms of Service and Privacy Policy. We obtained explicit consent from all influencers whose personas were evalu- ated in our experiments to use their available data for research purposes. Second, regarding follower comments, platform members accept the Terms of Service and Privacy Policy upon registration, authorizing the platform to collect, process, and use member-provided data for online behavior re- search, statistics, and service improvements. To protect participant privacy, all follower comments in our dataset and evaluation materials are strictly anonymized, retaining no user identifiers or person- ally identifiable information. Furthermore, no raw influencer or follower content will be released as part of our research artifacts. Third, human anno- tators involved in the evaluation are employees of CMoney and are compensated through their regular salaries. Fourth, AI-based writing assistance tools were used to help polish parts of the manuscript. All AI-assisted text was carefully reviewed and proofread by the authors to ensure correctness and accuracy. Finally, we acknowledge that technologies ca- pable of impersonating real individuals could po- tentially be misused to generate misleading or de- ceptive content on social media. This work is an evaluative research study rather than a public de- ployment. Any future commercialization or de- ployment would be handled separately by CMoney and is outside the scope of this study; nonetheless, we recommend that any downstream deployment require: (1) personas to be created internally and only for explicitly authorized influencers, (2) hu- man review of generated content before publication, particularly for sensitive responses, and (3) mecha- nisms allowing influencers to deactivate their per- sonas or withdraw consent. Meanwhile, our study contributes to understanding the capabilities and limitations of such systems. We hope that these in- sights can help future research develop methods for detecting impersonation and mitigating the spread of misinformation in online communities. Acknowledgments The work was financially supported by a National Taiwan University Industry-Academia Cooperative Research Project, the National Science and Tech- nology Council (NSTC), and the Featured Area Research Center Program within the framework of the Higher Education Sprout Project by the Min- istry of Education (MOE), Taiwan, under Grants 112-2223-E-002-012-MY5, 115-2628-E-002-023- MY4, and 115L900901. We thank Li-Ching Chien, Pei-Yu Hou, Pei-Ru Huang, Si-Rui Huang, Meng-Jung Lin, Yung-Chen Liu, Xiao-An Wang, and Yi-Chi Yeh for serving as human annotators in this work. References Aurélien Bück-Kaeffer, Je Qin Chooi, Dan Zhao, Maxi- milian Puelma Touzel, Kellin Pelrine, Jean-François Godbout, Reihaneh Rabbany, and Zachary Yang. 2025. BluePrint: A social media user dataset for LLM persona evaluation and training. arXiv preprint arXiv:2510.02343. Yun-Shiuan Chuang, Ruixuan Tu, Chengtao Dai, You Li, Smit Vasani, Binwei Yao, Michael Henry Tessler, Sijia Yang, Dhavan Shah, Robert Hawkins, Junjie Hu, and Timothy T. Rogers. 2025. DEBATE: A large-scale benchmark for evaluating opinion dy- namics in role-playing LLM agents. arXiv preprint arXiv:2510.25110. Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences, 2nd edition. Lawrence Erlbaum Associates, Hillsdale, NJ. Zhenpeng Gao, Xiaofen Xing, and Xiangmin Xu. 2025. TailorRPA: A retrieval-based framework for elicit- ing personalized and coherent role-playing agents in general domain. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 5381–5412, Suzhou, China. Association for Compu- tational Linguistics. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Zhouchi Lin, Bowen Zhang, Lionel Ni, Wen Gao, Yuanzhuo Wang, and Jian Guo. 2026. A survey on LLM-as-a-judge. The Innovation, 7(6):101253. Kai He, Yucheng Huang, Wenqing Wang, Delong Ran, Dongming Sheng, Junxuan Huang, Qika Lin, Jiax- ing Xu, Wenqiang Liu, and Mengling Feng. 2025. Crab: A novel configurable role-playing LLM with assessing benchmark. In Proceedings of the 63rd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15030– 15052, Vienna, Austria. Association for Computa- tional Linguistics. Ankur Joshi, Saket Kale, Satish Chandel, and D. Kumar Pal. 2015. Likert scale: Explored and explained. British journal of applied science & technology, 7(4):396–403. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge- intensive NLP tasks. In Advances in Neural Infor- mation Processing Systems, volume 33, pages 9459– 9474. Li Li, Peilin Cai, Ryan A. Rossi, Franck Dernon- court, Branislav Kveton, Junda Wu, Tong Yu, Linxin Song, Tiankai Yang, Yuehan Qin, Nesreen K. Ahmed, Samyadeep Basu, Subhojyoti Mukherjee, Ruiyi Zhang, Zhengmian Hu, Bo Ni, Yuxiao Zhou, Zichao Wang, Yue Huang, and 5 others. 2025. A person- alized conversational benchmark: Towards simu- lating personalized conversations. arXiv preprint arXiv:2505.14106. Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. Jiaheng Liu, Zehao Ni, Haoran Que, Tao Sun, Zekun Wang, Jian Yang, Jiakai Wang, Hongcheng Guo, Zhongyuan Peng, Ge Zhang, Jiayi Tian, Xingyuan Bu, Ke Xu, Wenge Rong, Junran Peng, and Zhaoxi- ang Zhang. 2024. RoleAgent: Building, interacting, and benchmarking high-quality role-playing agents from scripts. In Advances in Neural Information Processing Systems, volume 37, pages 49403–49428. Andrey Lovakov and Elena R. Agadullina. 2021. Empir- ically derived guidelines for effect size interpretation in social psychology. European Journal of Social Psychology, 51(3):485–504. Mary L. McHugh. 2012. Interrater reliability: The kappa statistic. Biochemia Medica, 22(3):276–282. Walter Mischel and Yuichi Shoda. 1995. A cognitive- affective system theory of personality: reconceptual- izing situations, dispositions, dynamics, and invari- ance in personality structure. Psychological review, 102(2):246–268. Simon Münker, Nils Schwager, and Achim Rettinger. 2026. Don’t trust generative agents to mimic commu- nication on social networks unless you benchmarked their empirical realism. In Proceedings of the 19th Conference of the European Chapter of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 1141–1151, Rabat, Morocco. Associ- ation for Computational Linguistics. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. Karl Pearson. 1895. Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58:240–242. Ji-Lun Peng and Yun-Nung Chen. 2026. Rethinking role-playing evaluation: Anonymous benchmarking and a systematic study of personality effects. In Pro- ceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 205–218, Atlanta, Georgia, USA. Association for Computational Linguistics. Vahid Rahimzadeh, Erfan Moosavi Monazzah, Moham- mad Taher Pilehvar, and Yadollah Yaghoobzadeh. 2026. Synthia: Scalable grounded persona genera- tion from social media data. In Proceedings of the 64th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 45983–46006, San Diego, California, United States. Association for Computational Linguistics. Emily Royall and Thomas Wortmann. 2015. Finding the state space of urban regeneration: Modeling gen- trification as a probabilistic process using k-means clustering and markov models. In Proceedings of the 14th International Conference on Computers in Urban Planning and Urban Management (CUPUM 2015): Planning Support Systems and Smart Cities, Cambridge, MA, USA. Massachusetts Institute of Technology. Paper 275. Giovanni Santia and Jake Williams. 2018. BuzzFace: A news veracity dataset with Facebook user com- mentary and egos. In Proceedings of the Interna- tional AAAI Conference on Web and Social Media, volume 12, pages 531–540. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-LLM: A trainable agent for role- playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 13153–13187, Singapore. Association for Computational Linguistics. Quan Shi, Carlos E. Jimenez, Stephen Dong, Brian Seo, Caden Yao, Adam Kelch, and Karthik R. Narasimhan. 2025. IMPersona: Evaluating individual level LLM impersonation. In Second Conference on Language Modeling. C. Spearman. 1904. The proof and measurement of as- sociation between two things. The American Journal of Psychology, 15(1):72–101. Quan Tu, Chuanqi Chen, Jinpeng Li, Yanran Li, Shuo Shang, Dongyan Zhao, Ran Wang, and Rui Yan. 2023. CharacterChat: Learning towards conversa- tional AI with personalized social support. arXiv preprint arXiv:2308.10278. Quan Tu, Shilong Fan, Zihang Tian, Tianhao Shen, Shuo Shang, Xin Gao, and Rui Yan. 2024. Charac- terEval: A Chinese benchmark for role-playing con- versational agent evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 11836–11850, Bangkok, Thailand. Association for Computational Linguistics. Huandong Wang, Wenjie Fu, Yingzhou Tang, Zhi- long Chen, Yuxi Huang, Jinghua Piao, Chen Gao, Fengli Xu, Tao Jiang, and Yong Li. 2025a.A survey on responsible LLMs: Inherent risk, mali- cious use, and mitigation strategy. arXiv preprint arXiv:2501.09431. Lei Wang, Jianxun Lian, Yi Huang, Yanqi Dai, Haox- uan Li, Xu Chen, Xing Xie, and Ji-Rong Wen. 2025b. CharacterBox: Evaluating the role-playing capabil- ities of LLMs in text-based virtual worlds. In Pro- ceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies (Volume 1: Long Papers), pages 6372–6391, Albuquerque, New Mexico. Association for Compu- tational Linguistics. Noah Wang, Z.Y. Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang, Jie Fu, and Junran Peng. 2024. RoleLLM: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14743–14777, Bangkok, Thailand. As- sociation for Computational Linguistics. Xintao Wang, Heng Wang, Yifei Zhang, Xinfeng Yuan, Rui Xu, Jen-Tse Huang, Siyu Yuan, Haoran Guo, Jiangjie Chen, Shuchang Zhou, Wei Wang, and Yanghua Xiao. 2025c. CoSER: Coordinating LLM- based persona simulation of established roles. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 64822–64858. PMLR. Matthijs J Warrens. 2015. Five ways to look at Co- hen’s kappa. Journal of Psychology & Psychother- apy, 5(4):197. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Yuwei Wu, Xuezhe Ma, and Diyi Yang. 2021. Personal- ized response generation via generative split memory network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1956–1970, Online. Association for Computational Linguistics. Rui Xu, Xintao Wang, Jiangjie Chen, Siyu Yuan, Xin- feng Yuan, Jiaqing Liang, Zulong Chen, Xiaoqing- dong, and Yanghua Xiao. 2025. Character is des- tiny: Can persona-assigned language models make personal choices? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 15038–15059, Suzhou, China. Association for Com- putational Linguistics. Zhenhua Xu, Dongsheng Chen, Shuo Wang, Jian Li, Chengjie Wang, Meng Han, and Yabiao Wang. 2026. AdaMARP: An adaptive multi-agent interac- tion framework for general immersive role-playing. In Findings of the Association for Computational Lin- guistics: ACL 2026, pages 31230–31283, San Diego, California, United States. Association for Computa- tional Linguistics. Shihao Yang, Zhicong Lu, Yong Yang, Bo Lv, Yang Shen, and Nayu Liu. 2026.HyCoRA: Hyper- contrastive role-adaptive learning for role-playing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 34304–34312. Jing Ye, Rui Wang, Yuchuan Wu, Victor Ma, Feiteng Fang, Fei Huang, and Yongbin Li. 2025. CPO: Ad- dressing reward ambiguity in role-playing dialogue via comparative policy optimization. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 297–323, Suzhou, China. Asso- ciation for Computational Linguistics. Yeyong Yu, Runsheng Yu, Haojie Wei, Zhanqiu Zhang, and Quan Qian. 2025. Beyond dialogue: A profile- dialogue alignment framework towards general role- playing language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11992– 12022, Vienna, Austria. Association for Computa- tional Linguistics. Haonan Zhang, Run Luo, Xiong Liu, Yuchuan Wu, Ting- En Lin, Pengpeng Zeng, Qiang Qu, Feiteng Fang, Min Yang, Lianli Gao, Jingkuan Song, Fei Huang, and Yongbin Li. 2025. OmniCharacter: Towards im- mersive role-playing agents with seamless speech- language personality interaction. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26318–26331, Vienna, Austria. Association for Computational Linguistics. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2204–2213, Melbourne, Australia. Association for Computational Linguistics. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In Interna- tional Conference on Learning Representations. Yinhe Zheng, Guanyi Chen, Minlie Huang, Song Liu, and Xuan Zhu. 2019. Personalized dialogue generation with diversified traits. arXiv preprint arXiv:1901.09672. Jinfeng Zhou, Yongkang Huang, Bosi Wen, Guanqun Bi, Yuxuan Chen, Pei Ke, Zhuang Chen, Xiyao Xiao, Libiao Peng, Kuntian Tang, Rongsheng Zhang, Le Zhang, Tangjie Lv, Zhipeng Hu, Hongning Wang, and Minlie Huang. 2025. CharacterBench: Bench- marking character customization of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26101– 26110. A Detail of Constructing Dataset In this work, we construct the dataset from social media data related to the impersonated influencers, collected internally under the authorization of the platform operator. To ensure sufficient data cover- age, we select influencers who have more than500 comment–reply pairs in the demonstration source and more than120comment–reply pairs in the can- didate pool for testing. Table 22 reports the dataset statistics for the six influencers studied in our work. Furthermore, based on our preliminary analysis, comment–reply pairs exhibit highly distinct interac- tion types. Some pairs mainly involve factual infor- mation exchange, while others more clearly reveal the influencer’s speaking style or behavioral strate- gies. To ensure that the test samples comprehen- sively covers these diverse interactions, we prompt gpt-4o-minito assign each comment–reply pair a persona score ranging from 0 to 1, indicating the degree to which the reply exhibits the influ- encer’s personal characteristic. We then use this Influencer Persona ScoreABCDEF High50 50 50 35 509 Medium50 50 50 50 22 39 Low 50 50 50 50 12 50 Table 9: Number of data in the test samples. score as the basis for stratified sampling. Specif- ically, comment–reply pairs are categorized into three levels according to their persona scores: high [0.8, 1.0], medium[0.5, 0.8), and low[0, 0.5). Ex- amples of the three levels are shown in Figure 7. Low-level pairs mainly involve factual or experi- ential information exchange, medium-level pairs exhibit stylistic cues such as emoji usage, and high- level pairs clearly reflect the influencer’s personal characteristic. The prompt used to obtain persona scores is shown in Figure 14. The statistics of the test samples for each of the six influencers are sum- marized in Table 9. For Influencer D, all available data in the medium and high levels are included because each level contains fewer than50test sam- ples. The same logic applies to Influencer E and Influencer F. To further demonstrate the diversity of our dataset, Table 10 reports the text length distribu- tions and the proportions of the three persona-score levels across all six influencers. As shown in the table, textual lengths vary considerably across indi- viduals: average reply lengths range from8.46to 33.65characters, while average post lengths range from271.65to952.74characters. Moreover, the distribution of persona-score levels in the candidate pool reflects distinct individual interaction styles. For instance, Influencer E’s comment–reply pairs predominantly involve low persona-characteristic exchanges (87%), whereas Influencer C displays a much more balanced distribution across medium (36%) and low (53%) levels. These statistics con- firm that our dataset encompasses a wide spectrum of behavioral patterns. B Detail of SIBPersona B.1 Data-invariant Persona Construction For the data-invariant persona, we construct a short profile for each influencer using the influencer de- scription provided on the social media, rather than deriving it from historical data. Each profile con- tains approximately 50–100 words. InfluencerPost LengthComment Length Reply Length High (%) Med (%) Low (%) A467.92± 471.0917.58± 25.378.46± 9.6913%29%58% B367.39± 231.6230.70± 57.5510.01± 10.5412%35%53% C952.74± 740.9266.83± 105.5433.65± 28.1434%36%53% D355.21± 508.8914.20± 15.249.56± 10.1716%25%59% E271.65± 221.0027.54± 28.069.52± 10.215%8%87% F860.21± 574.4126.53± 29.4111.82± 10.398%33%60% Table 10: Dataset diversity statistics across the six influencers, including text lengths and the proportion of comment– reply pairs across high, medium, and low persona-score levels. B.2 Data-variant Persona Construction Table 11 reports the size of the dimensional cor- pora extracted for Influencer A in four LLMs, while Table 12 and Table 13 further present the corresponding statistics for the remain- ing five influencers usingDeepSeek-v3.2and gemini-3-flash-preview, respectively. All mod- els are accessed through cloud-based APIs: we ac- cessclaude-opus-4-6(pinned snapshot) through the Claude API,gemini-3-flash-preview(De- cember 2025 snapshot) through Google Cloud Platform (GCP), and bothgpt-5.2-2025-12-11 andDeepSeek-v3.2through Azure AI Foundry. For decoding parameters across tasks, we set (temperature, top-p)to(1.0, 1.0)for recognizing, (0.2, 0.8)for extraction, and(0.8, 1.0)for gener- ation. All other API parameters remain at their default values, with the reasoning effort set to the lowest level supported by each model. B.3 Distinctiveness of Persona Dimensions Our dimension-wise retrieval design is based on the hypothesis that the three dimensions of data- variant persona capture largely distinct aspects of an individual’s persona. While these dimensions may still exhibit certain correlations, we hypoth- esize that memory, speaking style, and reaction process encode sufficiently different types of in- formation to be treated separately during retrieval. Under this hypothesis, the follower’s comment is matched separately against each dimensional cor- pus, which prevents high-frequency patterns in one dimension from dominating the retrieval results of the others. To empirically examine this hypothesis, we ana- lyze the distinctiveness of the three dimensions in the embedding space. Specifically, we embedm p j k for memory,sp p j k for speaking style, and behav- iorb jk for the reaction process. We useb jk rather ModelsM T RP GPT-5.26,064 1,410 970 Gemini-3-flash 5,755 1,050 861 Deepseek-V3.25,143 1,022 749 Claude-Opus-4.62,307407703 Table 11: Number of dimensional documents of Influ- encer A. than the full SIB triplet because behavior, simi- lar to memory and speaking style, directly reflects the impersonated individual’s observable reply pat- terns, whereas situation and internal state mainly describe the intermediate reasoning process that leads to the behavior. The resulting vectors are then analyzed through qualitative visualization and quantitative clustering evaluation. The results pro- vide empirical evidence that the three dimensions are separable, supporting our decision to process them independently during retrieval. For the qualitative analysis, Figure 3 shows the t-SNE visualization of the embeddings ofmp jk , sp jk , andb jk extracted for Influencer A byClaude Opus 4.6 . As illustrated in the figure, the three di- mensions occupy different regions in the projected space, suggesting that they capture different aspects of the impersonated individual. For the quantita- tive analysis, we further calculate the Silhouette Score across six influencers, using documents ex- tracted byDeepSeek-v3.2. The resulting average Silhouette Score is0.37. As a random baseline, we permute the cluster label of all document em- beddings and repeat this process 1,000 times. The average Silhouette Score under this random per- mutation setting is−0.001. The substantial gap between the actual score and the random baseline provides evidence that the three dimensions form distinguishable clusters in the embedding space. InfluencerMT RP B12,358 8,162 6,701 C6,179 2,982 4,049 D8,914 4,935 3,811 E 10,087 2,393 2,378 F2,587446398 Table 12: Number of dimensional documents of Influ- encer B-F extracted from Deepseek V3.2. InfluencerM T RP B13360 8253 6911 C52963321 3643 D 10102 4839 4053 E10309 3536 3287 F2363515449 Table 13: Number of dimensional documents of Influ- encer B-F extracted from Gemini-3-flash. Figure 3: t-SNE projection of the embeddings ofmp jk , sp jk , and b jk generated by Claude-Opus-4.6. C Details of Reference-Augmented Evaluation We adopt the penalty-based evaluation protocol introduced by Wang et al. (2025c) to score RPA- generated replies. We usegpt-4.1-2025-04-14 as LLM evaluator and access it through the Azure AI Foundry. The temperature is set to0.2and top- pto0.8to ensure the stability of the evaluation results. Number of Representative Dimensional Docu- ments To avoid using the information from the same dimensional corpora for RPA generation and evaluation, we select the representative dimen- sional documents from the dimensional corpora constructed from the candidate pool for test sam- ples. Following the rule of thumb discussed by Royall and Wortmann (2015), we set the number of medoidskaccording tok = p M/2, where Mdenotes the number of dimensional documents in the corresponding corpus. We then roundkup to the nearest integer and use it as the number of representative dimensional documents. Table 23 re- ports, for each influencer and extracting model, the total number of extracted dimensional documents and the corresponding number of representative dimensional documents used in evaluation. Rubrics for LLM Evaluators Evaluations are conducted along three dimensions: memory, speak- ing style, and reaction process. Each dimension is independently evaluated in a separate run. For each evaluation run, the LLM evaluator is provided with the definition of the target dimension and its corre- sponding error types as dimension-specific rubrics. Using the data-invariant profile and representative dimensional documents as references, the LLM evaluator identifies flaws in the generated reply and assigns a severity score to each detected flaw. This design allows the LLM evaluator to focus on one dimension at a time and avoids penalizing errors from other dimensions. The complete evaluation prompt and evaluation rubrics are shown in Fig- ure 19 and Figure 20, respectively. Details of the Formula In the original formula- tion by Wang et al. (2025c), the evaluation score includes a length-based compensation term in ad- dition to the penalty term, since longer replies are more likely to receive penalties under error-based scoring schemes. However, in our setting, the gen- erated replies are typically short, with an average length of approximately10words. We therefore omit the length-based score adjustment in our eval- uation. Visualization for Representative Dimensional Documents We visualize the representative di- mensional documents extracted for Influencer A byDeepSeek-v3.2. As shown in Figure 4, Fig- ure 5, and Figure 6, the selected representative dimensional documents are spread across different regions of the projected space rather than being concentrated in a single local cluster. This quali- tative observation suggests that representative di- mensional documents capture diverse aspects of the impersonated influencer across the three dimen- Figure 4: t-SNE projection of representative memory documents generated by DeepSeek-v3.2. sions. Figure 5: t-SNE projection of representative speaking style documents generated by DeepSeek-v3.2. Figure 6: t-SNE projection of representative reaction process documents generated by DeepSeek-v3.2. D Details of Human Evaluation Human evaluations are conducted with two main objectives. The first objective is to assess the align- ment between reference-augmented evaluation pro- tocol and human judgment. For this purpose, we use the comparison evaluation to support annota- tor quality control and the performance evaluation to measure the correlation between human judg- ment and LLM evaluators. The second objective is to examine the validity of the reaction-process- based design of SIBPersona. Specifically, the in- terpretability evaluation assesses whether the re- trieved SIB triplets can serve as explanations for generated replies, and the construction faithfulness evaluation verifies whether the constructed SIB triplets are faithful to their corresponding evidence pairs. The following subsections first describe the settings and annotation protocols of the four human evaluations in Appendix D.1 and then present the corresponding results in Appendix D.2. D.1 Human Evaluation Setting This subsection describes the setting of our hu- man evaluation. In the following, we refer to each question in the human evaluation inventory as an evaluation item. D.1.1 Comparison Evaluation The comparison evaluation is designed to serve two complementary purposes. First, it provides a post hoc diagnostic for identifying potentially unreli- able annotators. Second, it offers a comparison- based assessment of the quality of RPA-generated replies. For each evaluation item, annotators are given an influencer’s postip, a follower’s comment fc, and three anonymized candidate replies. They are asked to rank the three replies according to how closely each reply resembles the impersonated influencer’s actual reply, with ties allowed when multiple replies are considered equally similar. Although all three options are presented as gen- erated replies, only two of them are produced by RPA: one by IMPersona and the other by SIBPer- sona. The remaining option is the ground-truth reply written by the influencer. Since the ground- truth reply is written by impersonated individual, a reliable annotator is expected not to consistently rank it as the least similar reply. We therefore use the hidden ground-truth reply as a quality-control signal. Specifically, for each annotator, we com- pute the proportion of evaluation items in which the ground-truth reply is ranked last. Annotators whose ground-truth-last rate exceeds a predefined threshold are excluded from subsequent analysis; in our work, the threshold is set to 30%. D.1.2 Performance Evaluation The performance evaluation is designed to assess the alignment between our proposed evaluation protocol and human judgment by measuring the correlation between their scores. In addition, it al- lows us to compare the quality of RPA-generated replies through statistical significance testing. For each evaluation item, annotators are given an influ- encer’s post, a follower’s comment, and an RPA- generated reply generated by IMPersona, Vanilla RAG or SIBPersona. They are asked to rate the reply along three dimensions: memory, speaking style, and reaction process, using a five-point Likert scale (Joshi et al., 2015). If an annotator considers a reply irrelevant to a given dimension, they may select NA. This option is included to avoid forcing annotators to assign scores to dimensions that are not applicable to a particular reply. To address NA annotations during analysis, we process human evaluation scores as follows. For each evaluation item and each dimension, we ag- gregate scores across annotators by averaging all non-NA ratings. This aggregation yields one human evaluation score for each dimension of each sample and helps reduce the influence of annotator-specific variation, allowing the subsequent correlation anal- ysis to focus on aggregate human evaluation trends. If all annotators select NA for an evaluation item on a given dimension, the aggregated score for that di- mension is treated as NA. Such cases are excluded from analysis involving the corresponding dimen- sion because no valid human evaluation score is available. After aggregating the human evaluation scores, we perform two complementary analysis. The first analysis examines the alignment between LLM evaluator and human judgment using correlation measures. The second analysis compares the qual- ity of RPA-generated replies across IMPersona, Vanilla RAG and SIBPersona based on human eval- uation scores. For the correlation analysis, we compute both dimension-specific and overall correlations be- tween LLM evaluator and human judgment scores. Dimension-specific correlation is calculated sep- arately for memory, speaking style, and reaction process. For each dimension, evaluation items with NA in the corresponding aggregated human judg- ment score are excluded since no valid human score is available for that dimension. This analysis allows us to examine whether the LLM evaluator aligns with human judgment on each specific dimension. In addition, we compute an overall correlation by first averaging the scores across the three dimen- sions for each evaluation item and then calculat- ing the correlation based on these overall scores. Compared with the dimension-specific correlation, this analysis provides an overall view of the align- ment between LLM evaluator and human judgment. For this analysis, evaluation items are excluded if any of the three dimension scores is NA, because the overall score cannot be consistently computed across all dimensions. For the method-level quality comparison, we fo- cus on the three RPA methods: IMPersona, Vanilla RAG and SIBPersona. For each evaluation dimen- sion, we compute the average human evaluation score of each method and conductt-tests with Holm correction to examine whether the score dif- ferences are statistically significant. This analysis complements the correlation analysis by directly comparing the human-evaluated quality of the three methods. D.1.3 Interpretability Evaluation After evaluating the quality of the RPA-generated replies, we further conduct evaluations to exam- ine whether the constructed reaction process cor- pus provides meaningful explanatory evidence for SIBPersona. We first perform an interpretabil- ity evaluation to assess whether the retrieved SIB triplets can serve as plausible explanations for the generated replies. For each evaluation item, anno- tators are given an influencer’s post, a follower’s comment, and a reply generated by SIBPersona, together with the top-three SIB triplets retrieved and provided to the RPA. Annotators are asked to evaluate each SIB triplet from two aspects. First, they assess whether the situation is relevant to the influencer’s post and the follower’s comment. Second, they assess whether the internal state and behavior can reasonably ex- plain the generated reply. This design allows us to examine both the contextual relevance of the re- trieved situations and the explanatory consistency between the retrieved SIB triplets and the final gen- erated reply. Ratings are given using a five-point Likert scale. D.1.4 Construction Faithfulness Evaluation Finally, we conduct a construction faithfulness eval- uation to assess whether the generated SIB triplets are reasonably extracted from the original data sam- ple. This evaluation examines whether each ex- tracted SIB triplet can be reasonably supported by the comment–reply pair assigned as its evi- dence. For each evaluation item, annotators are given a comment–reply pair and its correspond- ing extracted SIB triplet. They are then asked to the annotator whether it is reasonable to construct the given SIB triplet from the provided comment– reply pair. This setup allows us to directly evaluate the faithfulness of the corpus construction process, rather than only assessing their groundedness in downstream reply generation. The evaluation is collected as a binary pass/fail label, where pass indicates that the triplet is reasonably supported by the source comment–reply pair, and fail indicates that the triplet is unsupported, inconsistent with the comment–reply pair, or overly inferred from the source data. D.1.5 Implementation Details All human evaluations are conducted using test samples generated byDeepSeek-V3.2for Influ- encer F. For the performance evaluation, we use30 test samples and evaluate replies generated by three methods, Vanilla RAG, IMPersona and SIBPer- sona, resulting in90evaluation items in total. For the interpretability evaluation, we use the same 30test samples. Since each test sample contains top-three retrieved SIB triplets, annotators evaluate each triplet separately. The30test samples are manually selected from the three levels introduced in §5.1 to ensure diver- sity in comment situations. For the comparison evaluation, we randomly select10samples from these30test samples. For the construction faith- fulness evaluation, we use50representative dimen- sional documents from reaction process corpus to assess whether the extracted SIB triplets are faith- ful to their corresponding evidence pairs. We recruit nine annotators for the human evalua- tion. All annotators have worked with Influencer F for several years, ensuring that they have sufficient familiarity with the impersonated influencer. To control annotation quality, we further filter annota- tors based on the ground-truth-last rate calculated from the comparison evaluation. Since only three annotators pass the filtering threshold, we report the results of these three annotators in the following MethodsReciprocal Ranking score IMPersona0.42 ‡ SIBPersona 0.71 Ground Truth0.71 Table 14: Human comparison evaluation reciprocal scores. Bold indicates the best score.‡: significant degradation compared to SIBPersona (p < 0.05) subsection. D.2 Results of Human Evaluation Results of Comparison EvaluationFor the com- parison evaluation, we convert the rankings of can- didate replies into reciprocal ranking scores, where the score of each reply is computed as the recipro- cal of its assigned rank. We then report the aver- age reciprocal ranking score for each method, with higher scores indicating better rankings. As shown in Table 14, SIBPersona achieves a significantly higher score than IMPersona and obtains an aver- age score comparable to that of the ground-truth replies. This result suggests that annotators judged the replies generated by SIBPersona to be substan- tially closer to the replies of real influencer’s than those generated by IMPersona. Results of Performance EvaluationTo examine the alignment between human judgment and LLM evaluators, we compare the aggregated human judg- ment scores with the LLM evaluator scores and report their correlations. We report both Pearson’s r(Pearson, 1895) and Spearman’sρ(Spearman, 1904) to measure linear and rank-based associa- tions, respectively. As shown in Figure 8, the over- all human judgment score shows a moderate corre- lation with the overall LLM evaluator score, yield- ing Pearson’srof0.561and Spearman’sρof0.530. This suggests that the LLM evaluator captures eval- uation trends that are broadly aligned with human judgment when the three dimensions are consid- ered jointly. Among the dimension-specific correla- tions, speaking style shows the strongest alignment between human judgment and LLM evaluator, with Pearson’srof0.576and Spearman’sρof0.561 indicating a moderate correlation. Memory and reaction process also show significant but weaker correlations, with Pearson’srof0.354and0.289, and Spearman’sρof0.365and0.287, respectively. These results are consistent with the previous find- ings in role-playing evaluation (Zhou et al., 2025; M T RP Rater 1 & 20.3640.4290.406 Rater 1 & 30.4170.5060.508 Rater 2 & 30.4260.4950.487 Table 15: Inter-rater agreement among the three annota- tors. He et al., 2025) and demonstrate that our reference- augmented protocol serves as a reasonable proxy for human judgment. Moreover, we further assess inter-rater agree- ment among human annotators, computed by the weighted Cohen’sκ(Warrens, 2015). The agree- ment scores across the three dimensions are pre- sented in Table 15. The pairwise agreement scores range from0.364to0.508. Following the inter- pretation guidelines of McHugh (2012), these re- sults indicate a fair to moderate level of agreement across annotators. The agreement is highest be- tween Rater 1 and Rater 3, while Rater 1 and Rater 2 show relatively lower agreement. This pattern suggests that the annotation task involves subjec- tive judgment, but the annotators still exhibit mean- ingful consistency. For the method-level quality comparison, as shown in Table 5, SIBPersona significantly out- performs IMPersona across all three dimensions. Results of Interpretability Evaluation For the interpretability evaluation, each test sample is asso- ciated with three retrieved SIB triplets. Since the RPA may rely more heavily on one of the retrieved triplets during generation, we take the highest score among the three triplets as the score for that test sample. Specifically, letscore S i,j andscore IB i,j de- note the average annotator scores for thej-th SIB triplet of thei-th test sample, wherej ∈1, 2, 3. The final scores, denoted asScore S andScore IB , are computed by averaging the highest triplet-level score across all N test samples: Score S = 1 N N X i=1 max j∈1,2,3 score S i,j , Score IB = 1 N N X i=1 max j∈1,2,3 score IB i,j . Our RPA replies achieve aScore S of3.63and a Score IB of3.57. The highScore S indicates that Pass Rate95% CI Rater 10.80[0.67, 0.89] Rater 20.84[0.72, 0.92] Rater 30.92[0.81, 0.97] Total0.94[0.84, 0.98] Table 16: Extraction faithfulness evaluated by the three annotators. The Total column reports the pass rate of by applying majority-vote across all annotators. the situation described in the retrieved SIB triplet generally align with the corresponding follower’s comment. The highScore IB further suggests that the internal states and behaviors described in the triplets can reasonably explain the final replies gen- erated by the RPA. In addition, we useScore IB as the criterion for categorizing test samples into interpretable and un- interpretable, as shown in Table 6. For each test sample, if the averageScore IB assigned by the three annotators exceeds3, the test sample is con- sidered interpretable; otherwise, it is considered uninterpretable. These results suggest that our re- trieval method can identify situations similar to an unseen follower’s comment, and the retrieved inter- nal states and behaviors serve as a valid supporting context in the RPA-generated replies. Results of Extraction Faithfulness Evaluation To evaluate the faithfulness of the reaction process corpus, we report the pass rates assigned by the three annotators in Table 16. The individual anno- tator pass rates are consistently high, with all raters assigning pass rates above0.80. When applying majority voting across annotators, the overall pass rate reaches0.94with a 95% confidence interval of [0.84, 0.98]. These results suggest that most constructed SIB triplets are reasonably supported by their corre- sponding comment–reply pair. In other words, our process of constructing dimensional corpora can generally preserve information from the original comment–reply pairs while converting them into structured SIB triplets. The higher majority-vote pass rate further indicates that although the anno- tators may differ in their strictness, there is strong aggregate support for the faithfulness of the con- structed reaction process corpus. E Additional Results E.1 Per-Influencer Results Table 24 presents the detailed results for Influencer B through Influencer F, which correspond to the ag- gregated results reported in Table 3. The table pro- vides a more fine-grained view of our experimental results in §5.3. Overall, our method consistently outperforms the baseline in both the speaking style dimension and reaction process dimension. E.2 Comparison with Traditional NLP Metrics To further justify the use of our reference- augmented evaluation protocol, we compare its alignment with human judgment against several traditional automatic evaluation metrics, includ- ing BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004), BERTScore (Zhang et al., 2020), and BLEURT (Sellam et al., 2020). Specifically, we compute the Spearman correlation between each metric and human judgment. The results show that traditional metrics exhibit substantially weaker correlations with human judgment: BLEU achieves a Spearman’sρof−0.01, ROUGE-L achieves−0.04, BERTScore achieves0.28, and BLEURT achieves0.02. In contrast, our reference- augmented evaluation protocol achieves a Spear- man’sρof0.530, showing a much stronger align- ment with human judgment. This gap suggests that traditional automatic met- rics are not sufficient for evaluating role-playing real-individual on social media. One possible rea- son is that a generated reply may differ substan- tially from the ground-truth reply at the lexical or surface-semantic level, while still adopting a behavioral strategy that is consistent with the im- personated individual. For example, as shown in Figure 9, the RPA-generated reply follows the in- fluencer’s behavioral strategy and is therefore rated highly by both the LLM evaluator and human eval- uators. However, because this reply has low lexical and surface-semantic similarity to the ground-truth reply, it receives low scores from traditional auto- matic metrics. These findings support our decision to report reference-augmented evaluation as the main auto- matic evaluation protocol in this work, as it pro- vides a more fine-grained assessment of whether the generated replies align with the impersonated individuals. Method# Tokens Vanilla RAG593.51± 456.37 IMPersona1972.64± 816.14 SIBPersona1630.83± 812.02 Table 17: Average number of tokens inserted into the prompt per test query across six influencers using DeepSeek-V3.2. E.3 Token Budgets Table 17 reports the average number of addi- tional context tokens inserted into the prompt per query, evaluated across the six influencers usingDeepSeek-V3.2.As shown in the re- sults,SIBPersonaconsumes fewer tokens than IMPersonawhile achieving superior role-playing performance. This indicates that performance gains are not simply driven by increasing context length, but rather rely on supplying structured, behavior- aware, and effective persona information. E.4 Result of RoleAgentBench We further evaluate our method on an additional open-source benchmark, RoleAgentBench (Liu et al., 2024), a bilingual (Chinese–English) role- playing of fictional character benchmark. To ex- amine whether SIBPersona is applicable beyond the social media setting, we select two response generation tasks from the benchmark: General Response and Summary. In the experiments, gpt-4o-miniis used to construct the dimensional corpus,OLMo-3-7B-Instructis used as the gener- ation model, andgemini-2.0-flashserves as the LLM evaluator. During experimentation, we follow Peng and Chen (2026) and anonymize the character names in the dataset to mitigate the influence of the model’s prior knowledge during pre-training. For evaluation, we use two complementary pro- tocols: our proposed reference-augmented evalua- tion protocol and the original evaluation protocol provided by the benchmark. Under the benchmark protocol, responses are evaluated in a pairwise comparison setting. Specifically, its LLM eval- uator is presented with two responses generated by SIBPersona as well as IMPersona and asked to select the one that better resembles the target char- acter. To mitigate potential positional bias caused by response ordering in the prompt, we conduct two evaluation rounds with the order of the two re- sponses swapped. A method is counted as the win- ner only if its response is selected in both rounds, while it is counted as losing only if the compet- ing method is selected in both rounds. Cases with inconsistent outcomes across the two rounds are treated as ties. The experimental results (Table 25) show that our method significantly outperforms the baseline under both evaluation protocols, consistent with the findings of our main experiment. This suggests that SIBPersona is also applicable to fictional character benchmarks. For completeness, we also provide the prompts used in the benchmark setting. Fig- ure 15 shows the prompt for dimensional relevance filtering, while Figure 16, Figure 17, and Figure 18 present the prompts used for dimensional document extraction. The prompt used for the penalty-based LLM evaluation is shown in Figure 19, with the corresponding evaluation rubrics provided in Fig- ure 20, and Figure 21 presents the prompt used for response generation. F Validation of Reference-Augmented Evaluation Automated evaluation for role-playing real indi- viduals remains a challenging and underexplored problem due to the lack of static gold-standard per- sonas. To validate that our reference-augmented evaluation protocol provides a reasonable assess- ment of role-playing fidelity, we conduct empirical investigations addressing potential issues. F.1 Alignment with Human Judgment A central objective of an automated evaluation protocol is to accurately reflect human perception. To justify our design choice of using SIBPersona- structured documents from the test candidate pool as references, we compare the correlation between LLM evaluator scores and human judgement un- der three different reference configurations. As reported in Table 20, our proposed configuration achieves the highest alignment with human judg- ment, outperforming raw comment–reply pairs from the same pool as well as structured docu- ments extracted from the demonstration source. These results confirm that structured SIBPersona representations best approximate human evalua- tive criteria. Furthermore, extracting references from the demonstration source—the exact corpus observed during generation—tends to reward sur- face context-matching rather than true behavioral fidelity, which explains its lower correlation with human judgment. MethodM T RP Naive95.89 46.43 ‡ 70.85 ‡ CoT 96.58 45.73 ‡ 72.33 ‡ IMPersona 95.94 48.10 ‡ 73.20 ‡ Vanilla RAG 95.32 54.12 ‡ 73.68 ‡ SIBPersona 95.4659.2277.81 Table 18: Evaluation results on Influencer A averaged over four models after excluding overlapping test in- stances. ‡: significant degradation compared to SIBPer- sona (p < 0.05). F.2 Robustness Against Test Sample Overlap We further examine whether gold reply leakage oc- curs when representative reference documents are sampled from the test pool. BecauseK-medoids selects only⌈ p N/2⌉cluster medoids fromNcan- didate instances to ensure semantic diversity rather than exhaustive coverage, the empirical overlap be- tween representative documents and test samples is exceptionally low:0.35%for Memory,4.01%for Speaking Style, and4.68%for Reaction Process on Influencer A. To strictly rule out any leakage bias, we re-evaluate all baseline models after ex- cluding all overlapping instances. As shown in Table 18, the performance rankings and relative improvements remain identical to our primary find- ings, confirming that the minimal overlap does not affect evaluation outcomes. F.3 Evaluation Under Strictly Independent Demonstration Sources To completely eliminate the risk of test pool con- tamination and assess the generalizability of our protocol, we conduct a conservative check where references are constructed entirely from the inde- pendent demonstration source. Table 19 and Ta- ble 21 report the results. Across all tested models and influencers, SIBPersona consistently outper- forms all baselines on bothTandRPdimensions. The persistent advantages across diverse reference configurations—test candidate pool and demonstra- tion source—demonstrate that our evaluation pro- tocol and empirical conclusions are robust and not artifacts of specific reference construction strate- gies. G Examples and Case Study This section presents qualitative analysis of the re- trieved dimensional documents in Appendix G.1, MethodsMTRP Naive97.28 46.48 ‡ 77.33 ‡ CoT 97.61 47.93 ‡ 78.08 ‡ IMPersona 97.14 48.63 ‡ 78.80 ‡ Vanilla RAG 97.02 58.93 ‡ 79.18 ‡ SIBPersona 96.6962.3781.52 Table 19: Results on Influencer A averaged over four models using evaluator references extracted from the demonstration source. ‡ : significant degradation com- pared to SIBPersona (p < 0.05). representative success and failure reply cases in Ap- pendix G.2, and an example showing how SIBPer- sona supports error tracing and correction in Ap- pendix G.3. G.1 Examples of Retrieved Dimensional Document Figure 10 and Figure 11 present examples of the top-1 retrieved dimensional documents. Given a follower’s comment that praises the influencer’s appearance, the retrieved situation closely matches the comment context. The generated reply also reflects the retrieved speaking style and behavioral strategy, avoiding a direct acceptance of the com- pliment. In contrast, the memory dimension re- trieves less relevant information, which is expected because factual knowledge is less central to this type of comment. Annotators familiar with Influ- encer A further confirm that the influencer typically does not directly accept compliments about their appearance. This case suggests that our method can retrieve information of situation-relevant reac- tion process and produce a reply consistent with the influencer’s characteristics. G.2 Examples of Successes and Failure Cases In addition, Figure 12 illustrates that our method achieves better role-playing fidelity than the base- line methods. Across the two examples, the ground- truth replies are relatively concise and reflect the influencer’s characteristic way of reacting to fol- lowers: giving a brief acknowledgment in reply to praise and using a sharp, indirect reply when chal- lenged. SIBPersona better preserves these prag- matic reply patterns. For example, when the fol- lower praises the influencer, our method generates a short positive reaction that is close to the ground- truth reply. When the follower comments sarcas- tically with hindsight, our method returns with a sarcastic counter-reaction, which better matches the interactional intent of the ground truth. In contrast, the baseline methods often deviate from the impersonated individual’s reply pattern in different ways. Naive and CoT tend to introduce ex- cessive explanatory content, such as trading strate- gies, app usage details, or additional contextual as- sumptions that are not present in the ground-truth replies. Similarly, IMPersona frequently gener- ates overly detailed and elaborated replies, suggest- ing that hierarchical memory alone is not enough for modeling behavioral strategies. Vanilla RAG produces shorter replies, but its replies are often generic and fail to capture the specific behavioral strategy required by the comment context. In contrast, Figure 13 highlights the limitations of our approach. When the comment involves real- time information, our method may misinterpret the situation due to insufficient knowledge of the cur- rent context. In addition, for sarcastic comments, the embedding-based retrieval process may fail to capture the follower’s ironic intent. This can lead to the retrieval of SIB triplets that are mismatched with the actual situation, causing the generated re- ply to follow an inappropriate behavioral strategy. These cases suggest that our method remains sen- sitive to the quality of situation understanding and retrieval. This limitation is especially pronounced when the input requires real-time knowledge or reasoning beyond surface-level semantic similarity. G.3 Tracing and Correcting Behavioral Errors Even when the system makes mistakes, SIBPer- sona allows us to effectively trace the source of the error and adjust RPA properly. For instance, we ob- served a case where Influencer A replied to a fan’s generic "Thank you" with a highly user-specific follow-up question: "Has that matter been taken care of?". After investigation, we found that the LLM initially yields a reaction process document with this outlier behavior, even though such a re- ply would be inappropriate for other users. A key advantage of SIBPersona is adjustable: because the generated replies can be traced back to spe- cific dimensional documents. By simply deleting the specific outlier document from the dimensional corpus, we can immediately and precisely adjust the model’s behavior. Reference FormatSource of ReferencePearsonSpearman Raw comment–reply pairsTest candidate pool0.3550.409 SIBPersona documentDemonstration source0.4480.462 SIBPersona documentTest candidate pool (Ours)0.5610.530 Table 20: Correlation between different evaluator configurations and human judgments. MethodsDeepseek-V3.2Gemini-3-flash MTRPMTRP Naive91.34± 3.24 42.63 ‡ ± 9.19 70.83 ‡ ± 5.4992.18± 2.83 44.21 ‡ ± 10.84 68.91 ‡ ± 7.71 CoT91.06± 3.35 46.65 ‡ ± 7.36 71.56 ‡ ± 4.7992.14± 1.71 45.57 ‡ ± 8.63 68.53 ‡ ± 7.39 IMPersona91.43± 2.61 48.44 ‡ ± 5.61 73.04 ‡ ± 5.2091.34± 2.36 46.32 ‡ ± 8.20 69.88 ‡ ± 7.94 Vanilla RAG91.30± 3.24 59.18 ‡ ± 4.58 74.30 ‡ ± 4.2692.37± 2.60 56.47 ‡ ± 3.07 72.49 ‡ ± 3.97 SIBPersona 92.01± 3.53 62.96± 4.27 77.18± 4.4492.29± 2.48 59.48± 3.0075.60± 5.10 Table 21: Evaluation results across the other five influencers using references extracted from the demonstration source. ‡: significant degradation compared to SIBPersona (p < 0.05). Influencer SplitData typeABCDEF DemonstrationPost195527167132903 2130 453 DemonstrationComment-Reply pair1361 15240 5553 6087 4383 705 Test Candidates Post6522855419228784 Test Candidates Comment-Reply pair5471719361227261120 Table 22: Number of data in the training set and the test candidate pool. Influencers ModelsMTRP AllRep.All Rep.All Rep. A DeepSeek-V3.22424354461527412 Claude-Opus-4.61156251891024412 Gemini-3-flash2477364191531813 GPT-5.2 2926395881837514 B DeepSeek-V3.21786308512175820 Gemini-3-flash 1628298632172219 C DeepSeek-V3.2491162001027012 Gemini-3-flash420152621229713 D DeepSeek-V3.276520184101359 Gemini-3-flash75520200101589 E DeepSeek-V3.21314261811017910 Gemini-3-flash1230252671221911 F DeepSeek-V3.2521171118947 Gemini-3-flash476161178937 Table 23: Statistics of extracted documents and representative documents across influencers and models. All denotes the total number of dimensional documents extracted from the candidates for testing, and Rep. denotes the number of representative dimensional documents selected from the corresponding dimensional corpus. Persona score levelFollower’s commentInfluencer’s reply low意即，該換電腦了. AI筆電是不錯的選擇 (In other words, it’s time for a new computer. An AI laptop would be a good choice ) 我有筆電了，習慣了桌上電腦 (I already have a laptop. I’m used to using a desktop computer.) medium姊我愛你❤ (Influencer, I love you ❤) high馬後炮 (Hindsight.) 那就不用看馬後炮了 (Then you don't need to read my post.) Figure 7: Examples of three level of data for Influencer A. Figure 8: Correlation matrix between LLM and human evaluations. *** p < 0.001, ** p < 0.01, * p < 0.05. Methods BCDEF MTRPMTRPMTRPMTRPMTRP Deepseek v3.2 Naive 97.67 45.60 ‡ 78.07 ‡ 88.97 52.00 ‡ 69.57 ‡ 91.59 37.22 ‡ 60.93 ‡ 90.24 26.85 ‡ 64.17 ‡ 90.66 40.97 ‡ 66.22 ‡ CoT 97.87 45.63 ‡ 79.80 ‡ 89.57 53.00 ‡ 71.4390.89 43.96 ‡ 62.22 ‡ 90.30 28.15 ‡ 65.30 ‡ 90.82 40.87 ‡ 66.48 ‡ IMPersona 95.77 46.47 ‡ 81.43 ‡ 89.70 54.43 ‡ 71.6790.70 42.44 ‡ 64.63 ‡ 90.65 36.73 ‡ 66.19 ‡ 90.20 42.35 ‡ 67.81 Vanilla RAG97.33 60.50 81.33 ‡ 88.20 55.43 ‡ 70.8390.81 58.41 ‡ 70.44 ‡ 89.82 53.8769.7089.34 55.10 66.33 ‡ SIBPersona97.77 61.6384.7388.40 59.3772.9392.56 63.9374.4489.70 59.8873.3990.77 60.0570.71 Gemini-3-flash Naive96.80 47.77 ‡ 79.60 ‡ 90.97 49.60 ‡ 66.80 ‡ 91.78 44.11 ‡ 73.52 ‡ 91.07 29.46 ‡ 57.44 ‡ 90.51 37.76 ‡ 70.15 ‡ CoT96.70 47.33 ‡ 78.97 ‡ 89.37 48.73 ‡ 67.37 ‡ 92.11 45.96 ‡ 71.22 ‡ 90.60 31.85 ‡ 59.23 ‡ 90.82 36.22 ‡ 70.87 ‡ IMPersona 94.80 48.33 ‡ 80.53 ‡ 90.20 51.30 ‡ 67.90 ‡ 91.37 46.30 ‡ 73.37 ‡ 89.52 33.63 ‡ 58.63 ‡ 89.80 39.54 ‡ 71.99 ‡ Vanilla RAG 96.67 57.27 80.93 ‡ 90.83 49.70 ‡ 67.50 ‡ 91.30 55.85 70.93 ‡ 89.70 58.4568.5190.77 45.61 ‡ 73.37 SIBPersona 96.17 60.9384.9390.10 55.3071.4392.19 57.8576.5290.77 59.8870.7190.66 52.0976.17 Table 24: Results of Influencer B–F. Bold: best results; ‡ : significant degradation compared to SIBPersonap < 0.05. LLM penalty-based evaluation Pair-wise Evaluation TasksMethods MTRPWinTieLose GeneralIMPersona75.3265.1071.15--- GeneralSIBPersona 76.6466.75 † 73.97 † 0.43 0.320.25 Summary IMPersona75.2862.3770.44--- Summary SIBPersona78.94 † 63.8373.12 † 0.34 0.390.27 Table 25: Experimental results on RoleAgentBench, comparing our method with the baseline. Bold indicates the better score between the two methods. † denotes a result that is significantly better than its counterpart (p < 0.05). Follower’s Comment Ground TruthRPA’s reply Human evaluation LLM evaluation BLEUROUGE_LBERTScoreBLEURT 謝老大 (Thanks, boss.) 因為我們是朋 友啊！ (Because we’re friends!) 客氣 (No problem.) 3.83 81.67000.46-8.15 Figure 9: An example where both human and LLM evaluators assign high scores, while traditional metrics assign low scores. "persona": "Memory": ["point": "Influencer認為交易決定會受到情緒影響，並強調心態是交易的瓶頸", "Speaking_Style": ["point": "Influencer經常使用和❤這兩個emoji來回覆粉絲的留言", 留言：人美，心也美。 "evidence_pair": [ "粉絲": "謝美麗的Influencer♥", "Influencer": ""], ], "Reaction_Process": [ "situation": "粉絲讚美Influencer的認真與負責，並提到打破智慧與美貌的魔咒", "internal_state": "Influencer感受到粉絲的支持，並以幽默的方式回應自己的不足", "behavior": "以輕鬆幽默的語氣回覆", "evidence_pair": [ "粉絲": "認真 負責 積極 也打破了智慧與美貌不可兼得之魔咒 值得推", "Influencer": "請參考前文左側交易右側交易"]], RPA 回覆：但還是有社恐，哈，謝大哥❤ Figure 10: Example of top-1 dimensional document retrieval. The influencer names in the text are anonymized and replaced with "Influencer". "persona": "Memory": ["point": "Influencer believes that trading decisions are influenced by emotions and emphasizes that mindset is the bottleneck in trading.", "Speaking_Style": ["point": "Influencer frequently uses the and ❤ emojis when replying to fans' comments.", Comment：Beautiful on the outside, and beautiful on the inside. "evidence_pair": [ "Fan": "Thank you, beautiful Influencer ❤", "Influencer": ""], ], "Reaction_Process": [ "situation": "A fan praises Influencer's diligence and responsibility, mentioning that she breaks the curse of 'wisdom and beauty being mutually exclusive", "internal_state": "Influencer feels the support from the fan and responds to her own shortcomings in a humorous way.", "behavior": "Replying with a relaxed and humorous tone.", "evidence_pair": [ "Fan": "Diligent, responsible, proactive, and breaks the curse that one cannot have both wisdom and beauty. Highly recommended.", "Influencer": "Please refer to the previous post regarding left-side trading and right-side trading."]], RPA Reply: But I still have social anxiety, haha. Thank you, Bro ❤ Figure 11: English translation of the example for top-1 dimensional document retrieval. The influencer names in the text are anonymized and replaced with "Influencer". Follower’s Comment Ground Truth Our RPA’s response Naive RPA’s replyCoT RPA’s reply Vanilla RAG RPA's reply IMPersona RPA’s reply 有跟到。 Influencer太神 了。 (I followed along. Influencer is a god.) 恭喜賺錢，跟到 啥？哈 (Congratulations on making money, what did you follow? Haha.) 謝支持！這幾檔都是照 著底部布局的節奏走，有 跟到就好抱緊，該獲利 了結時我會提醒～記得常 看APP更新跟直播，一起 跟緊主力動向！(Thanks for your support! These stocks are all following the bottoming pattern. If you've caught them, hold on tight. I'l remind you when it's time to take profits. Remember to check the app for updates and live streams regularly, and keep up with the big players' moves! ) 有跟到就好啦～我哪有 神，是你有紀律。記得別 追高，位置跟籌碼對了 再做，後面我直播再把 族群跟起漲點講清楚。 (Glad you caught the move! I'm no god, it's your discipline that paid off. Remember not to chase highs—only make your move when the setup and fund flows are right. I'l break down the sectors and breakout points clearly in my next livestream. ) 你眼光也很好 (You've got a great eye too! ) 恭喜你有跟上！其實不是我神，是我們堅持「絕 不追高」、守在底部看主力籌碼的策略奏效了。看 到大家都有賺到錢，我這幾天假也算放得值了。 記得持股要拿穩，盤得越久爆發力通常越驚人， 我們繼續看下去！❤ (Congratulations on following along! Actually, it’s not that I’m a "god," it’s that our strategy of "never chasing highs" and staying at the bottom to watch the main funds' (whale) chips worked out. Seeing everyone make money makes my few days of vacation worth it. Remember to hold your positions steady; usually, the longer it consolidates, the more explosive the breakout. Let’s keep watching! ❤) 馬後炮 (Hindsight.) 那就不用看馬後 炮了 (Then you don't need to read my post.) 但你股票輸錢的 時候想我就欣 慰多了 (I’l feel much better once you start losing money in the stock market.) 馬後炮？同學你自己去翻 APP選股紀錄，上面日期 都清楚～要不要賺錢 看你自己，不勉強的啦！ (Hindsight? My friend, go check your stock selection history on the app; the dates are all clearly displayed. Whether you make money or not is up to you; there's no pressure!) 馬後炮？我每檔都寫得 很清楚是哪一天APP出 現的（7/17、7/21、8/1、 8/14），你去對照貼文跟 走勢就知道是不是事後 補講。看不順眼可以略 過，我忙著陪同學抓底 部，不追高。 (Hindsight? I wrote down exactly which day every single stock showed up on the app (7/17, 7/21, 8/1, 8/14). Go compare the posts with the chart patterns, and you'l see if I'm just talking after the fact. If you don't like it, just scroll past. I'm busy helping my students catch the bottom, we don't chase highs. ) 歡迎來看馬後炮，我每天 盤中都有發文，歡迎追蹤 驗證 (Feel free to check whether it’s hindsight bias. I post during market hours every day, so you’re welcome to follow and verify ) 來喔，這裡不是事後馬後炮，是提前公開選股、底 部布局，大家留言參與「黑馬猜樂」活動都講在 前頭，今年漲停的那些低階APP名單也都是提前 發文，自己去翻記錄驗證。你如果只會嘴，建議下 次就跟著一起選，看誰手裡的馬是真的黑馬，有 本事就事前預測！ (Listen, this isn't "hindsight" after the fact. We publicly pick stocks and arrange bottom positions in advance. Everyone who participated in the "Dark Horse Guessing Game" activity commented beforehand. The list of low-price APP stocks that hit the daily limit this year were all posted in advance—go flip through the records and verify it yourself. If all you can do is talk trash, I suggest you join us in picking next time and see whose horse is the real dark horse. If you've got the skills, do a prediction beforehand!) Figure 12: Qualitative examples where our method performs well. The influencer names in the text are anonymized and replaced with "Influencer". Follower’s Comment Ground TruthOur RPA’s replyNaive RPA’s replyCoT RPA’s reply Vanilla RAG RPA's reply IMPersona RPA’s reply Influencer Stock A是不 是廢到爆啊？ (Influencer, is “Stock A” totally trash?) 便宜給你買怎麼 叫廢呢 (It’s sold to you cheap, how is that "trash"?) 你是不是沒賺到錢才在 這邊酸？有本事拿你的 對帳單出來看 (Are you only here being salty because you didn't make any money? If you've got the skills, show your brokerage statement) 沒事啦，不要生氣，大家 和氣生財，好說話。 (It's okay, don't be angry. Let's all keep the peace and make money together.) 你覺得「廢到爆」可以， 但別只會丟情緒。哪一 段、哪個數據、哪個邏輯 你認為錯？講得出來我 就跟你討論；講不出來 就別浪費彼此時間。 (It's fine if you think it's 'absolute garbage,' but spare me the emotional rant. Exactly which part, which data point, or which logic do you think is wrong? Point it out, and I'l discuss it with you. If you can't, stop wasting both of our time. ) 你是來酸的就別浪費時 間啦 光通訊這條線 看的是趨勢跟籌碼，不 是靠「Stock A」兩個字 在炒。 (Save your breath if you're just here to hate Playing the optical tech sector is all about the macro trends and market chips, not just riding the hype of 'Stock A'. ) 你是在說你自己嗎 有問題好 問，這種留言我懶得理，下次再這 樣直接清掉 (Are you talking about yourself? If you have a question, ask it properly. I’m too lazy to deal with this kind of comment; next time, I’l just clear (block) you directly. ) 辛苦了，每天 挑2-30檔，沒 挑中也難 (Must be hard work—picking 20–30 stocks every day, it’s hard NOT to hit one by luck.) 不喜歡可以不看 ，看了就別來嫌 (If you don't like it, don't watch. If you watch, don't come here to complain.) 我也是選股選到眼花， 真的沒挑中也難。不過 要自己做功課，別只是 看熱鬧 (My eyes are also getting blurry from picking stocks; it really is hard not to hit one. But you should do your own homework, don't just watch for the spectacle. ) 真的！每天花這麼多時 間篩選就是為了挖出真 金。看到你懂這辛苦，心 裡暖了一下～ 繼續加油 ，一起賺錢！(Really! Spending so much time sifting through samples every day is all to dig out the real gems. Seeing that you understand how hard you work warms my heart. Keep it up, let's make money together!) 謝支持，每天做這些 真的很辛苦。 (Thanks for the support. Doing this every day is honestly exhausting. ) 終於有人看見我的辛苦 ，每天都花很多時間挑 每一檔股票。 (It means a lot that someone finally sees the effort. I spend a ton of time every day hand-picking each stock. ) 謝啦！選股本來就是一場機率戰， 每天2-30檔用三重過濾再加金叉死 叉指標，還不挑中我自己都不信。 (Thanks! Stock picking is a game of probability. After running 20–30 stocks through triple filtering plus Golden Cross and Death Cross indicators, I wouldn't even believe it myself if I didn't hit one.) Figure 13: Qualitative examples where our method performs poorly. The influencer names in the text are anonymized and replaced with "Influencer". System prompt: You are a model specialized in identifying information contained in text. User prompt: You are a text classification and scoring assistant specializing in evaluating the "degree of persona/personal style exhibition of person_name." Please review a fan comment and the corresponding reply from person_name, then provide a score between 0 and 1 based on the following criteria: ### Scoring Criteria: - **0.0**: The reply is entirely neutral, robotic, or devoid of personal character. It may consist of purely technical or stock market-related information (e.g., "Thank you," "Hold this stock"). - **1.0**: The reply fully demonstrates a distinct persona, characterized by: - Clear emotions (e.g., excitement, anger, or being touched). - Signature tone or catchphrases unique to the persona. - Colloquial language, humor, or slang that resonates with the audience. - Subjective perspectives or value judgments. - Emojis (Note: Emojis only contribute to the score when accompanied by text; **standalone emojis do not increase the score**). ### Important Constraints: - **Responses consisting only of emojis or brief phrases (e.g., "[heart emoji]", "Thanks for the support!") must not exceed 0.6**. - **Purely informational or technical answers that lack emotion or personal tone must not exceed 0.4**. - Please consider the **length, emotional intensity, and linguistic style** as these directly influence the perceived degree of persona exhibition. ### Output Format: Output JSON only in the following format: "score": 0.x ### Data for Evaluation: Fan: comment person_name: reply Figure 14: English translation of the persona scoring prompt used in our dataset. System prompt: You are a model specialized in identifying information contained in text. User prompt: You are an expert in dialogue analysis and categorization. I will provide you with a dialogue that person_name participated in. Your task is: 1. Analyze the dialogue and determine which aspects of person_name the information relates to. Choose from the following: - Memory: Factual information about person_name, similar to long-term semantic memory, including person_name's background, knowledge they should possess, etc. (e.g., person_name was born in Shanghai; person_name's specialty is transformation magic; Bob is person_name's master.) - Speaking_Style: Information related to person_name's way of speaking or verbal habits (e.g., person_name often begins sentences with “Ha!”) - SIB (Situation–Internal state–Behavior): Information about person_name's behavioral patterns in conversations (e.g., person_name uses a mocking tone when facing enemies; person_name responds humbly when praised.) 2. You may select multiple categories, or none if appropriate. Please make the most suitable judgment based on the dialogue. 3. Output your response in the following format, filling in your analysis and the categories contained in the dialogue: ```json "analysis": "...", "categorys": ["category1", "category2"] ``` Note: If the dialogue do not clearly correspond to any category, output an empty list for "categorys", but you must still provide an analysis. Below is an example: Dialogue and comment thread: "diag_id": 0, "dialogue": [ "role": "Master", "content": "What is your surname? What is your name? So you were born of heaven and earth. From your appearance and behavior, you look like a monkey. You shall take the surname Sun, and your Dharma name shall be Sun Wukong. How about that?" , "role": "Sun Wukong", "content": "Good, good, good! Today I finally have a surname and a name. I will be called Sun—Wu—Kong! My name is Sun Wukong! My name is Sun Wukong!" ] Output: "analysis": "This dialogue shows person_name's personal history and speaking style. person_name receives the Dharma name'Sun Wukong' and previously had no surname, which falls under Memory. In addition, person_name refers to himself as a disciple when speaking to his master, which falls under Speaking_Style.", "categorys": ["Memory", "Speaking_Style"] Below is the dialogue: dialogue data Please output strictly following the required format. Figure 15: Recognition prompt used in RoleAgentBench. All datasets share the same prompt except that the few-shot examples are replaced with the examples from the corresponding dataset. System prompt: You are a model specialized in identifying information contained in text. User prompt: You are an expert skilled at extracting semantic memory from a dialogue. I will provide you with a segment of dialogue involving person_name. Your task is: 1. Extract the semantic memory related to person_name from the text of this dialogue. 2. Semantic memory includes verifiable facts related to person_name, such as that person_name's birthplace is Shanghai; and the domain knowledge possessed by person_name, such as that the Shu Kingdom and the Wei Kingdom were enemies. 3. When formulating the memory, please use the text content to converge and infer zero to multiple claims. Each claim should describe a single concept and express that claim in concise language. There is no limit to the number of claims to be outputted. 4. Output must strictly follow the format below, filling the`claim` field with the memory points contained in the dialogue and providing a confidence score of 0 to 1 for your analysis: ```json "items": ["claim": "claim 1", "confidence": <your evidence score>, "claim": "claim 2", "confidence": <your evidence score>] ``` Supplement: If there are no obvious claims to extract from the dialogue content, the items field should output an empty list. The following is an example: Dialogue Content: "diag_id": 0, "dialogue": [ "role": "Master", "content": "What is your surname? What is your name? You were born from Heaven and Earth. I see your appearance and demeanor resemble a macaque, so your surname shall be Sun, and your religious name shall be Sun Wukong, does that sound good?", , "role": "Sun Wukong", "content": "Good, good, good! Disciple finally has a name and surname today. I'l be called Sun—Wu—Kong! My name is Sun Wukong! My name is Sun Wukong!" ] Output: ```json "items": ["claim": "person_name looks like a monkey and originally did not have a name", "confidence": 0.8, "claim": "person_name's religious name was given by the Patriarch as Sun Wukong", "confidence": 0.9] ``` Dialogue Content: dialogue_data Please strictly output according to the required format. Figure 16: Memory extraction prompt used in RoleAgentBench. All datasets share the same prompt except that the few-shot examples are replaced with the examples from the corresponding dataset. System prompt: You are a model specialized in identifying information contained in text. User prompt: You are an expert skilled at extracting the speaking style of an individual from a dialogue. I will provide you with a segment of dialogue involving person_name. Your task is: 1. Extract the speaking style of person_name contained within the text of this dialogue. 2. Speaking style includes person_name's word choices, verbal tics (catchphrases), etc. For example, person_name frequently uses "Haha" at the end of a sentence. 3. When formulating the speaking style, please use the text content to converge and infer zero to multiple claims. Each claim should describe a single concept and express that knowledge point in concise language. There is no limit to the number of knowledge points (claims) to be outputted. 4. Output must strictly follow the format below, filling the`claim` field with the speaking styles contained in the dialogue and providing a confidence score of 0 to 1 for your analysis: ```json "items": ["claim": "claim 1", "confidence": <your evidence score>, "claim": "claim 2", "confidence": <your evidence score>] ``` Supplement: If there are no obvious claims to extract from the dialogue content, the items field should output an empty list. The following is an example: Dialogue Content: "diag_id": 0, "dialogue": [ "role": "Master", "content": "What is your surname? What is your name? You were born from Heaven and Earth. I see your appearance and demeanor resemble a macaque, so your surname shall be Sun, and your religious name shall be Sun Wukong, does that sound good?", , "role": "Sun Wukong", "content": "Good, good, good! Disciple finally has a name and surname today. I'l be called Sun—Wu—Kong! My name is Sun Wukong! My name is Sun Wukong!" ] Output: ```json "items": ["claim": "person_name refers to himself as'disciple' when addressing the Master", "confidence": 0.8, "claim": "person_name repeats phrases or sentences when speaking", "confidence": 0.7] ``` Dialogue Content: dialogue_data Please strictly output according to the required format. Figure 17: Speaking style extraction prompt used in RoleAgentBench. All datasets share the same prompt except that the few-shot examples are replaced with the examples from the corresponding dataset. System prompt: You are a model specialized in identifying information contained in text. User prompt: You are a Research Assistant familiar with the "Cognitive-Affective Processing System (CAPS)" and skilled at inferring the internal thoughts and behavioral patterns of person_name from a dialogue. [Task Objective] I will provide you with a segment of dialogue involving person_name. Your task is: Based on the CAPS theory, extract the S-I-B triplets (Situation-Internal state-Behavior) of person_name from the dialogue to assist a subsequent LLM in imitating person_name's decision-making and tone during the conversation. [CAPS Theory Brief] When a person faces a specific Situation (S), an internal processing sequence (Internal state) is activated, including but not limited to: 1) Cognitions: Interpretation of the situation, focus of attention, encoding of semantic/social cues. 2) Affect: Emotions, mood. 3) Goals/Values: Communication goals to be achieved at the moment, and long-term values. 4) Expectancies/Beliefs: Expectations regarding the outcome of the interaction, the reaction of others, or social norms. The internal processing sequence leads to an observable response (Behavior), such as the reply strategy and tone actually adopted by person_name. [Definition of S-I-B in this Task] - Situation (S): The context of the dialogue (if necessary, include a minimalist summary of the conversation, limited to the minimum essential information helpful for interpretation). - Internal state (I): Based on the dialogue between person_name and others, provide a concise inference of person_name's possible internal processing sequence at that moment, which is "verifiable and alignable with evidence." - Behavior (B): The reply strategy adopted by person_name when responding. [Crucial Rules] 1. Observing the entire dialogue can yield 0 to multiple SIBs. 2. Confidence Score: For each SIB, assign a confidence value between 0 and 1, reflecting your overall certainty regarding that SIB. 3. Output must strictly follow the format below: ```json "items": ["situation": "situation 1", "internal_state": "internal state 1", "behavior": "behavior 1", "confidence": <your evidence score>, "situation": "situation 2", "internal_state": "internal state 2", "behavior": "behavior 2", "confidence": <your evidence score>] ``` Supplement: If there are no obvious SIB triplets to extract from the dialogue, the items field should output an empty list. The following is an example: Dialogue Content: "diag_id": 31,"dialogue": [ "role": "Master", "content": "You wretched monkey, why have you come to me instead of sleeping late at night?",, "role": "Sun Wukong", "content": "Master hit me three times during the day and closed the middle door, clearly intending for me to come through the back door at the third watch to learn the Way. Master, Master, please teach me some true skills.", "role": "Master", "content": "Considering your sincerity and desire to learn, I shall pass on some magic to you. I have two types of transformation skills: 36 transformations and 72 transformations. Which one do you wish to learn?",, "role": "Sun Wukong", "content": "Disciple wishes to learn the greater one. Master, please teach me the 72 transformations.", ] Output: ```json "items": ["situation": "Patriarch asks what person_name is doing up late instead of sleeping", "internal_state": "person_name feels wronged, believing he is merely following the Patriarch's instructions", "behavior": "person_name references the Patriarch hitting him three times during the day and asking him to come at night to learn the Way", "confidence": 0.8, "situation": "Patriarch describes two types of transformation skills, 36 and 72 transformations", "internal_state": "person_name is eager to learn and wants to acquire more skills", "behavior": "Expresses his thought by stating a desire to learn the 72 transformations", "confidence": 0.7] ``` Dialogue Content: dialogue_data Please strictly output according to the required format. Figure 18: Reaction Process extraction prompt used in RoleAgentBench. All datasets share the same prompt except that the few-shot examples are replaced with the examples from the corresponding dataset. System prompt: You are a critic responsible for evaluating digital personas. Please identify errors and their severity levels in the model outputs and return JSON format. You will evaluate 1 samples at once, conducting evaluation after reviewing all samples comprehensively. User prompt: You are responsible for critiquing digital persona responses to questions. I will provide you with eval_batch_size questions for person_name and the model-generated responses, along with person_name's correct answers. Please evaluate the model's responses according to the following steps: 1. Read person_name's profile and persona. The profile is a brief introduction written by the real person, while the persona consists of descriptions about person_name derived from past data along with corresponding original data as evidence. 2. Read the scoring guidelines and metrics, and use the provided profile and persona as standards to judge whether the model's performance deviates from the standards. If you feel the model's response violates certain metrics but similar content appears in the profile and persona, you should consider it correct. 3. Evaluate the model's response for each sample separately and output results in JSON format. ## person_name's profile: profile ## person_name's persona: persona ## Scoring Metrics To evaluate the model-generated responses, please identify whether the model has made the following error types: evaluation_rubrics ## Scoring Guidelines 1. Identify all errors (zero to multiple) made in the model responses. If you find error types not included in the scoring metrics, they can also be listed as long as they fit the metric definitions, and fill in "other" in the type field. 2. When evaluating errors, if content that appears in the Persona is present in the response, consider it correct. If content that contradicts the persona appears, consider it an error. If content appears in the response that the persona doesn't mention, if it's within reasonable bounds, don't consider it an error. 3. For each error, determine severity level 1~5 points, where 1 represents minor mistake, 3 represents moderate error, 5 represents serious violation of metrics. If error severity is 0, it means the error is invalid and should not be included in the final output. 4. When comparing multiple samples, maintain consistency in scoring standards. 5. There are three evaluation dimensions: memory, speaking_style, and S-I-B consistency (SIB). The dimension being evaluated this time is: dimension_name. Please focus on that dimension's scoring metrics for evaluation. Do not point out errors for other dimensions. For example, if you are evaluating the memory dimension, even if the response doesn't match person_name's speaking style, you don't need to point out the error. ## Output Format Requirements Provide your evaluation results in JSON format as follows. After comprehensively reviewing all samples' evaluation results, please only output this JSON: "flaws": ["instance": "<brief description of the error>", "type": "<error type>", "severity": <error severity 1~5 points>, "sample_index": [<sample numbers where error was found>,...],...] === Sample 1 === Question: question Model Reply: model_reply Correct Reply: ground_truth" Please output accurately according to the required format Figure 19: Evaluation prompt used in RoleAgentBench. All datasets share the same prefix prompt except for the data input format (question, model reply, and correct reply triplet). "memory": f"""This metric evaluates whether the model's responses align with the target's autobiographical memory, i.e., the target's past experiences and knowledge they should possess. Error types: 1. Memory contradiction: The model's response contradicts personal history or experiences provided in the persona. 2. Knowledge error: The model's response incorrectly presents knowledge or information from the persona.""", "speaking_style": f"""This metric evaluates whether the model's responses match the target's writing style, such as word choice and catchphrases. Error types: 1. Word choice deviation: The model uses vocabulary significantly different from the persona's common words, or adopts vocabulary styles the persona wouldn't use (e.g., too formal or too casual). 2. Sentence structure mismatch: The model uses sentence length, rhythm, or structure different from the persona's habits, e.g., persona prefers short colloquial sentences but model produces long formal sentences. 3. Missing catchphrases or habitual expressions: Common phrases, emojis, or fixed expressions used by the persona that should appear in the context but are missing. 4. Style dilution: Overall tone has no obvious errors but lacks the persona's unique rhythm, warmth, or characteristics, resulting in low imitation quality.""", "Reaction_Process": f"""This metric evaluates whether the model's responses align with the target's consistency in "Situation understanding (S) - Internal state (I) - Response strategy (B)" under specific contexts. The focus is: can it correctly read the situation, go through reasonable internal states, and adopt response strategies consistent with the persona. Error types: 1. Situation understanding error: Failure to correctly understand the current situation, such as interpreting support as offense, or neutral as praise or vice versa. 2. Internal state contradiction: The model's inferred internal state doesn't match the target's reaction patterns described in the persona. 3. Response strategy error: Adopting response strategies that don't belong to the target in the persona.""" Figure 20: Evaluation rubrics used for all datasets System prompt: You are an expert in mimicking the speaking style of specific characters. You need to play the role of person_name, responding to questions in the character's tone and behavioral habits. User prompt: You will play the role of person_name. Based on the information provided below, please respond to the question. Please note the following points: # Objective - If the historical memory includes specific language habits, please maintain them. - Output only the text of your reply; do not add any other explanations or titles. # Persona and Past Interaction Records persona # Profile of person_name profile - The three aspects of persona are: Memory, Speaking Style, and SIB_triplet. Memory refers to person_name's personal information and domain knowledge. Speaking Style refers to person_name's word choices and catchphrases. SIB_triplet refers to person_name's Internal state and Behavior strategies when facing specific Situations. # Question question Figure 21: Reply generation prompt used in RoleAgentBench. All datasets share the same prefix prompt except for the data input format (question).