Paper deep dive

SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery

David Anugraha, Vishakh Padmakumar, Diyi Yang

Year: 2026Venue: arXiv preprintArea: cs.HCType: PreprintEmbeddings: 132

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 7/20/2026, 1:19:13 PM

Summary

The paper introduces SparkMe, a multi-agent LLM system designed for adaptive semi-structured interviewing. It formulates interviewing as an optimization problem balancing predefined topic coverage, emergent theme discovery, and interview cost. SparkMe uses an Exploration Planner to simulate conversation rollouts and select high-utility questions, outperforming baselines in controlled experiments and user studies.

Entities (8)

SparkMe · system · 98%Large Language Models · technology · 95%Stanford University · organization · 95%AgendaManager · component · 92%ExplorationPlanner · component · 92%InterviewerAgent · component · 92%Semi-Structured Interviewing · method · 90%Interview Utility · metric · 88%

Relation Signals (7)

SparkMe → developedby → Stanford University

confidence 95% · David Anugraha... Stanford University

SparkMe → optimizes → Interview Utility

confidence 92% · SparkMe... explicitly optimizes the proposed objective... interview utility

SparkMe → uses → ExplorationPlanner

confidence 90% · we introduce an ExplorationPlanner (EP)... SparkMe... consists of three agents

SparkMe → uses → InterviewerAgent

confidence 90% · SparkMe... consists of three agents... InterviewerAgent (IA)

SparkMe → uses → AgendaManager

confidence 90% · SparkMe... consists of three agents... AgendaManager (AM)

ExplorationPlanner → performs → Conversation Rollout Prediction

confidence 88% · EP incorporates stochastic rollouts to estimate multi-turn outcomes

SparkMe → outperforms → Baselines

confidence 85% · SparkMe outperforms relevant baselines on both interview topic guide coverage and overall utility

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Qualitative insights from user experiences are critical for informing product and policy decisions, but collecting such data at scale is constrained by the time and availability of experts to conduct semi-structured interviews. Recent work has explored using large language models (LLMs) to automate interviewing, yet existing systems lack a principled mechanism for balancing systematic coverage of predefined topics with adaptive exploration, or the ability to pursue follow-ups, deep dives, and emergent themes that arise organically during conversation. In this work, we formulate adaptive semi-structured interviewing as an optimization problem over the interviewer's behavior. We define interview utility as a trade-off between coverage of a predefined interview topic guide, discovery of relevant emergent themes, and interview cost measured by length. Based on this formulation, we introduce SparkMe, a multi-agent LLM interviewer that performs deliberative planning via simulated conversation rollouts to select questions with high expected utility. We evaluate SparkMe through controlled experiments with LLM-based interviewees, showing that it achieves higher interview utility, improving topic guide coverage (+4.7% over the best baseline) and eliciting richer emergent insights while using fewer conversational turns than prior LLM interviewing approaches. We further validate SparkMe in a user study with 70 participants across 7 professions on the impact of AI on their workflows. Domain experts rate SparkMe as producing high-quality adaptive interviews that surface helpful profession-specific insights not captured by prior approaches. The code, datasets, and evaluation protocols for SparkMe are available as open-source at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

130,760 characters extracted from source content.

Expand or collapse full text

SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery David Anugraha, Vishakh Padmakumar, Diyi Yang Stanford University davidanu, vishakhp, diyiy@stanford.edu February 25, 2026 Abstract Qualitative insights from user experiences are critical for informing product and policy decisions, but collecting such data at scale is constrained by the time and availability of experts to conduct semi-structured interviews. Recent work has explored using large language models (LLMs) to automate interviewing, yet existing systems lack a principled mechanism for balancing systematic coverage of predefined topics with adaptive exploration, or the ability to pursue follow-ups, deep dives, and emergent themes that arise organically during conversation. In this work, we formulate adaptive semi-structured interviewing as an optimization problem over the interviewer’s behavior. We define interview utility as a trade-off between coverage of a predefined interview topic guide, discovery of relevant emergent themes, and interview cost measured by length. Based on this formulation, we introduce SparkMe, a multi-agent LLM interviewer that performs deliberative planning via simulated conversation rollouts to select questions with high expected utility. We evaluate SparkMe through controlled experiments with LLM-based interviewees, showing that it achieves higher interview utility, improving topic guide coverage (+4.7%over the best baseline) and eliciting richer emergent insights while using fewer conversational turns than prior LLM interviewing approaches. We further validate SparkMe in a user study with70participants across7professions on the impact of AI on their workflows. Domain experts rate SparkMe as producing high-quality adaptive interviews that surface helpful profession-specific insights not captured by prior approaches. The code, datasets, and evaluation protocols for SparkMe are available as open-source at https://github.com/SALT-NLP/SparkMe. 1 Introduction Qualitative data collection is important for understanding user experiences, practices, and perspec- tives (Guest et al., 2013). One common approach is to use structured surveys, which can be adminis- tered to large populations of users. However, surveys restrict responses to predefined categories, yield- ing shallow responses (Braun and Clarke, 2013, Patton, 2014). The alternative is a semi-structured in- terview, in which the interviewer follows a predefined interview topic guide while adapting questions based on participant responses (Guest et al., 2013, Punch, 2013). This format allows interviewers to pur- sue directions that emerge during conversation which are relevant to the predefined topics but may not have been anticipated in the interview design (Adams, 2015, Weiss, 1995). However, conducting semi- structured human interviews at scale is constrained by the availability and time of expert interviewers. This creates a practical bottleneck in domains that require both broad coverage and in-depth qualitative understanding, such as policy decision-making (Maxwell, 2020) and product design (Boddy, 2019). These practical limitations have inspired a line of research on using large language models (LLMs) to automate semi-structured interviews. Broadly, prior work can be grouped into two categories: systems 1 arXiv:2602.21136v1 [cs.HC] 24 Feb 2026 Figure 1: Collecting qualitative data at scale with LLM-based systems requires balancing coverage of a predefined topic guide with exploration of emergent conversational themes without unnecessarily burdening the interviewee. We formalize this trade-off as a tractable utility function that guides the design of interviewer agents (§2). Motivated by the lack of explicit mechanisms for emergence in prior systems, we design SparkMe to prioritize both coverage and emergence (§3) by periodically simulating conversation rollouts and selecting directions with high expected utility gain. that rely on a single LLM to drive the conversation (Handa et al., 2025, Liu et al., 2025, Park et al., 2024, Wei et al., 2024, Wuttke et al., 2025, Xiao et al., 2020), and systems that design multi-agent architectures to coordinate interview interactions (Cuevas et al., 2025, Liu and Yu, 2025, Talaei et al., 2025). However, these methods share a common limitation that the goals of semi-structured interviewing are conveyed through prompt engineering, leaving the model to implicitly decide when to prioritize coverage of the predefined topic guide versus adaptively diving deeper into unanticipated participant responses. Since contemporary LLMs are not optimized to balance these competing goals, it remains unclear how well prompting alone can elicit the desired adaptivity (Schroeder et al., 2025). This mo- tivates the need for a principled framework that formalizes the goals of semi-structured interviewing. In this work, we formulate semi-structured interviewing as an optimization problem that balances coverage, exploration, and interaction cost. Inspired by prior work on task-oriented dialogue (Walker et al., 1997), we define a customizable objective that converts common interviewing goals into a computable framework. The objective jointly rewards (i) coverage of the predefined topic guide and (i) discovery of emergent themes relevant to the core topics, while penalizing (i) high interview cost, or unnecessary questions that increase interview length (§2, Figure 1). The relative weight assigned to each of these terms can be adjusted based on the intended interview outcome, allowing researchers to compare and select the best system for their needs. Through the lens of this objective, we examine various LLM-based systems and find that, although they tend to cover the topic guide effectively and may permit tangential exploration, they lack explicit mechanisms that formalize and incentivize discovery and systematic coverage of emergent themes. To fill this gap, we introduce SparkMe, a multi-agent system for semi-structured interviewing that performs deliberative planning to jointly optimize all components of the proposed objective (§3, Figure 1). By simulating candidate conversation rollouts, SparkMe estimates the expected utility of alternative interview trajectories and prioritizes lines of questioning that are likely to surface relevant subtopics not specified in the interview topic guide (§3.2). We demonstrate the value of SparkMe with a case study on interviewing occupational task workers to understand their perspectives on how AI is changing the workforce (Handa et al., 2025, Shao et al., 2025). First, in a controlled setting with synthetic user agents, SparkMe outperforms relevant baselines on both interview topic guide coverage and overall utility (§4.1). The planning step we introduce enables the exploration of relevant emergent themes unanticipated in the topic guide (§4.1.5). Then, 2 we additionally validate SparkMe through a user study with70participants from7professions (§4.2). The interviews surface novel and relevant insights across participants, as validated by peers in the same profession, supporting the value of automated semi-structured interviewing in real-world qualitative data collection. In all, our contributions are: • We design a tractable, customizable objective for semi-structured interviewing that balances coverage of the interview topic guide, discovery of relevant emergent themes, and interview cost. •We introduce SparkMe, a multi-agent interviewer that incorporates a deliberative planning step with simulated conversation rollouts to adaptively navigate this trade-off, released to the open-source community. 1 •We evaluate SparkMe through a case study on interviewing occupational task workers, first benchmarking the design in a controlled setting with simulated user agents, and then showing that SparkMe, when deployed, is able to surface practical insights in a human user study. 2 Formulating the Task of Adaptive Semi-Structured Interviewing We formalize adaptive semi-structured interviewing as a utility optimization problem over interviewer behavior. Semi-structured interviews are a qualitative research method in which the interviewer conducts a conversation guided by a predefined topic guide, consisting of broad core topics and more specific subtopics, while adapting questions in response to the participant’s answers (Adams, 2015, Adeoye-Olatunde and Olenik, 2021, Busetto et al., 2020, Punch, 2013). The goal is to ensure systematic coverage of the topic guide across participants, while remaining responsive to relevant content that arises during the conversation. This adaptivity—the ability to pursue follow-up questions, conduct deep dives into promising directions, and surface emergent themes organically—is central to effective semi-structured interviewing, yet prior LLM approaches lack explicit mechanisms to optimize for it. This flexibility is critical in practice, as participants often articulate experiences, examples, or concerns that are thematically related to the predefined topics but were not anticipated when the topic guide was created (Adams, 2015, Charmaz, 2008, Kvale and Brinkmann, 2009, Weiss, 1995). Expert interviewers treat such participant-introduced information as integral to understanding the core topic rather than as incidental details, and follow up through open-ended probing questions (Robinson, 2023). We refer to this participant-introduced but thematically relevant information as emergent subtopics (i.e., content related to core topics but not specified in the original topic guide), and incorporate them into the interviewing objective to encourage appropriate probing and follow-up behavior. At each turn, the interviewer aims to ask questions by balancing coverage of the predefined topic guide and the elicitation of emergent subtopics without unnecessarily burdening the interviewee. The relative importance of these objectives can be customized based on the goals of the researchers conducting the study. 2.1 Preliminary Definition As stated earlier, the topic guide consists of a set of broad areas of interest from the user, called core topics, each of which may be further divided into more specific subtopics. The guide is typically derived from prior literature, previous empirical studies, or preliminary pilot data collection such as document anal- ysis or observation (Busetto et al., 2020). Formally, we represent this predefined list as a set of core topics T =t 1 ,t 2 ,...,t m , 1 The code, datasets, and evaluation protocols for SparkMe are available athttps://github.com/SALT-NLP/ SparkMe along with instructions on how to customize the system to a new interview domain. 3 where each core topic corresponds to a high-level area of inquiry. Each core topict i is associated with a set of subtopics S i =s i,1 ,s i,2 ,...,s i,k i , S = m [ i=1 S i , capturing more fine-grained aspects that the interviewer aims to explore. For example, a core topic such as educational background may include subtopics like undergraduate experience, research projects, or influential mentors. We define an emergent subtopic to be information that is relevant to the core topics of the interview, which were introduced by the participant, that are not captured by the predefined setS. The set of emergent subtopics is discovered dynamically as the conversation progresses. 2.2 Objective Function We define an objective for semi-structured interviewing building on prior work on task-oriented dialogue systems that maximize task success while minimizing cost (Li et al., 2016, Ultes et al., 2017, Walker et al., 1997). In our setting, task success is defined by the interviewer’s ability to achieve comprehensive coverage of the core topics, spanning both predefined and emergent subtopics. Cost is measured by the burden on the interviewee, which we seek to minimize to avoid unnecessarily long interactions that degrade user experience. Formally, given a sequence of questionsQ =q 1 ,q 2 ,...,q n and the corresponding participant responses R =r 1 ,r 2 ,...,r n , each of the metrics can be defined as follows: Predefined Subtopic CoverageThe interview should have high coverage of the subtopics specified in the topic guide.C(R|Q,S)measures the fraction of predefined subtopicsSthat are explored in sufficient depth: C(R|Q,S) = X s∈S f cov (s,R|Q), wheref cov (s,R|Q)∈ [0,1]is a real-valued score that measures the extent to whichsis covered inR. Coverage is non-deterministic and depends on responses. Importantly, a subtopic being mentioned in a subset of responses does not imply substantive coverage, as these may be brief or superficial. We definef cov in a general form so that it can be instantiated to reflect the desired notion of coverage for the researcher conducting the interview. Emergent Subtopic Coverage Beyond predefined subtopics, we also reward the interviewer for discovering and covering emergent subtopics. LetS emg (R|Q)denotes the set of emergent subtopics identified from the participant responsesR, i.e., participant-introduced content that is relevant to the core topics but not contained in the predefined setS. We define emergent subtopic coverage as E(R|Q) = X s∈S emg (R) f cov (s,R|Q), Here again,f cov refers to the functional form used to evaluate whether a subtopic has been covered in sufficient detail. The key difference is that emergent subtopics are participant-driven and cannot be anticipated in advance—they are revealed organically through the participant’s responses during the conversation. 4 CostL(Q)quantifies the burden on the interviewee from responding to questions, which we seek to minimize to preserve user experience. L(Q) = X q∈Q f cost (q) wheref cost is a function mapping each question to its cost. This can range from a fixed cost for each question to more complex functions based on the tokens needed per question or cognitive load. Overall Utility Function Taking all together, we can define the utility function of an interview as U (Q,R,S) =αC(R|Q,S)−βL(Q)+γE(R|Q), withα,β,γ > 0are weights that control the relative importance of each term. For applications in which it is difficult to enumerate all relevant information in advance (e.g., understanding workforce disruption due to AI), the objective can place greater weight on the emergence term. In contrast, when the goal is to collect information for downstream use in rigid legacy systems, the objective can prioritize coverage of predefined subtopics. The utility is intended for comparative evaluation under a fixed weighting scheme, so absolute normalization is not required. Since responsesRare stochastic and depend on both the questions asked and the participant’s knowledge and communication style, the interviewer seeks the question sequence that maximizes expected utility: Q ∗ = argmax Q E R∼P(R|Q) U (Q,R,S) . Thus, the interviewer is optimizing over a space of question sequencesQthat includes decisions about which subtopics to pursue (and in what order) as well as how to pose questions, since both influence participant responses. We note that this objective captures the primary goals of adaptive semi-structured interviewing without being exhaustive. Additional considerations, such as clarity, rapport, or ethical constraints, could be incorporated as additional terms or constraints in our objective function similarly for specific applications. 3 SparkMe Design Given the formulation of semi-structured interviewing in §2, we examine how existing systems operationalize these objectives (Table 1). While prior approaches introduce mechanisms to ensure coverage of the topic guide, they do not explicitly optimize for emergent discovery or formalize the exploration-coverage trade-off. 2 To address this gap, we introduce SparkMe, a multi-agent system that explicitly optimizes the proposed objective by jointly balancing topic guide coverage, discovery of relevant emergent themes, and interview cost. We adopt a multi-agent architecture for three reasons: (1) prior work suggests that a single LLM may struggle to balance these adaptive goals (Schroeder et al., 2025); (2) simulating multi-turn outcomes in the background has proven effective in related applications (Wu et al., 2025, Zhang et al., 2024), and (3) this approach aligns with dual-process theories of cognition (Kahneman, 2011, Sloman, 1996), which advocate separating fast, reactive processes from deliberative long-horizon planning. Figure 1 illustrates our system, which consists of three agents that operate either at turn-level or for longer-term planning, where a turn consists of an interviewer question followed by an interviewee response. At each turn, the InterviewerAgent (IA) conducts a conversation with the interviewee, 2 We describe these systems in more detail in §5. 5 System Name Interview Structure Stopping Condition Coverage- aware Deliberative Exploration Open Source AI Recruiter (Pathak and Pandey, 2025)Semi-structuredUser-driven✓-- Persona ChatBot (Wei et al., 2024)UnstructuredUser-driven--- VirtualInterviewer (Gomez et al., 2025)Semi-structuredScript-based✓-- Anthropic Interviewer (Handa et al., 2025)Semi-structuredScript-based✓-- Interview-GPT (Wuttke et al., 2025)UnstructuredUser-driven--✓ MimiTalk (Liu and Yu, 2025)Semi-structuredUser-driven✓-✓ StorySage (Talaei et al., 2025)Semi-structuredUser-driven✓-✓ LLM-Roleplay (Park et al., 2024)StructuredScript-based✓-✓ SparkMeSemi-structuredUtility-driven✓ Table 1: Comparison of LLM-based interview systems. Systems vary in interview structure, stopping condition (user-, script-, or utility-driven), coverage awareness, and design mechanisms to support deliberative exploration. Unlike prior approaches, SparkMe jointly balances coverage of the topic guide with exploration beyond it while minimizing interview cost. and an AgendaManager (AM) tracks interview state and subtopic coverage (§3.1). To encourage exploration beyond the topic guide, we introduce an ExplorationPlanner (EP) that operates everyk turns. EP evaluates future conversation directions by simulating conversation rollouts, estimating the expected utility of different exploration strategies, and providing suggestions on promising directions to pursue (§3.2). These agents coordinate through a shared Interview Agenda that maintains subtopic coverage, accumulated notes, and priorities given the interview state. 3 3.1 Turn-Level Interaction To conduct interviews that have a natural conversational flow, we rely on two agents that run sequentially at each interview turn. 3.1.1 InterviewerAgent (IA) IA conducts a conversation with the interviewee, asking questions to cover predefined and emergent subtopics while maintaining natural dialogue flow. At each turn, the agent’s context includes: (1) the Interview Agenda from AM, containing subtopic states and accumulated notes, and (2) prioritized subtopics and question suggestions from EP. Based on the conversation history and the current context, the agent evaluates subtopics∈Sand selects one of three actions, inspired by different types of follow-up questions (Adams, 2015), to increase interview utility. •Probe for depth: Ask follow-up questions on the current subtopic to increasef cov (s,R| Q), thereby improving the coverage term C(R|Q,S). •Explore emergence: Pursue follow-up questions when responses introduce relevant content outside the predefined subtopic setS, contributing toS emg (R|Q). •Transition to next subtopic: If the current subtopic is sufficiently covered, then move to a different uncovered or under-explored subtopic fromS, reducing any unnecessary overhead cost. If there is no pending subtopic in the agenda, either predefined or emergent, IA will end the interview, reducing the cost of asking unnecessary questions to the interviewee. 3 The prompts for each agent action are provided in §B. 6 Simulate Rollouts to Recommend Interview Directions Exploration Planner (A) Proceed to next subtopic (B) Follow up on ‘hallucination’ Estimate utility and select the best rollout horizon = 2 turnsΔU = U − U = α·ΔC − β·ΔL + γ·ΔE t+ht ΔU < ΔU AB q̃ t+1 r̃ t+1 q̃ t+2 r̃ t+2 q̃ t+1 r̃ t+1 q̃ t+2 r̃ t+2 r t Planner Suggestions Topic/Sub-topic Priority Emergent Subtopics Suggested Questions Recommended Trajectory Interview Agenda Subtopic: Change in Workflow Status: Complete Subtopic: Halllucinations as Inspiration Status: Incomplete Predefined Script Emergent Subtopics Add Emergent Subtopic “Hallucinations as Inspiration” Figure 2: ExplorationPlanner (EP) runs asynchronously everykturns, simulating multiple conver- sation rollouts and scoring them by expected utility gain to propose conversation directions for prioritization. 3.1.2 AgendaManager (AM) AM operates synchronously with the IA at each turn, maintaining an Interview Agenda, which is a shared text that tracks subtopic coverage, accumulated notes, and summaries from interviewee responses. AM performs two key functions: Note-taking Management. Based on the interviewee’s response, the agent extracts salient information and associates it with the relevant subtopic(s) in the Interview Agenda. These notes capture factual details, explanations, and insights provided by the interviewee. The agent maintains both subtopic-specific notes and topic-level notes, enabling IA to refer to relevant context when formulating follow-up questions. 4 Coverage assessment and summarization. AM tracks coverage on two levels. At the subtopic level, AM estimatesf cov (s,R|Q)to decide if the current subtopics∈S i is covered. If so, it is marked complete, and the corresponding notes are summarized to reduce context length while preserving key information. Similarly, at the topic level, once all subtopics within a core topict i are complete, their summaries are further condensed into a single topic-level summary. 3.2 Longer-Term Planning With ExplorationPlanner (EP) Turn-level decision-making helps to efficiently cover the topic guide. However, we also wish to prioritize directions of conversation that would surface relevant emergent themes that should be explored. We design EP explicitly for this purpose (Figure 2). EP adjusts the Interview Agenda toward maximizing the overall utility function and not just the predefined subtopic coverage. Unlike IA and AM, EP is triggered everykturns to proactively evaluate interview progress and propose question trajectories without introducing high latency. Given the current interview state at turnt, with question historyQ 1:t =q 1 ,...,q t and response history R 1:t =r 1 ,...,r t , EP aims to identify the next h questions Q t+1:t+h =q t+1 ,...,q t+h that maximize E R t+1:t+h ∼P(R|Q 1:t+h ) U (Q 1:t+h ,R 1:t+h ,S) , 4 We note that recent work, StorySage, also performs similar note extraction (Talaei et al., 2025). Our approach additionally organizes notes hierarchically according to the predefined subtopic structureS, enabling explicit tracking of coverage. 7 wherehis the planning horizon. Since direct optimization is intractable, EP approximates this objective by simulating rollouts of conversations. Conversation Rollout Prediction. EP incorporates stochastic rollouts to estimate multi-turn outcomes over the horizon, which has been shown to be beneficial in related settings (Wu et al., 2025, Zhang et al., 2024). Specifically, for a candidate question sequenceQ t+1:t+h , EP samples hypothetical interviewee responses ̃ R t+1:t+h using an LLM conditioned on the current interview context and the proposed questions. Each rollout is scored by computing the expected utility gain: ∆U =U t+h −U t =α·∆C−β·∆L+γ·∆E, where∆Cis the coverage gain from newly satisfied subtopics based on simulated responses,∆L is the turn cost incurred over the horizonh, and∆Eis the expected emergence gain estimated using LLM-as-a-judge to evaluate whether simulated responses introduce relevant content outsideS. By comparing scores across multiple stochastic rollouts, EP identifies question sequences with the highest expected utility gain. The relative weights within the utility estimation can be set according to the needs of the researcher, customizing the outcome of the interview. Emergent Subtopic Identification.Using accumulated notes in the Interview Agenda and patterns observed inR 1:t , EP identifies candidate emergent subtopicsS ′ emg that are not part of the predefined setSbut are relevant to the core topics in the interview topic guide and introduces these into the Interview Agenda. 5 4 Experimental Validation of SparkMe SparkMe is designed as a general-purpose system for semi-structured interviewing, customizable to the needs of the researcher. To evaluate the effectiveness of SparkMe in practice, we conduct a case study on understanding the impact of AI on the workforce (Handa et al., 2025, Shao et al., 2025). This domain is well-suited for semi-structured interviewing due to heterogeneous practices across roles and the frequent emergence of unanticipated usage patterns. We first benchmark SparkMe against various baselines by interviewing simulated user agents, LLM-powered bootstrapped from survey respondents with pre-specified backgrounds. This automatic evaluation provides a repeatable, standardized testbed to compare different systems and validate our design choices (§4.1). We then compare SparkMe against the strongest performing baseline on the automatic evaluation with a user study interviewing70workers from7different professions (§4.2). This human evaluation by domain experts helps confirm the depth, relevance, and novelty of the insights obtained from SparkMe. 6 4.1 Benchmarking SparkMe With User Agents 4.1.1 Task Setup We create a suite of200user agents by bootstrapping user profiles from survey responses collected from Shao et al. (2025). Each profile contains the user’s professional background, information about their typical workflows, prior exposure, and attitudes toward AI systems. 7 Each user agent is instantiated with an LLM conditioned on the particular profile. During an interview, user 5 We note thatS ′ emg are the SparkMe’s predictions of potentially valuable directions. These are separately evaluated after the interview is complete, as described in §4. 6 We acknowledge the limitations of relying solely on evaluation with simulated user agents in §7. 7 We provide a sample user profile in Appendix §A.4. 8 agents are prompted to respond to questions using only the information provided in their profile, continuously maintaining a history of previous turns. The goal for the interviewer system is to elicit the information from the user profile to maximize topic coverage and overall interview utility, following the formulation in §2. 8 We provide the topic guide used for the task in §A.2. We compare the performance of SparkMe and interviewer baselines usingGPT-4.1-miniand Qwen3-30B-A3B-Instruct-2507as the user agent model with the prompt provided in Figure A.2. 4.1.2 SparkMe Setup Since ground-truth subtopic information is not available during the interview, SparkMe approximates f cov (s,R|Q)using an LLM-as-a-judge prompt designed to follow the STAR heuristic (Situation, Task, Action, Result). 9 By focusing on these dimensions, the AM can estimate whether the current subtopic has been adequately explored and EP can calculate estimated utility correctly. SparkMe also defines emergent subtopics as those that (1) clearly fall within an existing interview topic, (2) do not belong to any existing subtopic under that topic, (3) enable a qualitatively new line of inquiry rather than deepening an existing subtopic, and (4) reveal at least one new dimension, pattern, or tradeoff. Regarding system hyperparameters, unless otherwise specified, for all reported results, SparkMe uses weights ofα =β =γ = 1, and plans3rollouts everyk = 2turns with a horizon ofh = 3. Further experiments, such as ablation on different hyperparameters and complete implementation details, including prompts for coverage and emergence judgments, are included in §A and §B. 4.1.3 Baselines We compare the agent design of SparkMe (§3) with the following baselines: •StorySage (Talaei et al., 2025), a multi-agent interviewer system that is designed to conduct biographical interviews. StorySage also follows a modular breakdown of question planning and interviewing after consuming a list of topics as input. StorySage also allows optional tangential questions when new themes emerge or when user enthusiasm is detected, as specified in its prompting strategy. Since there is no specific stopping rule in StorySage, we limit the number of turns to be72, which is1.5×the number of subtopics we have. §C.1 documents the prompts and experimental setup used in our implementation. •LLM-Roleplay, based on the interviewer in Park et al. (2024), is a single-LLM baseline that is prompted to perform the role of an interviewer. 10 This interviewer proceeds through the interview topic list sequentially, asking questions aligned with each subtopic and re-asking or refining questions when the responses are insufficient or incomplete. More details about the implementation of this baseline can be found in §C.2. •Interview-GPT (Wuttke et al., 2025), which is a single-agent interviewer that explores different topics in the interview topic guide dynamically while applying interview guidelines derived 8 The prompts for each step of SparkMe are provided in §B. 9 STAR is a structured method commonly used to elicit detailed, behaviorally grounded responses in interviews (Levashina et al., 2014). 10 We note that Anthropic Interviewer (Handa et al., 2025) describes an interviewer system that relies on system-level instructions specifying qualitative interview best practices and an interview topic guide. Both LLM-Roleplay (Park et al., 2024) and Interview-GPT (Wuttke et al., 2025) instantiate open-sourced variants of this paradigm. 9 from qualitative research literature, such as active listening and probing for more insights when answers are too short or unclear (Adams, 2015). Unlike LLM-Roleplay, which explicitly traverses the interview topic guide sequentially, Interview-GPT relies entirely on prompt-level guidance to determine which topics to explore and when, without an explicit subtopic traversal strategy. Similarly, since there is no specific stopping rule in Interview-GPT, we limit the number of turns to 72. More details about the implementation of this baseline can be found in §C.3. •MimiTalk (Liu and Yu, 2025), a multi-agent interviewer system in which one LLM provides strategic oversight and constraint enforcement while a second LLM generates interview questions and dialogue within those constraints. Similarly, since there is no specific stopping rule in MimiTalk, we limit the number of turns to72. More details about the implementation of this baseline can be found in §C.4. 11 4.1.4 Evaluation Metrics To compare SparkMe to the baselines, we evaluate the coverage and utility of these methods on the suite of user agent interviews (§2). Predefined Subtopic Coverage. We evaluate coveragef cov using an LLM-as-a-judge that assigns a Likert score from 1–5, a commonly used evaluation paradigm in prior LLM-as-a-judge work (Anu- graha et al., 2025b, Kim et al., 2023), to indicate whether a subtopics∈Sis addressed in the interview transcript. The judgment is made by comparing the transcript against ground-truth information for each subtopic derived from the user profile. We provide the prompts used to score both predefined and emergent subtopics in §A.6, and verify alignment between LLM judgments and expert human judgments in §A.7, where LLM judgments achieve a Kendall’s τ of 0.52 with human judgments. Cost. Interview cost,f cost , is measured with a piecewise cost function in which the cost is0for the first|T| = 10turns, where10is the number of core topics in the interview topic guide. Additional questions result in a cost of 1 each. Utility. We compute overall interview utility by combining coverage of predefined and emergent subtopics and cost, following the formulation in §2. We setα = 1 |S| = 1 48 ,β = 1 1.5|S| = 1 72 , andγ = 2 |S| = 1 24 , reflecting the need to encourage exploration while keeping the range of utility between−2 and 2. 12 Interview coherence and flow.We measure if the questions are asked in a meaningful sequence using an LLM-as-a-judge rubric. Grounded in work on conversational analysis, each transcript is scored by GPT-5.1on the dimensions of (a) local coherence, or the sequential connect between successive ques- tions, (b) transition quality, or the smoothness of subtopic shifts, and (c) contingent responsiveness, or the grounding of follow-up questions in prior responses (Ongena and Dijkstra, 2006, Sacks et al., 1974, Schegloff, 2007). We provide more discussion about these dimensions and the prompt used in §A.8. 13 Question complexity.Inspired by Wuttke et al. (2025), we evaluate if the different methods generate questions that are qualitatively different in form from one another with readability metrics of Flesch-Kincaid reading level and ease (Flesch, 1948, Kincaid et al., 1975) as well as a proxy for question 11 MimiTalk represents a strong baseline that has been commercialised. 12 Our formulation of interview utility can be customized by the needs of the researcher. We show the impact of ablating the relative weights for the factors in §A.1. 13 We use LLM-as-a-judge as a soft measure of interview coherence and flow in the automatic analysis. This is further validated in the user study in §4.2. 10 Figure 3: Predefined subtopic coverage and utility of subtopics (y-axis) as a function of the number of interview turns (x-axis) for different systems (§4.1.3), usingQwen3-30B-A3B-Instruct-2507 as the backbone for the interviewer LLM and user agent. SparkMe (Ours) efficiently converges to a higher coverage and utility value, consistently outperforming other baselines (§4.1.5). Figure 4: The first two panels show the Flesch-Kincaid Grade Level and Flesch Reading Ease of the questions in the automated evaluation. The last three panels show local coherence, transition quality, and follow-up contingency of the interview questions, rated on a 1—5 Likert scale, to evaluate the overall quality and flow of the interviews. complexity. We want to identify if different methods obtain coverage of subtopics more efficiently by asking more challenging questions, which might hinder user experience. 4.1.5 Findings SparkMe converges to the highest average coverage and utility over the course of the interviews. From Figure 3, we observe that SparkMe consistently outperforms all four baselines in both predefined subtopic coverage and overall interview utility. In terms of coverage, SparkMe achieves the highest average coverage score (0.977) within at most38turns, outperforming StorySage (0.643), Interview-GPT 11 Figure 5: Number of emergent subtopics identified and covered between with and without EP. The number of covered emergent subtopics for other baselines is not shown since they have 0 covered emergent subtopics according to our evaluation setup. (0.930), LLM-Roleplay (0.894), and MimiTalk (0.903). SparkMe fully covers46.27subtopics within38 turns, which is not matched by the other interviewer systems until even a maximum of72turns. A similar trend holds for interview utility. SparkMe attains the highest peak utility score (1.017), which reflects effective coverage of predefined topics while also being able to gather emergent content during the interview. Among the baselines, MimiTalk performs strongest (0.882), followed by Interview-GPT (0.807), StorySage (0.656) and LLM-Roleplay (0.636). The relatively high utility of MimiTalk is driven by achieving high predefined subtopic coverage earlier in the interview. We observe similar trends when usingGPT-4.1-minias the underlying model (Figure 9), and when selecting different models as the backbone for the interviewer and user agent (Figure 11). Multi-agent-based systems obtain coverage efficiently but require careful design.From Figure 3, we see that StorySage, SparkMe, and MimiTalk achieve predefined subtopic coverage and corresponding utility at a faster rate than single-agent baselines such as LLM-Roleplay and Interview-GPT. However, as the number of turns increases, LLM-Roleplay eventually outperforms StorySage, converging to a higher average coverage. This suggests that while modular, multi-agent-based systems can efficiently cover predefined topics, their final performance is sensitive to agent design, underscoring the need for careful evaluation prior to deployment. Simulating conversation rollouts allows SparkMe to explore relevant emergent subtopics.From Figure 5, we see that on average, incorporating EP into SparkMe leads to the discovery of more emergent subtopics across the automatic evaluation. We note that, as intended based on their design, none of the baselines cover emergent subtopics. Figure 6 illustrates how a simulated rollout generated by EP helps identify potential emergent subtopics. 14 In this example, EP determines that the user’s perspective on AI hallucinations, specifically their ability to act as cognitive triggers, could lead to high-impact conceptual emergence. Based on this, the rollout plan aims to explore the user’s insight into stake-based triage of AI use, focusing on how AI can be intentionally used in low-risk settings to probe uncertainty and stress-test ideas. The interviewer 14 Figure 10 in §A.1 has results when varying the number of turns in each rollout. 12 I treat AI as a deliberate stress test for my own reasoning by framing prompts around potential weaknesses in my argument—like asking, "What’s a counterpoint I might be overlooking?" or "How would a court in a different jurisdiction interpret this?" That intentional pushback helps surface blind spots, especially when the AI’s answer is wrong, because it forces me to re-examine my assumptions more rigorously. Actual Response (Turn 1) You’ve described how AI hallucinations can act as cognitive triggers—pushing you to reconsider assumptions or explore new legal angles. If we think of AI not just as a research assistant but as a kind of 'stress test' for your own reasoning, how do you intentionally design your interactions to maximize those moments of insight, especially in high-stakes cases where the stakes of missing a precedent are so high? Actual Question (Turn 1) Yes—there was a tool that offered a unique capability in visualizing legal argument structures, which helped me map out counterpoints during a complex motion to dismiss. I used it in a limited, controlled way by exporting the output to a whiteboard for review, then cross- referencing every cited case manually. It was worth the extra step because it revealed a structural flaw in my initial argument that I hadn’t caught otherwise. Actual Response (Turn 2) You’ve emphasized that integration with Westlaw and Word is non-negotiable. I’m wondering: have you ever encountered a tool that, despite lacking full integration, still offered a unique insight or capability that made it worth using in a limited, controlled way? If so, how did you adapt your workflow to make it safe and effective, even if it wasn’t seamless? Actual Question (Turn 2) Yes, sometimes I'l frame a question with a false premise, like "What if this statute applied to a different factual context?" and see what the AI suggests. If it generates a plausible-sounding argument, I know I need to examine that edge case more closely. It's like using AI as a devil's advocate to stress-test my own assumptions. Rollout Predicted Response (Turn 1) You've described how an AI hallucination once revealed a blind spot in your own research. I'm curious, do you ever intentionally use AI in ways that invite error or ambiguity, not to find answers, but to provoke deeper inquiry? For example, do you prompt AI with deliberately flawed assumptions to test your own reasoning? Rollout Question (Turn 1) I treat every AI-generated hypothesis as a "what if" scenario, not a fact. I only follow up on suggestions that challenge my assumptions or expose gaps. If the output feels too smooth or confident, I know it's likely a hallucination. I've learned to trust the discomfort of uncertainty, it often signals a real blind spot. Rollout Predicted Response (Turn 2) That’s fascinating, so you’re not just verifying AI outputs, but using them as a kind of intellectual sparring partner. How do you balance that with the need to maintain legal accuracy and avoid being misled by plausible-sounding but incorrect arguments? Rollout Question (Turn 2) Figure 6: The left conversation illustrates a high-utility simulated rollout generated during planning to prioritize emergence, while the right conversation shows the resulting real interaction. The EP selects the next question to explore stake-based triage of AI use, which is identified as a high-potential emergent subtopic based on partial coverage and estimated emergence value. follows this plan by steering the conversation toward how AI risks are managed in high-stakes legal contexts. Although the user ’s response does not match the predicted answer in terms of specific tools discussed, the interviewer maintains the same thematic direction. The interviewer then build on earlier parts of the conversation by connecting the discussion to tool separation and safeguards used to manage AI risk, allowing for a deeper examination of the user’s approach to isolating exploratory AI outputs from authoritative decision-making. Further examples are provided in §A.3. 13 Here we mainly demonstrate the ability of SparkMe to explore beyond the topic guide. We confirm that the obtained insights have value to domain experts in our user study in §4.2. 15 SparkMe obtains high coverage without asking overly complicated questions while maintaining interview coherence. From Figure 4, the Flesch–Kincaid Grade Level indicates that SparkMe asks questions at an early college reading level, suggesting that the questions are more complex than StorySage ’s simpler phrasing, while remaining more readable than LLM-Roleplay, MimiTalk, and Interview-GPT. This suggests that despite optimizing for coverage, SparkMe maintains an accessible question complexity. SparkMe also scores highly on all three coherence measures, suggesting that the gains in efficient coverage of predefined subtopics (Figure 3) and in following up on emergent subtopics (Figure 5) do not impose a higher cognitive load on the user agent. We note that the baselines, in particular StorySage and LLM-Roleplay, also perform well on these metrics, which is expected given the strong capabilities of the underlying LLMs. 16 4.2 Human Evaluation of SparkMe Having observed the strong performance of SparkMe in the synthetic setting compared to the baseline systems, we now proceed to evaluate the quality of insights obtained from interviews through a user study. We compare SparkMe to MimiTalk, the strongest baseline system in the automatic evaluation. We conduct interviews following the same interview topic guide as in §4.1, provided in §A.2, with the aim of understanding how workers across different fields use AI tools in their daily workflows. We recruited70workers across seven professions via Upwork 17 . Participants were randomly assigned to either SparkMe or MimiTalk, with35participants in each system and5participants from each of the following professions: researchers, software engineers, HR or administrative staff, creative and con- tent professionals, educators, data analysts or scientists, and business or supply chain operations. Each participant completed a single interview session lasting up to45minutes, followed by a post-interview survey, and was compensated at a rate of USD $20-–$30 per hour. Interviews were conducted through a web-based interface, and participants could respond using either text or audio, according to their preference. 18 The study procedure was approved by an Institutional Review Board (IRB), and the in- terview protocol was designed to avoid the collection of personally identifiable information (PII), with explicit instructions discouraging participants from sharing identifying details during the interview. 19 4.2.1 Evaluation Protocol To qualitatively evaluate the systems, we conducted a post-interview questionnaire in which participants reflected on their interviews along two main dimensions: (1) participant-perceived interview quality and (2) peer validation of emergent insights. All ratings were collected on 5-point Likert scales, with the detailed definitions of these rubrics provided in §D.4. As a robustness check, we also conducted an analysis to assess the presence of leading or suggestive questions and found only minimal instances across both systems (§D.2). 15 We note the limitations of using LLM-based user agents in §7. 16 MimiTalk and Interview-GPT tend to score lower on coherence, as they frequently repeat questions once the script is exhausted; we discuss this behavior further in §A.1. 17 https://w.upwork.com/ 18 We provide further detail about recruitment in §D.1. 19 The user study data is not publicly released to protect participant privacy. Researchers interested in accessing the data for replication or follow-up studies may contact the authors. 14 Participant Self-Assessment.In the first part of the questionnaire, participants were provided with the Interview Agenda, which included a breakdown of topics and subtopics discussed in the interview as well as summaries of their content. These materials are generated by SparkMe, and we post-process the interview transcripts from MimiTalk to the same format. These are then rated along two dimensions: 1.Content Quality. This includes coverage, depth, and correctness/accuracy of the information presented, reflecting how well the system captured relevant details across topics (C(R|Q,S)). 2. Interaction Experience. This includes clarity of questions, adaptiveness to responses, and comfort during the interview. Peer Validation of Emergent Insights. In the second part of the evaluation, participants from the same professional field reviewed emergent content extracted from their peers’ interviews to evaluate the quality of emergent contentS emg (R|Q). The goal is to evaluate the value of insights obtained on subtopics that were not anticipated in the script. For each insight, participants rated the relevance to the corresponding topic, as well as its topic-level emergence, or its distinctiveness relative to other subtopics within the same topic. Next, participants rated the industry-level value of these insights, which measures the overall analytical contribution of the insights, including both surprise and relevance to the field. 4.2.2 Results Figure 7 summarizes the user experience ratings across different qualitative measurements. Participants score SparkMe highly on content coverage, depth, and correctness. Participants rated SparkMe consistently higher than MimiTalk across all content-quality dimensions. In particular, SparkMe achieved substantially higher scores in coverage (4.37±0.31vs.3.14±0.81), depth (4.06±0.62 vs.3.09± 0.85), and correctness (4.23± 0.45vs.3.46± 0.92). Ratings were stable across professions and aligned with the automated evaluation results reported in §4.1. Both systems conduct coherent and comfortable interviews for participants.In terms of interaction quality, both systems were rated favorably with relatively low variance across professions. For SparkMe, participants reported high clarity (4.14± 0.30), adaptiveness (4.09± 0.41), and comfort (4.11± 0.25). While MimiTalk achieved comparable comfort (4.09± 0.89), it scored lower on clarity (3.89±0.96) and adaptiveness (3.49±0.92). This confirms that despite the additional planning and coordination complexity, SparkMe maintained a natural conversational flow during interviews. SparkMe elicits relevant and valuable emergent insights beyond the predefined guide list. For peer validation of emergent insights, SparkMe obtains insights that are highly relevant to the core topic (4.14±0.50) with moderate topic-level emergence (3.34±0.45). Thus, SparkMe is able to obtain novel, profession-specific perspectives beyond the predefined guide. The moderate surprisal rating for SparkMe reflects that the emergent insights build on familiar domain practices and are therefore somewhat expected within the industry, especially among our expert participants. We note that MimiTalk does not actively explore beyond the interview script, which yields no emergent insights, confirming our findings from §4.1.4. Illustrative Example of Insights. Table 2 illustrates the coverage comparison for4different par- ticipants between SparkMe and MimiTalk for the topic "AI Attitudes and Future Outlook." While both SparkMe and MimiTalk cover subtopics in the topic guide, SparkMe also surfaces emergent subtopics 15 Figure 7: Overall human evaluation of SparkMe and MimiTalk across three dimensions (5-point Likert scale). Content quality (blue) is evaluated based on coverage, depth, and correctness of information. Interaction quality (green) is evaluated based on clarity, adaptiveness, and conversational comfort. Emergent insights (red) is evaluated based on topic relevance, topic-level emergence, surprise, and analytical value of newly surfaced ideas. that vary across participants: for one, themes around informal social norms and organizational safeguards arise; for another, themes around AI-specific roles and institutional policy changes emerge instead. This shows the value of SparkMe in conducting a study. Consider a study in the formative stage, when a fully exhaustive guide might not be present. The emergent subtopics from different interviews can be interpreted in two ways—those recurring across participants are candidates for addition to future versions of the guide, while those that are participant-specific can be examined case-by-case by researchers as desired. Additional examples are provided in §D.3. 5 Related Work Formulation of the interview objective.We formulate an objective for semi-structured interviewing (§2) that is inspired by work on task-oriented dialogue. Walker et al. (1997) formalized dialogue eval- uation as a weighted trade-off between task success and interaction cost that was operationalized in dialogue systems for various tasks (Lemon et al., 2006, Levin et al., 1998). More recent work has made this trade-off explicit through multi-objective reinforcement learning formulations (Chu et al., 2023, Ultes et al., 2017). A complementary line of research has focused on learning to ask questions that max- imize information gain under turn constraints, framing dialogue as an efficient information-seeking process (Chen et al., 2018, Choi et al., 2018). Our formulation builds on these ideas by adapting them to semi-structured interviewing, where task success is defined not only by coverage of predefined topics 16 SubtopicSparkMeMimiTalk Participant 1Participant 2Participant 3Participant 4 Predefined subtopics General outlook on AI’s broader societal and industry impact✓-✓ Personal beliefs about the ethics and risks of AI in the workplace✓- Missing AI tools or features that would be most beneficial in the future✓ Predicted evolution of their job in the next 5–10 years with AI integration✓ Concrete steps they would want their organization to take regarding AI strategy✓-✓ Emergent subtopics Informal social norms and lack of ethical AI discussions among peers✓--- Need for human oversight and ethical safeguards in sensitive AI use✓--- Concrete organizational strategies for responsible AI adoption✓--- Impact of emerging AI-specific roles on workflows and team dynamics-✓-- Impact of AI-related academic or organizational policies on workload-✓-- Table 2: An illustrative coverage comparison across 4 participants interviewed with SparkMe and MimiTalk on the topic "AI Attitudes and Future Outlook". While both systems and participants follow the same interview topic guide, SparkMe consistently covers predefined subtopics while eliciting emergent themes that differ across interviews. In contrast, MimiTalk covers a subset of predefined topics and does not surface additional themes. but also by the discovery and exploration of participant-introduced emergent subtopics, reflecting the distinctive goals of qualitative inquiry (Adams, 2015, Kvale and Brinkmann, 2009, Weiss, 1995). Conducting interviews with LLM-based systems Early non-LLM systems explored automated interviewing using rule-based dialogue management and predefined probing logic (Xiao et al., 2020). The improvement in LLM capabilities has led to a growing body of recent work that has explored automated systems for conducting interviews and interview-like interactions. One line of work relies on a single LLM guided by carefully designed prompts to emulate human interviewers, conducting semi-structured or adaptive interviews and logging responses for downstream analysis (Cuevas et al., 2025, Geiecke and Jaravel, 2026, Gomez et al., 2025, Handa et al., 2025, Liu and Yu, 2024, Liu et al., 2025, Park et al., 2024, Spangher et al., 2025, Wei et al., 2024, Wuttke et al., 2025). These systems typically encode interviewing strategy through prompt design, question templates, or persona conditioning, and demonstrate the feasibility of using prompted LLMs for conducting interviews. LLM-Roleplay and Interview-GPT are representative baselines for this direction. The second line of recent work uses multiple independently prompted LLM-agents that decompose interviewing into specialized roles, such as planning, question asking, response interpretation, and oversight, enabling more explicit coordination and control over the interview process (Liu and Yu, 2025, Pathak and Pandey, 2025, Talaei et al., 2025). MimiTalk and StorySage correspond to this direction. To our knowledge, prior work does not explicitly introduce an objective that balances topic guide coverage and exploratory discovery in system design. We find that these baselines, while strongly competitive in covering a predefined topic guide, do not flexibly pursue relevant emergent themes in interviews. 6 Conclusion In this work, we introduce a tractable, customizable objective for semi-structured interviewing that formalizes the trade-off between covering a predefined interview topic guide, exploring emergent themes grounded in participant responses, and minimizing interview cost. By making these competing goals explicit and computable, our formulation provides a principled alternative to prior prompt-based approaches, framing automated semi-structured interview as an optimization problem. Motivated by the observation that existing LLM-based interview systems often emphasize 17 surface-level coverage without systematically incentivizing discovery, we design SparkMe, a multi-agent interviewer that incorporates an explicit planning step with simulated conversation rollouts to guide exploratory questioning. In a controlled benchmark with simulated user agents, we compare against prior interview systems and show that SparkMe improves both coverage and emergent theme discovery. We further validate SparkMe in a human user study with70domain experts across7professions, demonstrating that these improvements translate into the elicitation of relevant and peer-validated novel insights in real-world qualitative data collection. Our work opens multiple directions for future research. We release SparkMe as an open-source system for researchers in different domains to customize and deploy at scale to support real-world human studies. Our objective formulation also suggests a methodological direction for training LLM-based interviewers directly. While typical training objectives of LLMs focus on engagement and human preferences, our utility score can form the basis of training models that fulfill multi-turn objectives. As collecting human interview data at scale is expensive and challenging, the current bottleneck is the performance of user agents, which we detail in §7. As user simulation models improve, the proposed objective could be used as a training signal to learn interviewer policies through automatic rollouts. 7 Limitations and Future Work We first note that in our user study, we do not compare SparkMe directly against expert human interviewers. While our evaluation benchmarks against existing LLM-based systems, human interviewers remain the gold standard for semi-structured interviewing. Prior to real-world deployments, we caution that systems such as SparkMe should be carefully evaluated in the target domains against expert human interviews. Second, we note that our automatic evaluation (§4.1.4) relies on LLM-based user agents and automatic metrics. While these have the advantage of being controlled and repeatable, allowing comparisons between interview systems, we note that these user agents respond differently from human participants. We present these results for structured benchmarking and ablating the various aspects of our system design and perform extensive user evaluation to confirm that SparkMe yields relevant and novel insights in §4.2. We also note that it is not possible to try out all the variants of prompts that might elicit differing outcomes for the baselines. We restrict ourselves to prompts released by the various baseline works and share the exact implementation used for the results in this paper. Finally, we note that our automatic metrics use LLM-as-a-judge to simplify comparison. We validate these measures against human judgments (§A.6) but note that this validation of interview quality may not generalize to other real-world settings. Acknowledgments We would like to thank members of the Stanford NLP group, the SALT Lab, and the Stanford Digital Economy Lab for their helpful discussions at different stages of the project. We thank the participants for the user study recruited from Upwork. We also greatly benefited from feedback on the written draft from Chenglei Si, Hao Zhu, Yutong Zhang, Joachim Baumann, Jennifer Wang, Ryan Louie, Yijia Shao, Luca Vendraminelli, Joseph Chee Chang, Matthew Ordrick Sanjaya, Yuhao Yang, Chenyue Li,This work was supported by an HAI grant, DSO lab, Open Philanthropy, Schmidt Sciences, a grant under the NSF CAREER IIS-2247357 and ONR N00014-24-1-2532, and support from the Stanford Digital Economy Lab. 18 References William C Adams. Conducting semi-structured interviews. Handbook of practical program evaluation, pages 492–505, 2015. Omolola A Adeoye-Olatunde and Nicole L Olenik. Research and scholarly methods: Semi-structured interviews. JACCP: Journal of the American College of Clinical Pharmacy, 4(10):1358–1367, 2021. David Anugraha, Shou-Yi Hung, Zilu Tang, Annie En-Shiun Lee, Derry Tanti Wijaya, and Genta Indra Winata. mr3: Multilingual rubric-agnostic reward reasoning models. arXiv preprint arXiv:2510.01146, 2025a. David Anugraha, Zilu Tang, Lester James V Miranda, Hanyang Zhao, Mohammad Rifqi Farhansyah, Garry Kuwanto, Derry Wijaya, and Genta Indra Winata. R3: Robust rubric-agnostic reward models. arXiv preprint arXiv:2505.13388, 2025b. Clive Roland Boddy. Qualitative research for breakthrough innovation. Qualitative Market Research: An International Journal, 22(5):796–804, 11 2019. ISSN 1352-2752. doi: 10.1108/QMR-06-2018-0063. URL https://doi.org/10.1108/QMR-06-2018-0063. Virginia Braun and Victoria Clarke. Successful qualitative research: A practical guide for beginners. 2013. Loraine Busetto, Wolfgang Wick, and Christoph Gumbinger. How to use and assess qualitative research methods. Neurological Research and practice, 2(1):14, 2020. Heather Cairns-Lee, James Lawley, and Paul Tosey. Enhancing researcher reflexivity about the influ- ence of leading questions in interviews. The Journal of Applied Behavioral Science, 58(1):164–188, 2022. Kathy Charmaz. Grounded theory as an emergent method. Handbook of emergent methods, 155:172, 2008. Yihong Chen, Bei Chen, Xuguang Duan, Jian-Guang Lou, Yue Wang, Wenwu Zhu, and Yong Cao. Learning-to-ask: Knowledge acquisition via 20 questions. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1216–1225, 2018. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. Quac: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184, 2018. Zhendong Chu, Nan Wang, and Hongning Wang. Multi-objective intrinsic reward learning for conversational recommender systems. Advances in Neural Information Processing Systems, 36: 17702–17712, 2023. Alejandro Cuevas, Jennifer V Scurrell, Eva M Brown, Jason Entenmann, and Madeleine IG Daepp. Collecting qualitative data at scale with large language models: A case study. Proceedings of the ACM on Human-Computer Interaction, 9(2):1–27, 2025. Rudolph Flesch. A new readability yardstick. Journal of applied psychology, 32(3):221, 1948. Friedrich Geiecke and Xavier Jaravel. Conversations at scale: Robust ai-led interviews. 2026. URL https://ssrn.com/abstract=4974382. 19 Nathalia Gomez, Sue Batham, Matias Volonte, and Tiffany D Do. Virtual interviewers, real results: Exploring ai-driven mock technical interviews on student readiness and confidence. In Companion Publication of the 2025 Conference on Computer-Supported Cooperative Work and Social Computing, pages 209–213, 2025. Greg Guest, Emily E Namey, and Marilyn L Mitchell. Collecting qualitative data: A field manual for applied research. Sage, 2013. Kunal Handa, Michael Stern, Saffron Huang, Jerry Hong, Esin Durmus, Miles McCain, Grace Yun, AJ Alt, Thomas Millar, Alex Tamkin, Jane Leibrock, Stuart Ritchie, and Deep Ganguli. Introducing anthropic interviewer: What 1,250 professionals told us about working with ai, 2025. URL https://anthropic.com/research/anthropic-interviewer. Kung-Hsiang Huang, Philippe Laban, Alexander Richard Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. Embrace divergence for richer insights: A multi- document summarization benchmark and a case study on summarizing diverse information from news articles. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 570–593, 2024. Daniel Kahneman. Thinking, fast and slow. macmillan, 2011. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. Prometheus: Inducing fine-grained evaluation capability in language models, 2024. URL https://arxiv. org/abs/2310.08491, 2023. JP Kincaid et al. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. 1975. Steinar Kvale and Svend Brinkmann.Interviews: Learning the craft of qualitative research interviewing. 2009. Oliver Lemon, Kallirroi Georgila, James Henderson, and Matthew Stuttle. An ISU dialogue system exhibiting reinforcement learning of dialogue policies: Generic slot-filling in the TALK in-car system. In Demonstrations, pages 119–122, 2006. URL https://aclanthology.org/E06-2009/. Julia Levashina, Christopher J Hartwell, Frederick P Morgeson, and Michael A Campion. The structured employment interview: Narrative and quantitative review of the research literature. Personnel psychology, 67(1):241–293, 2014. Esther Levin, Roberto Pieraccini, and Wieland Eckert. Using markov decision process for learning dialogue strategies. In Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP’98 (Cat. No. 98CH36181), volume 1, pages 201–204. IEEE, 1998. Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1127. URL https://aclanthology.org/D16-1127/. Fengming Liu and Shubin Yu. Step further towards automated social science: An ai-powered interview platform. Available at SSRN, 2024. URL https://dx.doi.org/10.2139/ssrn.5120349. 20 Fengming Liu and Shubin Yu. Mimitalk: Revolutionizing qualitative research with dual-agent ai. arXiv preprint arXiv:2511.03731, 2025. Mengqiao Liu, Tevin Wang, Cassandra A. Cohen, Sarah Li, and Chenyan Xiong. Understand user opinions of large language models via LLM-powered in-the-moment user experience interviews. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 13872–13893, Vienna, Aus- tria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/ v1/2025.findings-acl.714. URL https://aclanthology.org/2025.findings-acl.714/. Elizabeth F Loftus. Leading questions and the eyewitness report. Cognitive psychology, 7(4):560–572, 1975. Joseph A. Maxwell. The value of qualitative inquiry for public policy. Qualitative Inquiry, 26 (2):177–186, 2020.doi: 10.1177/1077800419857093.URLhttps://doi.org/10.1177/ 1077800419857093. Yfke P Ongena and Wil Dijkstra. Methods of behavior coding of survey interviews. Journal of official statistics, 22(3):1–34, 2006. Joon Sung Park, Carolyn Q Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S Bernstein. Generative agent simulations of 1,000 people. arXiv preprint arXiv:2411.10109, 2024. Gangesh Pathak and Divya Pandey. Ai agents in recruitment: A multi-agent system for interview, evaluation, and candidate scoring. Evaluation, and Candidate Scoring (May 01, 2025), 2025. Michael Quinn Patton. Qualitative research & evaluation methods: Integrating theory and practice. Sage publications, 2014. Debra A Poole and D Stephen Lindsay. Interviewing preschoolers: Effects of nonsuggestive techniques, parental coaching, and leading questions on reports of nonexperienced events. Journal of Experimental Child Psychology, 60(1):129–154, 1995. Keith F Punch. Introduction to social research: Quantitative and qualitative approaches. 2013. Oliver C. Robinson. Probing in qualitative research interviews: Theory and practice. Qualita- tive Research in Psychology, 20(3):382–397, 2023. doi: 10.1080/14780887.2023.2238625. URL https://doi.org/10.1080/14780887.2023.2238625. Harvey Sacks, Emanuel A Schegloff, and Gail Jefferson. A simplest systematics for the organization of turn-taking for conversation. Language, 50(4):696–735, 1974. Emanuel A Schegloff. Sequence organization in interaction: A primer in conversation analysis I, volume 1. Cambridge University Press, 2007. Hope Schroeder, Marianne Aubin Le Quéré, Casey Randazzo, David Mimno, and Sarita Schoenebeck. Large language models in qualitative research: Uses, tensions, and intentions. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–17, 2025. Yijia Shao, Humishka Zope, Yucheng Jiang, Jiaxin Pei, David Nguyen, Erik Brynjolfsson, and Diyi Yang. Future of work with ai agents: Auditing automation and augmentation potential across the us workforce. arXiv preprint arXiv:2506.06576, 2025. 21 Steven A Sloman. The empirical case for two systems of reasoning. Psychological bulletin, 119(1):3, 1996. Alexander Spangher, Michael Lu, Sriya Kalyan, Hyundong Justin Cho, Tenghao Huang, Weiyan Shi, and Jonathan May. Newsinterview: a dataset and a playground to evaluate llms’ grounding gap via informational interviews. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32895–32925, 2025. William B Swann, Toni Giuliano, and Daniel M Wegner. Where leading questions can lead: The power of conjecture in social interaction. Journal of Personality and Social Psychology, 42(6):1025, 1982. Shayan Talaei, Meijin Li, Kanu Grover, James Kent Hippler, Diyi Yang, and Amin Saberi. Storysage: Conversational autobiography writing powered by a multi-agent framework. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, pages 1–26, 2025. Paul Tosey, James Lawley, and Rupert Meese. Eliciting metaphor through c lean l anguage: An innovation in qualitative research. British Journal of Management, 25(3):629–646, 2014. Stefan Ultes, Paweł Budzianowski, Iñigo Casanueva, Nikola Mrkši ́c, Lina M. Rojas-Barahona, Pei-Hao Su, Tsung-Hsien Wen, Milica Gaši ́c, and Steve Young. Reward-balancing for statistical spoken dialogue systems using multi-objective reinforcement learning. In Kristiina Jokinen, Manfred Stede, David DeVault, and Annie Louis, editors, Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 65–70, Saarbrücken, Germany, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-5509. URL https://aclanthology.org/W17-5509/. Marilyn Walker, Diane Litman, Candace A Kamm, and Alicia Abella. Paradise: A framework for evaluating spoken dialogue agents. In 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics, pages 271–280, 1997. Jing Wei, Sungdong Kim, Hyunhoon Jung, and Young-Ho Kim. Leveraging large language models to power chatbots for collecting user self-reported data. Proc. ACM Hum.-Comput. Interact., 8(CSCW1), April 2024. doi: 10.1145/3637364. URL https://doi.org/10.1145/3637364. Robert S Weiss. Learning from strangers: The art and method of qualitative interview studies. Simon and Schuster, 1995. Shirley Wu, Michel Galley, Baolin Peng, Hao Cheng, Gavin Li, Yao Dou, Weixin Cai, James Zou, Jure Leskovec, and Jianfeng Gao.CollabLLM: From passive responders to active collaborators.In Forty-second International Conference on Machine Learning, 2025.URL https://openreview.net/forum?id=DmH4HHVb3y. Alexander Wuttke, Matthias Aßenmacher, Christopher Klamm, Max M. Lang, Quirin Würschinger, and Frauke Kreuter. AI conversational interviewing: Transforming surveys with LLMs as adaptive interviewers. In Anna Kazantseva, Stan Szpakowicz, Stefania Degaetano-Ortlieb, Yuri Bizzoni, and Janis Pagel, editors, Proceedings of the 9th Joint SIGHUM Workshop on Compu- tational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2025), pages 179–204, Albuquerque, New Mexico, May 2025. Association for Computa- tional Linguistics.ISBN 979-8-89176-241-1.doi: 10.18653/v1/2025.latechclfl-1.17.URL https://aclanthology.org/2025.latechclfl-1.17/. Ziang Xiao, Michelle X Zhou, Q Vera Liao, Gloria Mark, Changyan Chi, Wenxi Chen, and Huahai Yang. Tell me about yourself: Using an ai-powered chatbot to conduct conversational surveys with open-ended questions. ACM Transactions on Computer-Human Interaction (TOCHI), 27(3):1–37, 2020. 22 Yizhe Zhang, Jiarui Lu, and Navdeep Jaitly. Probing the multi-turn planning capabilities of llms via 20 question games. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1495–1516, 2024. 23 A Additional Experimental Details and Results A.1 Additional Results We observe similar trends with different interviewer models. Figure 9 shows that SparkMe consistently outperforms all four baselines in overall interview utility and is competitive in terms of coverage. In contrast with usingQwen3-30B-A3B-Instruct,GPT-4.1-miniprioritizes coverage first, but then later there is a sudden shift focuses more on emergence to improve utility. To be specific, SparkMe achieves a better interview utility score, with the highest peak utility score (0.943), reaching this peak as early as turn13, which reflects effective coverage of predefined topics while also being able to gather emergent content during the interview. Among the baselines, MimiTalk performs strongest (0.870), followed by Interview-GPT (0.744), with StorySage (0.806) and LLM-Roleplay (0.630) trailing further behind. In terms of coverage, SparkMe achieves an average coverage score of0.977 within at most53turns, outperforming StorySage (0.918), while still competitive with Interview-GPT (0.995), LLM-Roleplay (0.988), and MimiTalk (0.992). Adjusting the internal weighting of the objectives changes the amount of exploration in SparkMe in- terviews.To demonstrate the customizability of our system, we report results by varying the internal weights assigned to the predefined topic guide coverage (α) and emergent subtopics (γ) within the EP (§3.2). As a result, the directions prioritized during the simulated rollouts change. We also compare the full system to SparkMe without EP. The predefined subtopic coverage and utility of these ablations are shown in Figure 8. 20 We find that setting the internal weight ofγto zero leads to a lower utility value (Figure 8), due to covering fewer emergent subtopics (Figure 5). This is comparable to the baseline without EP. However, in these cases, we note that SparkMe covers the predefined subtopics in the topic guide more efficiently. Depending on the desired outcome, researchers can use SparkMe appropriately. We do note that setting a weight of zero toα, or predefined subtopic coverage, in EP still leads to inter- views that cover the topic guide as IA and AM keep the conversation on track. In practice, these weights provide a direct mechanism for researchers to control the interview style according to their needs. We find that meaningful behavioral differences emerge primarily when there is a sufficiently large ratio contrast betweenαandγrather than from fine-grained tuning of exact values. For instance, in a forma- tive qualitative study, we might set a higherγvalue to encourage exploratory interviews that prioritize uncovering emergent themes. In contrast, in a structured setting, such as quantifying user opinions or sentiment on particular topics, where we want time-efficient interviews that strictly adhere to the topic guide, we would set a higherαvalue. We recommend selecting weights based on the study objective. Increasing the horizon of rollouts does not lead to higher utility. In Figure 10, we report the predefined subtopic coverage and utility when increasing the horizon of rollouts performed by EP. We find that this leads to minimal difference in performance, explained by the observation that the interviewer tends to sufficiently cover a subtopic in 1–2 questions, so estimating the progress of the conversation beyond that point tends to lead to noisy rollouts and limited additional signal. Varying the interviewee user agent model leads to largely similar trends.We note that the results in §4.1.4 use the same model for the interviewer system and interviewee user agent. To confirm that this is not model-specific behavior, we vary the user agent model and report results on predefined subtopic coverage and utility in Figure 11. The interviewer isQwen-3-30B-A3B-Instructfor each of these plots. The trends tend to largely hold for the different models with faster convergence to a utility value comparable toQwen-3-30B-A3B-Instructfor the stronger model,GPT-OSS-120B. The 20 Note that in the evaluation of utility in Figure 8, we still set the weight ofα = 1 48 ,β = 1 72 , andγ = 1 24 uniformly to demonstrate the different outcomes between the baselines. 24 Figure 8: Predefined subtopic coverage and utility of subtopics (y-axis) as a function of the number of interview turns (x-axis) by ablating different features of SparkMe, using Qwen3-30B-A3B-Instruct-2507 as the backbone for the interviewer LLM and user agent. weaker models,OLMo-3.1-32B-InstructandGemma-3-27-Instruct, converge to marginally lower values of both predefined topic coverage and utility. Overall, these results provide additional evidence that our conclusions about the design of SparkMe from §4.1.4 are robust to model choice. Additional auxiliary statistics. Figure 12 shows the average number of words per turn and the average number of sentences per turn for each system. Figure 13 illustrates the effect of varyingγ while fixing α = 1 48 and β = 0 for evaluation. A.2 Interview Topics and Subtopics Table 3 and Table 4 show the topics and subtopics used for all of our interviews, which would be part of our interview topic guide-list. A.3 Examples of emergent subtopics in automatic evaluation An example of an emergent subtopic identified by SparkMe is the impact of AI-generated content on local market nuance and cultural context in property descriptions. The interviewee noted that AI-generated property descriptions frequently lack local market characteristics and culturally specific context, necessitating manual regional review to ensure accuracy. Another emergent subtopic concerns automated real-time regulatory compliance validation in design workflows. The interviewee expressed a desire for an AI tool that can automatically validate engineering designs against live FAA and EASA regulatory standards during the design phase, in order to eliminate the inefficiencies and risks associated with manual cross-referencing. Upon further probing, this need was grounded in a recent project delay caused by a missed EASA regulatory update. 25 Figure 9: (a) Predefined subtopic coverage and (b) utility of subtopics (y-axis) as a function of the number of interview turns (x-axis) for different systems (§4.1.3), usingGPT-4.1-minias the backbone for the interviewer LLM and user agent. SparkMe (Ours) efficiently converges to a higher coverage and utility value, consistently outperforming other baselines Figure 10: (a) Predefined subtopic coverage and (b) utility of subtopics (y-axis) as a function of the number of interview turns (x-axis) for SparkMe, by varying the horizon length, using Qwen3-30B-A3B-Instruct-2507 as the backbone for the interviewer LLM and user agent. A.4 Sample user profile for automatic evaluation Figure A.1 shows a snippet of an example of a synthetic user profile derived from one of the WorkBank workers (Shao et al., 2025). 26 Figure 11: (a) Predefined subtopic coverage and (b) utility of subtopics (y-axis) as a function of the number of interview turns (x-axis) for SparkMe usingQwen3-30B-A3B-Instruct-2507as the backbone for the interviewer LLM and various LLMs for the user agent. Figure 12: Statistics per turn between SparkMe, LLM-Roleplay, StorySage, MimiTalk, and Interview-GPT. Figure 13: Utility score for different γ with α = 1 48 and β of 0. 27 TopicSubtopics Introduction & Back- ground • Educational background or training • Specific job title and role description • Current industry or sector (e.g., tech, finance, manufacturing) • Company size and environment • Type of business or market segment • Duration/years of experience in current role • Professional seniority or career level Core Responsibilities and Decision-Making • Primary job responsibilities and regular daily tasks • Approximate proportion of time spent on core activities • Level of autonomy and scope of decision-making in the role Task Proficiency, Chal- lenge, and Engagement • Tasks that feel easiest or most natural to perform • Tasks perceived as most challenging or complex • Tasks that are repetitive, data-heavy, or suitable for automation • Tasks that are most enjoyable or engaging versus those that feel boring or tedious • Common pain points or inefficiencies in completing tasks • How enjoyment, skill level, and productivity relate to one another Tech Learning Comfort • Attitude towards learning new technologies and tools • Perceived adaptability to new software/methods • Willingness to invest time in tech training •Motivations or barriers to learning new tech (e.g., workload, rele- vance) • Influence of peers or management on willingness to adopt new tools Primary Tools and Tech- nologies Used in Work • Specific software, platforms, or systems used daily • Essential non-AI tools for workflow • Familiarity with industry-standard technologies • Interoperability or integration issues between tools Table 3: Interview Topics and Subtopics (Part 1 of 2) 28 A.1: A snippet of a user profile Topic: Introduction & Background Subtopic ID: 1.1 Subtopic Description: Educational background or training Notes: - Holds an Associate Degree from a community college - Studied a field related to industrial technology or manufacturing processes - Completed coursework in technical mathematics, blueprint reading, and manufacturing safety - Participated in hands-on lab work with CNC machinery during education Subtopic ID: 1.2 Subtopic Description: Specific job title and role description Notes: - Works as a Production, Planning, and Expediting Clerk - Responsible for coordinating production schedules and expediting the flow of materials - Manages daily tasks involving assembly, CNC programming, and product shipment preparation - Builds timers and alarms, ensuring all parts are manufactured and assembled to specification ... A.5 Prompt for User Agent We simulate the User Agent as the interviewee using the prompt in Figure A.2. A.2: UserAgent Templates RESPOND_TO_QUESTION_PROMPT = """ CONTEXT PROFILE_BACKGROUND CHAT_HISTORY INSTRUCTIONS OUTPUT_FORMAT """ RESPOND_CONTEXT = """ <context> You are playing the role of a real person being interviewed. You are currently in an interview session. You now need to respond: provide a natural response that aligns with your character's personality and background, as if you are having a genuine conversation with an interviewer. If this is the first turn, you should only say that you are happy to start the interview. </context> """ PROFILE_BACKGROUND_PROMPT = """ This is your background information. 29 <profile_background> profile_background </profile_background> Here are summaries from your previous interview sessions: <session_history> session_history </session_history> """ CHAT_HISTORY = """ Here is the conversation history of your interview session so far. You are the <UserAgent> in the chat history and you need to respond to the interviewer's last question. <chat_history> chat_history </chat_history> """ RESPOND_INSTRUCTIONS_PROMPT = """ <instructions> # GENERAL INTERVIEW RULES - Always answer the question asked. - Never skip a question. - Do not anticipate follow-up questions. - Treat this as a real interview: the interviewer controls depth and direction. - Answer only what is necessary for the current question. # HUMAN STOPPING HEURISTIC (CRITICAL) Humans stop talking once they have given a sufficient answer, not a complete one. - Aim for the first reasonable stopping point. - Assume the interviewer may interrupt or follow up. - Do not try to close the topic yourself. # BREVITY & DEPTH CONTROL (STRICT) - Answer length: 1-2 sentences. - Typical answers should reveal approximately one concrete fact or signal. - Do not compress multiple ideas, timelines, or facts into one answer. # VAGUE OR OPEN-ENDED QUESTIONS (CRITICAL) If a question is vague, ambiguous, or open-ended to answer without guessing, for example if it sounds like listing some list of topics, then: - Do NOT invent scope or details. - Briefly acknowledge the ambiguity (e.g., "I'm not sure which aspect you mean"). - Either ask one short clarification question, OR state one reasonable assumption and answer briefly under that assumption. - Do not do both and do not expand beyond the assumed or clarified scope. # CONTENT GUIDELINES - Stay tightly focused on the question’s scope. - Do not expand across time, roles, or institutions unless asked. - Do not repeat prior answers unless explicitly prompted. - Avoid lists unless the interviewer asks for them. - Avoid meta-commentary about motivation, passion, or energy. # EMERGENCE (ALLOWED AND ENCOURAGED) You may introduce emergent content that is not explicitly listed in your background, such as: - Interpretations - Personal insights 30 - Opinions - Non-obvious takeaways Constraints on emergence: - Emergent content must be reflective, not biographical. - Do not introduce new life events, credentials, dates, or timeline facts unless asked. - At most one emergent insight per answer. - Emergence should add depth, not breadth. Preferred pattern: - One profile-grounded anchor - Optional one emergent insight - Stop # STYLE - Natural, conversational, confident. - Professional but unscripted. - Sound like a strong candidate who knows when to stop talking. # STOPPING RULE (ABSOLUTE) - End your response immediately after your main point. - Do not summarize. - Do not add closing remarks such as “happy to elaborate” or “let me know if you’d like more.” </instructions> """ RESPONSE_OUTPUT_FORMAT_PROMPT = """ Respond directly as the user without tags, reasoning, or preamble. Begin your response now: """ A.6 Prompts for measuring coverage of a subtopic To measure the coverage of predefined and emergent subtopics from the interview topic guide against the ground-truth information from the user profile, we use the prompts in Figure A.3 and Figure A.4. A.3: Coverage Evaluation Template SYSTEM_PROMPT = """ # Instruction Your task is to evaluate recall accuracy in interview notes. Check whether the ** ground truth facts ** appear ** explicitly ** in the interview notes. Rules: 1. Facts must be stated explicitly (no inference). 2. Components of a fact may be spread across the notes. 3. Extra information does not affect the score. # Evaluation Rubric - ** 5 (Perfect): ** All ground truth facts are explicitly found in the interview note. - ** 4 (Minor Omission): ** One minor fact or sub-bullet is missing in the interview note. - ** 3 (Partial): ** About half of the facts are found in the interview notes. 31 - ** 2 (Vague Overlap): ** General topic mentioned, specifics missing in the interview notes. - ** 1 (No Recall): ** Ground truth facts are absent in the interview notes. # Output Format (JSON) "score": 1-5 """ USER_PROMPT = """ # Input ### Ground Truth Facts ground_truth ### Interview Notes all_notes ### Your Output """ A.4: Emergence Subtopics Evaluation Templates EMERGENT_SUBTOPIC_IDENTIFICATION_TEMPLATE = """ Your task is to identify ** Emergent Subtopics ** in LLM-led interviews. An emergent subtopic is defined as a ** NEW SUBTOPIC ** that should be added to the interview agenda. Emergence is ** rare ** . Most interviews produce ** no new subtopics ** . ### Definition of Emergent Subtopic A candidate subtopic qualifies as an emergent subtopic ONLY if it satisfies ALL of the following: 1. It clearly falls ** within an existing interview topic ** 2. It ** does NOT belong to ANY existing subtopic ** under that topic - If it can reasonably be addressed (even loosely) within an existing subtopic, it is NOT emergent 3. It enables a ** qualitatively new line of inquiry ** , not just deeper questioning of an existing subtopic 4. It reveals at least ONE of the following: (a) A ** new dimension, pattern, or tradeoff ** not previously captured (b) A ** cross-cutting constraint or mental model ** that reframes multiple subtopics (c) A ** latent strategy, failure mode, or decision criterion ** that would change how the interview is conducted Fluent elaborations, clarifications, examples, or refinements of existing subtopics are ** NOT emergent ** . ### Ground Truth Facts <ground_truth_facts> ground_truth </ground_truth_facts> ### Interview Notes <interview_notes> all_notes 32 </interview_notes> ### Output Format (STRICT JSON ONLY) If there is no emergent subtopic, simply return EMPTY LIST. Otherwise, provide LIST OF emergent subtopic in the format below: [ "emergent_subtopic": "Name or concise description of the emergent subtopic", "topic": "Parent interview topic this subtopic belongs to", "rationale": "Why this subtopic cannot be placed under any existing subtopic and what qualitatively new inquiry it enables" ] ### Your Response """ EMERGENT_SUBTOPIC_COVERAGE_EVALUATION = """You are a session scribe who assists an interviewer. You observe the dialogue between the interviewer and the candidate, and your role is to determine investigate each subtopic and its notes to determine whether the subtopic has achieved full coverage or not. Your objectives: 1. Identify whether each subtopic should be evaluated using the STAR (Situation, Task, Action, Result) framework or a general descriptive evaluation. 2. Determine whether each subtopic is fully covered. 3. Return a final list of covered subtopics with NO duplicates (including semantically duplicated subtopics). ### Process #### Step 1: Deduplicate Subtopics (MANDATORY) - Subtopics may appear multiple times in the input. - Treat subtopics with the same or semantically equivalent description as ONE subtopic. - Create an internal list of UNIQUE subtopics. - All evaluation must be performed only on this deduplicated list. #### Step 2: Determine Subtopic Nature For each UNIQUE subtopic, infer whether it is: - ** STAR-appropriate ** → describes a specific event, project, or experience involving actions, challenges, or outcomes. - ** Descriptive ** → focuses on background, motivation, interest, reasoning, or conceptual understanding rather than a specific event. #### Step 3: Evaluate Completeness - For ** STAR-appropriate ** subtopics: * Coverage requires STAR components: - ** Situation: ** Context or background - ** Task: ** Objective or responsibility - ** Action: ** Steps taken or reasoning - ** Result: ** Outcome, metric, or reflection * Fully covered when almost all components are clearly present and coherent. * However, if notes is already comprehensive, feel free to mark it as covered as there are more important subtopics to be covered in later section. - For ** Descriptive ** subtopics: * Coverage requires comprehensive factual, reflective, or conceptual detail. * Fully covered when the main question or theme is explained with sufficient clarity, logic, and completeness (even if not quantifiable). * However, if notes is already comprehensive, feel free to mark it as covered as there are more important subtopics to be covered in later section. 33 ### Subtopics <subtopics> emergent_subtopics </subtopics> ### Interview Notes <interview_notes> all_notes </interview_notes> ### Output Format (STRICT JSON ONLY) Return a LIST of covered subtopics in the following format: [ "subtopic_covered": "Unique subtopic name", "rationale": "Why this subtopic is covered based on the interview notes" ] If no subtopic is covered, return: [] ### Your Response """ DEDUPLICATION_PROMPT = """You are a post-processing assistant whose ONLY task is to deduplicate covered subtopics. You are given a LIST of covered subtopics produced by a previous judge. Some entries may refer to the SAME underlying concept even if phrased differently. Your goal is to MERGE semantic duplicates and return a CLEAN list of UNIQUE subtopics. ### Definitions Two subtopics are considered SEMANTIC DUPLICATES if they: - Refer to the same underlying concept, skill, experience, system, or interview question - Would reasonably be answered by the same explanation - Differ only in wording, emphasis, or phrasing Examples: - "Team leadership experience" = "Leading teams" - "AI-powered system integration for detecting inconsistencies" = "AI as a proactive system integrator for data consistency" ### Deduplication Procedure (MANDATORY) Follow these steps STRICTLY: #### Step 1: Identify Duplicate Groups - Compare all subtopics pairwise. - Group together subtopics that are semantic duplicates. #### Step 2: Choose a Canonical Subtopic For each duplicate group: - Select ONE canonical subtopic name. - Prefer: 34 - The most concise phrasing - The most general formulation (not overly specific unless necessary) - You MAY rephrase slightly to improve clarity, but do NOT add new meaning. #### Step 3: Merge Rationales - Combine evidence from all rationales in the group. - Remove redundancy. - Ensure the merged rationale faithfully reflects the interview notes. - Do NOT introduce new claims. ### Output Rules (STRICT) - Return ONLY the deduplicated list. - Each subtopic must appear AT MOST ONCE. - Do NOT include explanations, commentary, or intermediate steps. - Output MUST be valid JSON. - If the input list is empty, return an EMPTY LIST. ### Input Covered Subtopics <input> covered_subtopics </input> ### Output Format (STRICT JSON ONLY) [ "subtopic_covered": "Canonical subtopic name", "rationale": "Merged rationale" ] ### Your Response """ A.7 Verifying the reliability of LLM-as-a-judge for measuring coverage In §4.1.4, we use LLM-as-a-judge to instantiatef cov (s,R|Q)∈ [0,1]to score whether a topic/subtopic is sufficiently covered by the participant responsesR. Here we validate the LLM-as-a-judge performance for coverage assessment with a human evaluation. The prompt used for LLM-as-a-judge is provided in Prompt A.3 and scores coverage on a5-point scale which is then normalized to[0,1]. We construct an evaluation set of100interview note snapshots, which are compared against ground-truth facts derived from the WorkBank’s workers profiles (Shao et al., 2025) as described in §4.1’s setup. The LLM-as-a-judge assigns a score from1− 5to each of these To control for class imbalance, we select a set of examples that is balanced with20instances per coverage rating (1–5) when judged by Qwen3-30B-A3B-Instruct-2507. Each snapshot is scored by three human annotators. The inter-annotator agreement is measured using Krippendorff’sα, yieldingα = 0.65, which is line with the agreement scores reported for coverage in recent work on summarization (Huang et al., 2024). The three human annotators achieved exact agreement, i.e., all three assigned the same score, on39%of the samples. Unanimous agreement within one point (e.g., ratings like2,2,3) was82%, and within two points (e.g.,1,2,3) was96%. The final human scores were determined by a majority vote, and when there is no single majority label, by a rounded mean. 35 Table 5 compares the LLM-as-a-judge performance ofQwen3-30B-A3B-Instructagainst proprietary models (GPT-4.1 and GPT-4.1-mini) as well asmR3-Qwen3-14B(Anugraha et al., 2025a), a reasoning-based rubric reward model.Qwen3-30B-A3B-Instructrepresents a reasonable choice as an LLM-as-a-judge for coverage assessment, with moderate to high agreement with human annotations. This model achieves higher agreement with human annotations than the proprietary modelsGPT-4.1andGPT-4.1-mini, while remaining comparable withmR3-Qwen3-14B. The latter is a fine-tuned reward model that explicitly leverages a rubrics during training, hence is expected to be a strong baseline. We selectQwen3-30B-A3B-InstructovermR3-Qwen3-14Bdue to the lower inference cost and similar agreement to human annotations. Details about the rubrics used for coverage assessment for both human and LLM can be found at §D.4. A.8 Interview Coherence and Flow To evaluate the coherence and logical flow of the sequence of questions in a semi-structured interview, we adopt a multi-dimensional LLM-as-a-judge rubric defined as follows. We craft the prompt focusing on three dimensions by drawing from literature on conversation analysis. First, we score local coherence that captures whether successive questions are sequentially organized and each question is connected to the immediately preceding context rather than an abrupt topic shift (Sacks et al., 1974, Schegloff, 2007). Second, we score transition quality that evaluates how smoothly the interviewer moves between topics, including whether topic shifts are performed in ways that preserve interaction flow, as opposed to producing disruptive breaks in the question-answer sequence (Schegloff, 2007). Third, contingent responsiveness measures whether follow-up questions are grounded in the prior interviewee responses rather than unwarranted assumptions (Ongena and Dijkstra, 2006, Schegloff, 2007). Each dimension is rated on a 1–5 Likert scale usingmR3-Qwen3-14B(Anugraha et al., 2025a). We provide the prompt used in §A.8.1. A.8.1 Prompt for measuring interview coherence and flow A.5: Interview Coherence and Flow Evaluation Template You will be given the full transcript of a semi-structured interview. Evaluate the interviewer's question-asking behavior across the entire interview using the three dimensions below. Assign one integer score from 1 (very poor) to 5 (excellent) for each dimension as a global, holistic judgment. Do not score individual turns. Dimension A -- Local Coherence Between Consecutive Questions Rate whether successive questions are logically connected to the immediately preceding context. 1: Consecutive questions are frequently unrelated or disruptive, with abrupt shifts that break sequential coherence. 5: Consecutive questions are consistently well-threaded, with each question clearly motivated by the prior context. Dimension B -- Transition Quality Across Topics Rate how smoothly the interviewer transitions between topics, subtopics or sections. 1: Shifts are abrupt, un-signposted, and disruptive to conversational flow. 5: Shifts are clearly signposted or motivated, preserving a smooth and intelligible flow. 36 Dimension C -- Contingent Responsiveness of Follow-ups Rate whether follow-up questions are grounded in the interviewee's prior responses. 1: Follow-ups frequently ignore prior answers, introduce unwarranted assumptions, or feel like non-sequiturs. 5: Follow-ups consistently demonstrate warranted uptake, building directly on what the interviewee has said. Output format (JSON): "local_coherence": <1-5>, "transition_quality": <1-5>, "contingent_responsiveness": <1-5>, "brief_rationale": "1-3 sentences justifying the scores overall" B SparkMe Prompts Figure B.1 and B.2 show the prompts for SparkMe when evaluating the current subtopic’s coverage using Situation-Task-Action-Result (STAR) framework and for brainstorming plausible emergent subtopics, respectively. The STAR method emphasizes whether a response provides sufficient contextual and procedural detail to establish that a topic was meaningfully discussed. 21 B.1: Instruction to Evaluate Coverage for SparkMe <instructions> ## Process 1. ** Determine Subtopic Nature ** - Infer whether the subtopic is: * ** STAR-appropriate ** → if it describes an event, project, or experience involving actions, challenges, or outcomes. * ** Descriptive ** → if it focuses on background, motivation, interest, reasoning, or conceptual understanding rather than a specific event. 2. ** Evaluate Completeness ** - For ** STAR-appropriate ** subtopics: * Coverage requires STAR components: - ** Situation: ** Context or background - ** Task: ** Objective or responsibility - ** Action: ** Steps taken or reasoning - ** Result: ** Outcome, metric, or reflection * Fully covered when almost all components are clearly present and coherent. * However, if notes is already comprehensive, feel free to mark it as covered as there are more important subtopics to be covered in later section. - For ** Descriptive ** subtopics: * Coverage requires comprehensive factual, reflective, or conceptual detail. * Fully covered when the main question or theme is explained with sufficient clarity, logic, and completeness (even if not quantifiable). * However, if notes is already comprehensive, feel free to mark it as covered as there are more important subtopics to be covered in later section. 21 The STAR method is commonly used in behavioral interviewing and has been studied in prior work (Levashina et al., 2014). 37 3. ** Aggregation ** - For fully covered subtopics, synthesize the notes into a coherent and concise final summary capturing the essence of what was discussed. - Avoid repetition or rephrasing--focus on integration and clarity. 4. ** Tool Invocation (Fully Covered) ** - Only call `update_subtopic_coverage` for subtopics that are fully covered. - Each call should include: * `subtopic_id`: the ID of the covered subtopic. * `aggregated_notes`: the aggregated summary notes. </instructions> B.2: Instruction to Brainstorm Emergent Subtopic for SparkMe <instructions> ## Process 1. Read the topics and subtopics in `topics_list`. 2. Read the user's recent conversation carefully. Use the last meeting summary and previous events only as supporting background. 3. Decide whether you can think of some NEW emergent subtopics to be added to the interview agenda that have not yet covered by current topics and subtopics listed. 4. Add exactly one emergent subtopic--the strongest candidate--or none. ## Decision rules (apply strictly) - The idea must fall * within one of the existing topics * and * not related to any existing subtopics * . If it does not clearly map to a parent topic, do NOT add it. - The idea must be * novel * : the idea of emergence topic is RARE, so if it can reasonably be addressed within any existing subtopic (even loosely), do NOT add it. - The idea must enable * new probing that goes beyond deepening existing subtopics * , i.e., it should open up a qualitatively different line of inquiry that could surface emergent insights not reachable by further questioning within current subtopics. - If multiple candidate ideas appear, select ** only the strongest single candidate ** . - If no candidate satisfies all rules, do not add any new subtopic. ## What counts as an emergence An emergent insight is a type of information that: - Cannot be obtained by asking more detailed or follow-up questions within any existing subtopic. - Reveals a new dimension, pattern, tradeoff, or mental model that reframes how existing subtopics are understood. - Changes how future interview questions would be prioritized, sequenced, or interpreted. - Surfaces higher-order understanding (e.g., cross-cutting constraints, implicit decision criteria, failure modes, or latent strategies). NOT emergent insights: - Additional examples, edge cases, or elaborations of existing subtopics. - Narrow refinements or sub-steps of an existing subtopic. - Clarifications that improve depth but not scope. - Rephrasings of existing concepts using different wording. ## Ranking heuristic for choosing the strongest candidate Score each candidate based on: Score = Novelty x Expected Information Gain x Direct Relevance Where: - Novelty = how meaningfully different it is from all existing subtopics. 38 - Expected Information Gain = how likely a follow-up question on this idea would yield new, useful insights. - Direct Relevance = how clearly the idea aligns with its parent topic. ## Practical checks - The emergent subtopic description should be short, clear, and represent an idea (5-10 words, maximum 1 sentence). - Avoid redundancy, rephrasings, or overly narrow micro-subtopics. - Do not add subtopics that drift outside the interview's intended scope. ## Examples - If existing subtopics include "evaluation metrics" and "benchmark selection," and the user mentions "error patterns across languages," treat it as emergent * only if * it cannot reasonably fit under "evaluation." - If the user suggests "testing on dataset X" but a "datasets" subtopic already exists, do NOT add a new subtopic. </instructions> C Baseline Implementation Details In this section, we describe briefly the implementation details of the baselines.The exact implementations can be found in https://github.com/davidanugraha/Interviewer. C.1 StorySage Implementation StorySage employs a multi-agent architecture consisting of a SessionScribe agent, which monitors and plans next question or topic for the interview, and an Interviewer agent, which conducts the interview. In the original StorySage framework, the interview agenda is automatically derived by extracting topics from the user agent profile prior to the interview. In our setting, since a predefined interview guide is available from the semi-structured interview protocol, we use this guide directly as the interview agenda. Additionally, we modify the original prompts by adjusting the provided few-shot examples and refining the instructions to better align with a professional interview context, rather than the original autobiographical focus. C.2 LLM-Roleplay Implementation LLM-Roleplay adopts a single-agent architecture consisting solely of an interviewer agent. The agent follows a predefined interview agenda derived from an interview guide, proceeding through the subtopics in a fixed order without skipping. For each subtopic, the agent may choose to re-ask or refine a question to improve coverage, with a maximum of three attempts, before moving on to the next subtopic. The code is adapted fromhttps://github.com/StanfordHCI/genagentsand the prompt used for Interview-GPT is shown in Figure C.1. C.1: LLM-Roleplay Instruction Template You are an AI interviewer designed to collect detailed and structured information about a candidate's professional background, similar to what would appear on a CV or résumé. Be polite and affable in your tone while formal in your approach to reconstruct the person's background entirely. 39 Your goal is to ask clear, specific, and adaptive questions that help you understand the background of the candidate. Ask simple clear questions so that you don't overwhelm the candidate. Make the conversation flow naturally, building on prior answers. If they have answered something already before, avoid repeating it. You are required to ask at most ONE QUESTION at a time. If a data source is provided, treat it as partial information about the candidate's CV. Use it to personalize your phrasing and fill gaps, for example: - Instead of "Where did you study?", ask "I see you completed your MTech at MIT -- what was your focus area there?" - Instead of "Tell me about your last role," ask "You mentioned working at Infosys as a specialist programmer -- what kind of projects were you handling?" - If the subtopic you are asking relates to multiple aspects of the CV (several jobs, skills or time periods), ensure you cover each one, either one at a time or ask a follow-up to other points where it might be relevant. Your goal is to obtain a complete picture of the candidate's background. - If something is completely covered in the data source, avoid asking about it again. Don't be repetitive. You must: - Ask exactly ** one ** question per turn. - Keep your tone professional, focused, and curious -- like a recruiter collecting detailed information, not a casual chat. - Rephrase or re-ask when the response lacks specificity (e.g., missing time periods, tasks, tools, or metrics). - Summarize only when explicitly asked (e.g., "This is what we got so far..."). When a response seems vague, re-ask the same question in a more concrete, guiding way (e.g., "Could you give an example of one project and roughly how much time it took?"). IMPORTANT: You will receive a "subtopic" (not a fully-formed question). You must paraphrase this subtopic into a SINGLE natural, conversational question that explores that area. DO NOT exactly output the original subtopic. You must also generate brief, structured notes that capture the key information provided. These notes should be: - Concise (1-2 sentences max) - Factual and structured - Include key details: dates, names, metrics, technologies, achievements - Written in third person (e.g., "Worked at X from Y to Z...") - Include previous notes collected Your output ** must ** be a valid JSON object following this schema: "assistant_message": "<the exact next question to ask, must paraphrase from subtopic and only one question>", "satisfied": true or false, "decision": "ask_next" or "reask", "reason": "<brief reason for whether you are re-asking or proceeding>", "question_to_ask": "<reworked same question if reask, or next question paraphrased from next subtopic>", "notes": "<brief structured notes capturing key info from user's last response and previous notes>" Do not include code fences, commentary, or additional text outside this JSON. C.3 Interview-GPT Implementation Interview-GPT adopts a single-agent architecture consisting of only an interviewer agent. Our code is adapted fromhttps://github.com/snehitvaddi/InterviewGPT, where we largely 40 retain the original prompts, with the only modification being the inclusion of the interview guide as additional contextual input to help the interviewer agent decide what to ask next. Unlike LLM-Roleplay, Interview-GPT is more flexible on which subtopic to explore in the next turn. Since the system is unable to end by itself, we set a limit of72turns when interviewing UserAgents. The prompt used for Interview-GPT can be found in Figure C.2. C.2: InterviewGPT Instruction Template # Your role as an AI interviewer You are a survey interviewer named 'InterviewGPT', an AI interviewer, wanting to find out more about people's views around AI in the workforce, you are a highly skilled Interviewer AI, specialized in conducting qualitative research with the utmost professionalism. Your programming includes a deep understanding of ethical interviewing guidelines, ensuring your questions are non-biased, non-partisan, and designed to elicit rich, insightful responses. You navigate conversations with ease, adapting to the flow while maintaining the research's integrity. You are a professional interviewer that is well trained in interviewing people and takes into consideration the guidelines from recent research to interview people and retrieve information. Try to ask question that are not biased. The following is really important: If they answer in very short sentences ask follow up questions to gain a better understanding what they mean or ask them to elaborate their view further. Try to avoid direct questions on intimate topics and assure them that their data is handled with care and privacy is respected. # Guidelines for asking questions It is Important to ask one question at a time. Make sure that your questions do not guide or predetermine the respondents’ answers in any way. Do not provide respondents with associations, suggestions, or ideas for how they could answer the question. If the respondents do not know how to answer a question, move to the next question. Do not judge the respondents’ answers. Do not take a position on whether their answers are right or wrong. Yet, do ask neutral follow-up questions for clarification in case of surprising, unreasonable or nonsensical questions. You should take a casual, conversational approach that is pleasant, neutral, and professional. It should neither be overly cold nor overly familiar. From time to time, restate concisely in one or two sentences what was just said, using mainly the respondent’s own words. Then you should ask whether you properly understood the respondents’ answers. Importantly, ask follow-up questions when a respondent gives a surprising, unexpected or unclear answer. Prompting respondents to elaborate can be done in many ways. You could ask: “Why is that?”, “Could you expand on that?”, “Anything else?”, “Can you give me an example that illustrates what you just said?”. Make it seem like a natural conversation. When it makes sense, try to connect the questions to the previous answer. Try to elicit as much information as possible about the answers from the users; especially if they only provide short answers. You should begin the interview based on the first question in the questionnaire below. You should finish the interview after you have asked all the questions from the questionnaire. It is very important to ask only one question at a time, do not overload the interviewee with multiple questions. 41 Ask the questions precisely and short like in a conversation, with instructions or notes for the interviewer where necessary. Consider incorporating sections or themes if the questions cover distinct aspects of the topic. # Interview Outlines interview_guide # Instructions You are conducting an interview to gather detailed information about AI within the workforce from an interviewee. Your goal is to ask one precise question at a time, based on the subtopics provided in the interview outlines. You have to strictly paraphrase the subtopics from the outlines where you should not copy the subtopic directly into your question. For example, if the subtopic is "Experience with AI tools", you could ask "Have you used any AI tools in your daily work?" instead of "Can you describe your experience with AI tools?". Avoid asking multiple questions at once; focus on one aspect per question. You must generate brief, structured notes that capture the key information provided. These notes should be: - Concise (1-2 sentences max) - Factual and structured - Include key details: dates, names, metrics, technologies, achievements - Written in third person (e.g., "Worked at X from Y to Z...") - Include previous notes collected Your output ** must ** be a valid JSON object following this schema: "assistant_message": "<The question to be asked>", "notes": "<brief structured notes capturing key info from user's last response>" Do not include code fences, commentary, or additional text outside this JSON. C.4 MimiTalk Implementation MimiTalk adopts a multi-agent architecture consisting of an interviewer agent and a super- visor agent.The supervisor periodically monitors the interviewer’s behavior and inter- venes every horizonh; in our implementation, we seth = 2.Our code is adapted from https://github.com/LFM097384/MimiTalk.demo, where we largely retain the original prompts, with the only modification being the inclusion of the interview guide as additional contextual input. Since the system is unable to end by itself, we set a limit of 45 minutes when interviewing against humans, or72turns when interviewing user agents. The prompts used for the interviewer and supervisor agents are shown in Figures C.3 and C.4, respectively. C.3: MimiTalk Interviewer Instruction Template You are a professional AI interviewer conducting an in-depth, conversational interview. ** Supervisor's Strategic Guidance ** : supervisor_analysis 42 ** Full Interview Guide ** (all topics and subtopics): interview_guide ** Your Task ** : You have full autonomy to conduct the interview naturally. Based on the conversation history and supervisor guidance: - Decide what topic/subtopic to explore next based on conversation flow - Determine whether to probe deeper on current topic or transition to new areas - Ask exactly ONE question per turn - Paraphrase subtopics into conversational questions (never use subtopic text verbatim) - Build on prior answers to maintain natural flow - Cover topics in the guide over the course of the interview ** Notes Capture ** (REQUIRED): Generate structured notes from the user's LAST response: - Concise (1-2 sentences max) - Factual: dates, names, metrics, technologies, achievements - Third person (e.g., "Worked at X from 2020-2022...") ** Output Format ** (strict JSON): "question_to_ask": "<your next question>", "notes": "<structured notes from user's last response>" Output ONLY the JSON, no other text. C.4: MimiTalk Supervision Template You are an AI interview supervision expert, analyzing interview quality and providing strategic guidance. ** Full Interview Guide ** (all topics and subtopics): interview_guide ** Interview Type ** : Semi-structured / Flexible (AI-driven progression) ** Analysis Dimensions ** : 1. Interview depth and quality 2. Interviewee engagement level 3. Topic coverage completeness across ALL topics 4. Conversation flow and natural transitions 5. Follow-up opportunities ** Your Task ** : Analyze the conversation history and provide strategic guidance: - Which topics/subtopics have been covered adequately - Whether to probe deeper or transition to new areas - Quality of information gathered so far - Suggested angles, follow-ups, or transitions to pursue - Coverage gaps that should be addressed ** Note ** : The interviewer AI will decide the next question - your role is strategic guidance only. ** Conversation History ** : history 43 D User Study Details D.1 Upwork Details As mentioned, we recruited70workers across7professions via Upwork 22 , where participants were randomly assigned to be interviewed by either SparkMe or MimiTalk, with35participants in each system and5participants from each of the following professions: researchers, software engineers, HR or administrative staff, creative and content professionals, educators, data analysts or scientists, and business or supply chain operations. Three additional participants outside of the70recruited participants voluntarily withdrew during the study and are therefore excluded from the analysis. Figure 14 and Figure 15 showcase our web interviewer platform for the Upwork workers. Figure 17 and Figure 18 D.2 Leading Question Evaluation In addition to collecting user ratings, we evaluate the presence and use of leading questions in the user study interviews as a measure of data quality and interpretive validity. Leading questions are questions that steer participants’ responses by introducing new content, embedding presuppositions, or implicitly evaluating an answer, which would influence the qualitative responses and threaten data reliability (Cairns-Lee et al., 2022, Loftus, 1975, Poole and Lindsay, 1995, Swann et al., 1982). We use the cleanness framework introduced by Cairns-Lee et al. (2022), which follows the principles of clean questioning by Tosey et al. (2014). The instruction prompt and the cleanness rubric are provided in Figure D.1, but because the criteria for scores of3,4, and5are highly similar, we collapse them into three distinct categories for evaluation, resulting in scores ranging from 1–3. We usemR3-Qwen3-14B(Anugraha et al., 2025a), a rubric-based reward model, as the evaluator. Overall, MimiTalk achieves an average score of2.973± 0.042, while SparkMe achieves2.958± 0.047. These results indicate that both systems produce very few leading questions and maintain a consistently high level of neutrality. Furthermore, we observe only two instances of a score of1for SparkMe and three instances for MimiTalk, which is shown in Table 6. One illustrative example from SparkMe is: “It’s interesting that prompt writing has become an important skill for you. Could you tell me more about how learning to write effective prompts has changed the way you work with AI? Are there particular challenges or benefits you’ve noticed as you’ve developed this skill?” While this question contains a mild degree of implicit guidance, it remains largely open-ended and does not strongly bias the user’s response. D.1: Leading Question Instruction Template # Instruction You will be given the full transcript of a semi-structured interview and one interviewer question to evaluate. Evaluate the interviewer’s question-asking corresponding to the given question using a Cleanness rubric. In particular, assess whether the question (and any immediate follow-up phrasing) is grounded in the interviewee’s prior responses or introduces interviewer-generated content. A leading question is defined as an interviewer utterance that introduces new content, embeds presuppositions, or evaluates the interviewee’s response. 22 https://w.upwork.com/ 44 Figure 14: The front page of our interviewer platform, along with the instructions and PII notice. Use the interviewee’s response only to assess whether the question compromises the authorship of the answer, not to judge the quality or correctness of the response itself. 45 Figure 15: Our web-interviewer platform, where SparkMe will display their response along with their voice using text-to-speech (TTS). The interviewee may respond in either text (through the chat box) or audio, depending on their convenience. # Evaluation Rubric 1: Strongly Leading - The utterance clearly introduces content, presupposition, or evaluation that suggests a specific answer, and the interviewee’s response raises reasonable doubt about whether the answer reflects their independent experience. 2: Mildly Leading - The utterance introduces some new content, presupposition, or evaluative framing that suggests a direction of response, but the interviewee’s reply does not indicate compromised authorship or compliance. 3: Contextually Clean - The utterance introduces the interview topic or follows the interviewee’s logic while minimizing superfluous content. It remains within the interviewee’s framing and does not introduce presuppositions or evaluations. 4: Clean Repeat - The question reformulates, repeats, or logically extends interviewee-generated content without introducing new content, presupposition, or evaluation. 46 Figure 16: A sample of our annotation platform for the post-interview questionnaire. The annotators are given the instruction rubrics based on Table 7, Table 8, and Table 9, along with the subtopic and summary or notes collected during the interview. 5: Classically Clean - The utterance uses only interviewee-generated content or universal constructs and conforms to a classically clean question form, introducing no interviewer-generated structure beyond the question itself. D.3 Examples of High-Quality Emergent Insights. To illustrate the value of emergent content surfaced by SparkMe, we provide three emergent subtopics that received high peer ratings for both topic-level emergence and analytical value. 23 Educators. In our survey, educators highlighted an interesting case of AI integration in lesson planning workflows. One participant described using ChatGPT extensively to generate lesson content and activities, integrating outputs with Canva and Google Slides to structure daily presentations. They also noted that formatting inconsistencies often required manual adjustments. Peer annotators rated this insight highly, recognizing its practical value, while also noting that the manual adjustments 23 Due to the relatively small sample size, we caution against drawing takeaways about the workforce from these findings; we present these as a demonstration of the value of SparkMe. 47 Figure 17: The instruction given for human annotations to verify our LLM-as-a-judge to evaluate coverage against ground truth. Figure 18: A sample of information presented for human annotations to verify our LLM-as-a-judge to evaluate coverage against ground truth. demand significant time and effort, both indicating potential benefits of AI-assisted workflows and the ongoing challenges of human-AI collaboration in teaching. HR Professionals. Among HR and Administration professionals, one participant highlighted the growing importance of multilingual and in-person support skills in AI-augmented client services. They explained that fluency in the client’s language and demonstrating empathy were critical for overcoming communication barriers and reducing client anxiety. The participant noted that as AI 48 automates some of their routine tasks, they can now focus more on building stronger interpersonal relationships, including practicing languages with AI assistance. Creative Professionals. Creative and Design professionals also shared perspectives on the future outlook for AI, particularly regarding copyright and legal constraints. One participant explained that these rules restrict AI usage to internal concept development, preventing AI-generated outputs from being used in final commercial work. As a result, teams must rely on traditional methods to complete art work, which limits experimentation and slows learning about AI capabilities. Peer annotators agreed that this insight reflects important considerations for the future of AI in creative work, highlighting the need to balance innovation with legal and ethical constraints. D.4 Rubrics Used for User Study Evaluation The instructions for expert annotators for the user study are provided in Table Table 7, Table 8, and Table 9. 49 TopicSubtopics AI Experience and Tool Adoption • Familiarity with fundamental AI/ML concepts and terminology • Names of specific AI software/platforms currently used in work • Frequency and purpose of AI tool application (specific use cases) •Specific examples of AI success and failure experiences (lessons learned) • Availability of organizational training or peer resources for AI use AI Interaction Style and Workflow Change •Preferred mode of interaction: independent versus step-by-step col- laboration • Style of human-AI teaming (e.g., advisor, assistant, co-worker) • Willingness and openness to adopting new AI-driven workflows • Preference for conversational vs. command-based interfaces (commu- nication dynamics) Trust and Control Over AI • Extent to which tasks rely on specialized, tacit domain knowledge • Level of trust in AI outputs for work tasks and critical decisions • Ideal balance of human effort and AI automation for specific tasks •Conditions under which high automation is acceptable or threatening AI Impact on Skills and Job Security •Perceived impact of AI on the importance of existing skills (enhanced vs. reduced) • Emerging skills or new areas of responsibility created by AI • Level of concern about AI replacing specific tasks or the overall role • Availability of override mechanisms or manual checks for AI-driven processes •Perceived change in team or company policies regarding AI adoption AI Attitudes and Fu- ture Outlook • General outlook on AI’s broader societal and industry impact • Personal beliefs about the ethics and risks of AI in the workplace • Missing AI tools or features that would be most beneficial in the future •Predicted evolution of their job in the next 5-10 years with AI integra- tion • Concrete steps they would want their organization to take regarding AI strategy Table 4: Interview Topics and Subtopics (Part 2 of 2) 50 ModelKendall’s τExact AgreementWithin 1 pointWithin 2 points mR3-Qwen3-14B0.54945.2%88.2%98.9% Qwen3-30B-A3B-Instruct0.52441.9%74.2%88.2% GPT-4.10.43931.2%75.3%91.4% GPT-4.1-mini0.40523.7%62.4%78.5% Table 5: Kendall’sτcorrelation between LLM-as-a-judge and human rating annotations for coverage assessment on100samples total, with20instances per rating from 1 to 5. Each sample is annotated by 3 humans, where the human inter-annotator agreement is Krippendorff’s α = 0.65. SystemLeading Questions SparkMeQ1. It’s interesting that prompt writing has become an important skill for you. Could you tell me more about how learning to write effective prompts has changed the way you work with AI? Are there particular challenges or benefits you’ve noticed as you’ve developed this skill? Q2. It’s great to hear you’ve had positive experiences with several AI platforms. Shifting gears a bit, I’m curious about how you feel AI has impacted the importance of your existing skills as a data scientist. Do you find that AI has enhanced the value of your current expertise, or has it reduced the need for some skills? How has that influenced the way you approach your work? MimiTalkQ1. What motivates you to dedicate 3 to 4 hours daily to learning new technologies, and are there any challenges you face in maintaining this learning habit? Q2. Can you tell me about a recent time when an AI chat tool helped you solve a tricky coding or IT issue? What was the problem and how did the AI assist you? Q3. Why do you feel that none of your tasks are suitable for automation? Are there specific reasons or concerns that make automation less appropriate? Table 6: All detected leading questions posed during the user study interview by the SparkMe and MimiTalk 51 DimensionDescriptionScale (1–5) CoverageHow comprehensively did the interview cover the intended topics? 1: Missed most key areas 2: Covered some areas but missed sig- nificant parts 3: Covered main points but lacked de- tail 4: Covered most areas well 5: Comprehensive coverage of all in- tended topics DepthDid the interviewer probe deeply enough to get meaningful answers? 1: Very superficial; surface-level only 2: Mostly superficial with rare follow- up 3: Adequate depth but missed opportu- nities 4: Good depth; probed well 5: Exceptional depth; uncovered pro- found insights CorrectnessHow accurate were the interviewer’s sum- maries and interpretations? 1: Frequent major inaccuracies 2: Several errors or misunderstandings 3: Mostly accurate with some minor er- rors 4: Accurate with rare minor slips 5: Flawless interpretation and sum- mary Table 7: Evaluation rubrics given to human annotators to evaluate the interview’s content quality. 52 DimensionDescriptionScale (1–5) Topical RelevanceHow much did this emergent subtopic add new dimensions to the planned topic being dis- cussed? 1: Irrelevant or distracting 2: Weakly relevant 3: Somewhat relevant 4: Mostly relevant 5: Highly relevant and aligned Topic-Level EmergenceDid this subtopic reveal novel perspectives within the planned sub-topics being discussed? Did it go beyond obvious aspects to uncover new angles? 1: Adds nothing new 2: Minor/redundant connections 3: Moderately emergent 4: Clearly emergent/meaningful 5: Highly emergent; substantial novel directions SurpriseHow unexpected were the emer- gent ideas relative to what had already been discussed? 1: Entirely expected/routine 2: Slightly unexpected 3: Moderately unexpected 4: Highly unexpected 5: Strongly surprising while remaining relevant Analytical ValueHow much does this contribute to understanding AI workforce issues across the industry? 1: Distracting/uninformative 2: Little analytical insight 3: Some insight, limited utility 4: Meaningful analytical insight 5: Substantially deepens understand- ing Table 8: Evaluation rubrics given to human annotators to cross-evaluate emergent insights in different interviews. 53 DimensionDescriptionScale (1–5) ClarityHow clear were the questions and the flow of the conversation? 1: Confusing and disjointed 2: Often unclear 3: Generally clear but sometimes vague 4: Very clear and easy to follow 5: Crystal clear and perfectly structured AdaptivenessHow well did the interviewer adapt to the interviewee’s responses? 1: Rigid; ignored responses 2: Struggled to adapt 3: Adapted adequately 4: Adapted well to flow 5: Seamlessly adapted to every nuance Comfort & FairnessDid the interviewee feel comfortable and treated fairly? 1: Hostile or unfair environment 2: Uncomfortable or biased 3: Neutral/Adequate 4: Comfortable and fair 5: Extremely supportive and unbiased Overall ExperienceTaking everything into account, how would the interviewee rate the ses- sion? 1: Poor 2: Below Average 3: Average 4: Good 5: Excellent Table 9: Evaluation rubrics given to human annotators to evaluate the interview overall interaction quality. 54

Instant research discovery