Paper deep dive
Gricea: An Open Science Platform for Conversational AI Research
Nikhil Sharma, Yunlin Gong, Xinyang Cheng, Ziang Xiao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/21/2026, 4:55:27 AM
Summary
The paper introduces Gricea, an open-science platform for Conversational AI (CAI) research that represents studies as configurable, deployable, and reusable research artifacts. Gricea addresses the fragmentation in CAI reporting by coupling study procedures (Study Flow) with participant-facing systems and task behaviors (Task Flow). The platform was evaluated through a replication study of CUI 2026 papers (achieving 93% replication success) and a user study with researchers, demonstrating its ability to support the construction, reproduction, and extension of CAI studies, thereby facilitating cumulative knowledge building.
Entities (10)
Relation Signals (7)
Nikhil Sharma â affiliatedwith â Johns Hopkins University
confidence 95% · Nikhil Sharma ... Affiliation: Johns Hopkins University
Gricea â comprises â Study Flow
confidence 95% · Gricea represents a study through connected Study Flow and Task Flow definitions
Gricea â comprises â Task Flow
confidence 95% · Task Flow represents behavior within each task
Gricea â supportsrepresenting â Conversational AI Studies
confidence 95% · Gricea represents studies as configurable, deployable research artifacts
Gricea â addressesproblem â Fragmented Reporting
confidence 90% · fragmented reporting of systems and study configurations hinders replication
Gricea â replicated â CUI 2026 Papers
confidence 90% · we replicated configurations 93% of eligible CUI 2026 papers
oTree â supports â Browser-based Experiments
confidence 90% · oTree supports browser-based experiments through Python and HTML
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We need studies on conversational AI (CAI) at scale to understand human behavior and shape CAI design. However, fragmented reporting of systems and study configurations hinders replication, extension, and knowledge accumulation. We present Gricea, an open-science platform representing studies as configurable, deployable research artifacts that researchers can run, inspect, share, and reuse. Informed by a formative analysis of prior CAI research, Gricea couples study procedures, participant-facing systems, and conversational task behavior in. In a replication study using Gricea, we replicated configurations 93% of eligible CUI 2026 papers; while also flagging missing information in 96% of papers that hinder faithful replication --- further motivating Gricea's need. In a user study, researchers and practitioners from diverse backgrounds successfully constructed runnable studies addressing various open-ended research questions. Together, these findings demonstrate Gricea's support for constructing, reproducing, and extending CAI studies through shared research artifacts, enabling cumulative knowledge building through open science.
Tags
Links
- Source: https://arxiv.org/abs/2609.22039v1
- Canonical: https://arxiv.org/abs/2609.22039v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
113,584 characters extracted from source content.
Expand or collapse full text
Gricea: An Open Science Platform for Conversational AI Research CCS: Human-centered computing HCI design and evaluation methodsCCS: Human-centered computing User studiesCCS: Human-centered computing Empirical studies in HCICCS: Information systemsCCS: Human-centered computing Collaborative and social computingCCS: Human-centered computing Field studiesCCS: Human-centered computing Usability testingCCS: Human-centered computing User modelsCCS: Human-centered computing Natural language interfacesCCS: Human-centered computing Web-based interactionCCS: Human-centered computing Collaborative interactionCCS: Human-centered computing User interface management systemsCCS: Human-centered computing Interaction design process and methods Nikhil Sharma email: nsharm27@jhu.edu Affiliation: Johns Hopkins University, Baltimore, MD, USA , Yunlin Gong email: ygong37@jh.edu Affiliation: Johns Hopkins University, Baltimore, MD, USA , Xinyang Cheng email: xcheng45@jhu.edu Affiliation: Johns Hopkins University, Baltimore, MD, USA and Ziang Xiao email: ziang.xiao@jhu.edu Affiliation: Johns Hopkins University, Baltimore, MD, USA Figure 1. A shared research foundation for conversational AI. A: Gricea represents a study through connected Study Flow and Task Flow definitions, preserving both configured choices and unchanged defaults in an inspectable, executable artifact. B: Other researchers can inspect and extend the artifact by changing its grounding context while retaining the remaining study design; The updated study can be deployed and shared. C: The research community and AI agents can build on shared executable artifacts across successive studies, accumulating new findings into collective knowledgeThree panels show the lifecycle of a shared study artifact. A research configures a between-subject study using connected Study Flow and Task Flow graphs. Another researcher inspects that artifact, adds their configuration edit to the study while retaining other settings, and publishes a new version. The research community can through a shared executable representation builds collective knowledge. Abstract. We need studies on conversational AI (CAI) at scale to understand human behavior and shape CAI design. However, fragmented reporting of systems and study configurations hinders replication, extension, and knowledge accumulation. We present Gricea, an open-science platform representing studies as configurable, deployable research artifacts that researchers can run, inspect, share, and reuse. Informed by a formative analysis of prior CAI research, Gricea couples study procedures, participant-facing systems, and conversational task behavior in. In a replication study using Gricea, we replicated configurations 93% of eligible CUI 2026 papers; while also flagging missing information in 96% of papers that hinder faithful replication â further motivating Griceaâs need. In a user study, researchers and practitioners from diverse backgrounds successfully constructed runnable studies addressing various open-ended research questions. Together, these findings demonstrate Griceaâs support for constructing, reproducing, and extending CAI studies through shared research artifacts, enabling cumulative knowledge building through open science. Keywords: Conversational AI, Controlled Studies, Human Subject Studies, Research Platform Platform Tutorial 1. Introduction Conversational AI systems built on large language models are becoming a default interface for information access, communication, and everyday work (Chatterji et al., 2025). As these systems become everyday infrastructure, they shape how people learn, create, collaborate, make decisions, and form relationships with artificial agents (Jakesch et al., 2023; Kirk et al., 2025). Understanding these changes requires empirical research on human behavior and experience alongside evaluation of the systems people use. Such evidence is essential to the design, governance, and deployment of conversational AI. Prior humanâAI interaction research shows that outcomes depend on various factors such as model performance, how systems communicate uncertainty, present evidence, structure initiative, support verification, and distribute control (Amershi et al., 2019; Sharma et al., 2024; Zamfirescu-Pereira et al., 2023; Kirk et al., 2025). Misinformation exposure, overreliance, persuasion, and uneven information access emerge through the interplay of model behavior, interface design, and user context (Rathod M.S., 2024; Sharma et al., 2025; Sharma et al., 2026; Zhang et al., 2025; Shi et al., 2026). Studying these effects requires examining how people interact with conversational AI while pursuing goals in specific tasks and contexts; researchers use Controlled human-subject studies to isolate effects of different design choices (Liao and Xiao, 2025). Building cumulative knowledge using controlled studies about conversational AI requires a shared frame of reference for what participants experienced, how conditions were configured, and how outcomes were measured. When those configurations and materials are difficult to recover, subsequent researchers must reconstruct the interaction before they can reproduce or extend a study. Differences in interface behavior, agent configuration, or procedure can then obscure what was preserved and what changed. Open science therefore requires preserving these methodological choices in an inspectable, reusable form alongside the findings (Foster and Deardorff, 2017; Aguilar et al., 2024). A shared representation gives researchers a common starting point for reproducing conditions, systematically varying design choices, and relating new findings to prior work. However, representing these conditions is challenging because design choices span several connected parts of a study: the interface and actions available to participants, agent behavior context, task, modality, participant population, and experimental procedure. Furthermore, all of these conditions can interplay with each other creating a vast and diverse design space. A shared representation must make the design choices both at the study level and at the task configuration level explicit; preserving a comprehensive representation of based on the CAI design space (Section 3). Even without the overhead of a shared representation, conducting these studies has high costs beyond just recruiting and compensating participants. Researchers must coordinate study procedures, task interfaces, agent behavior, and data collection, often by integrating survey tools, custom interfaces, model services, and deployment infrastructure. Commercial conversational systems offer limited experimental control, while changes to their interfaces, retrieval policies, or models can alter the conditions under investigation. Building and maintaining custom systems therefore demands time and engineering expertise that can constrain both who conducts conversational AI research and which questions they pursue. Therefore, a successful shared representation artifact must not only find a scalable way to represent the vast and diverse design space but also reduce the costs of building these custom systems while making the artifact a by-product of the process rather than an additional cost to the researchers. In this paper we present Gricea, a platform that represents conversational AI study designs as configurable, deployable research artifacts that researchers can run, inspect, share, and reuse. Researchers visually author study procedures and configure complex interactive task conditions within a platform that supports participant-facing execution and instrumentation. The representation researchers inspect is also the definition the runtime executes, connecting experimental assignment and task behavior to the participant experience. Gricea reduces the technical overhead of constructing controlled studies and preserves immutable study versions as reusable templates. Other researchers can inspect the original design, reproduce its conditions, and extend it to new questions, making individual studies reusable resources enabling cumulative knowledge (Figure 1). Griceaâs design was informed by a formative analysis of 57 papers on conversational AI (Section 3). The resulting desiderata guided two complementary representations. Study Flow represents the experimental procedure, connecting surveys, tasks, and between- and within-subject structures. Task Flow represents behavior within each task, including agent configuration and the participant-facing interface. Researchers can combine these elements to implement existing designs and complex configurations involving multiple agents, multiple users, and customized interfaces. We evaluated Gricea with N=10N=10 researchers and practitioners from diverse disciplinary backgrounds, who completed an assisted authoring walkthrough before independently designing studies around their own research questions. Participants implemented valid, runnable studies that varied across procedures, interfaces, models, surveys, and outcomes, and the no-code interface reduced barriers for researchers without systems backgrounds. We also examined CUI 2026 full papers, none of which informed Griceaâs design. Of 29 eligible papers, we replicated designs from 27 as executable artifacts: 10 completely and 17 partially. Together, these evaluations demonstrate support for authoring diverse studies and reconstructing published designs for inspection and reuse. This paper makes three contributions. First, we contribute a shared, executable representation of conversational AI studies that connects study procedure, task behavior, participant-facing conditions, and instrumentation, enabling researchers to inspect, reproduce, and extend experimental designs. Second, we present Gricea, a configurable no-code research platform that operationalizes this representation through visual authoring, participant-facing execution, immutable study versions, shareable configurations, and reusable community templates. These mechanisms lower technical barriers while making individual studies available as resources for subsequent research. Third, we contribute empirical findings from an authoring study with researchers and practitioners from diverse disciplinary backgrounds and reproductions of CUI 2026 study configurations, demonstrating Griceaâs support for diverse research questions and published study designs while identifying remaining needs for guidance, validation, and workflow support. 2. Related Work Gricea builds on research that broadens participation in online studies, preserves methods as reusable research resources, and makes conversational systems configurable through higher-level representations. These efforts address complementary requirements for conducting research and building on its findings. Gricea brings these requirements together through a shared representation of study procedure and conversational task behavior. The representation used to design a study governs its deployment and data collection, preserving an inspectable research artifact as part of building and running the study. 2.1. Infrastructure for Crowdsourced and Online Studies Online research infrastructure has expanded where studies can be conducted and who can participate. Kittur et al. (2008) examined how task design and quality checks influence crowdsourced judgments, while Reinecke and Gajos (2015) used personalized feedback to attract uncompensated participants and evaluated online replications of laboratory studies. Subsequent comparisons examined differences in participant diversity and data quality across recruitment platforms (Peer et al., 2017; Peer et al., 2022). These efforts demonstrate the influence of recruitment in the quality of the online studies. Conducting experiments with these participant populations also requires infrastructure for implementing tasks, assigning conditions, coordinating interactions, and collecting responses. Reusable experiment frameworks address these requirements by providing components that researchers can adapt across studies. oTree supports browser-based experiments through Python and HTML, including a library of reusable game templates (Chen et al., 2016). Empirica supports configurable experimental designs and reusable protocols for real-time group experiments (Almaatouq et al., 2021), while jsPsych enables researchers to construct behavioral experiments from reusable plugins and contribute new tasks to a community ecosystem (de Leeuw et al., 2023). Across these systems, reusable components allow the implementation work behind one study to support subsequent studies, reducing the effort required to develop and extend experimental designs. HumanâAI research platforms bring agent behavior into this experimental infrastructure. Deliberate Lab combines no-code experimental stages, human and LLM participants, agent mediators, and cohort management for studying humanâAI group dynamics (Qian et al., 2025). For conversational AI studies, the experimental condition depends on more than the sequence of study stages or an agentâs configuration. To represent a broad set of CAI studies, we conduct a formative study to uncover the design space, allowing Gricea to extend existing efforts to a more general reusable research infrastructure through a coupled representation of study procedure and conversational task behavior. 2.2. Infrastructure for Open Science Open-science infrastructure supports the preservation and exchange of research materials across teams. Foster and Deardorff (2017) describe infrastructure for project organization, collaboration, file versioning, and registration, making materials easier to preserve and share. However, accessible materials must also be sufficiently specified and connected for others to use them. Iarygina et al. (2026) identified obstacles to computational reproduction among CHI papers that shared data and analysis code, illustrating the difference between making resources available and enabling others to reproduce the work. For conversational AI studies, researchers need to understand how the procedure, interface, agent behavior, and materials jointly determined what participants experienced. Executable research representations connect methodological specification to implementation. Aguilar et al. (2024) represent experiment components through automation code and digital documentation, including infrastructure, data collection, analysis, and management. Nobre et al. (2021) support inspecting participant behavior through interaction provenance and replay, while Cutler et al. (2026) connect study specification, execution, analysis, and dissemination within a browser-based framework. Subsequent LLM integration preserves conversation history and supports replay of chatbot interactions (He and Lex, 2026). Gricea builds on this connection between executable methods and inspectable interactions through a shared representation of conversational task logic and the surrounding experimental procedure which is also the same representation that the runtime executes. Reporting frameworks and agent-native research artifacts further clarify what must survive publication. Feuerriegel et al. (2026) call for explicit documentation of LLM use, including model versions, prompts, and configurations, while Liu et al. (2026b) connect scientific logic, executable code, exploration traces, and evidence so that humans and AI agents can understand and build on research. Gricea integrates artifact preservation into the development and execution of participant studies. The configured procedure, prompts, materials, and interaction logic constitute the study that is deployed, so researchers do not need to reconstruct a separate artifact after implementation. Sharing that representation makes the implemented method available for inspection, reconfiguration, and reuse within the same environment. 2.3. Infrastructure for visual programming of CAI studies Visual and declarative systems make computational choices accessible through representations that users can inspect and modify. Wu et al. (2022) support composing and debugging multi-step LLM chains, while Arawjo et al. (2024) support systematic comparison of prompt and model variations through a visual dataflow environment. Cai et al. (2024) allow users to edit a proposed workflow before an LLM executes it, and Feng et al. (2025) support structured specification and testing of model behavior within interface design work. Conversational application platforms also provide deployment environments, and live-traffic experiments (Google Cloud, 2026b; Google Cloud, 2026c; Google Cloud, 2026a). Gricea brings this control over computational behavior into the representation of a human-subject study, connecting experimental assignment, participant interaction, and measurement. Research-oriented representations bring methodological choices into these abstractions. Jun et al. (2019) allow researchers to declare study designs, assumptions, and hypotheses for statistical analysis. Yao et al. (2026) provide an experiment configuration language and controls over collaborative environments, agent perception and action, and synchronized interaction logs, while Zhang et al. (2024) support configurable humanâAI teaming environments and feedback collection. Gricea separates and couples study procedure and conversational task logic through the same graphs that drive execution. Researchers can examine how a procedural decision changes the participant-facing condition and preserve that relationship when a study is shared, reproduced, or extended. The formative analysis that follows identifies the recurring study requirements that informed this design. 3. Formative Analysis: The Science of Conversational AI Studies To scope the infrastructural requirements for Gricea, we conducted a formative design space analysis of papers on conversational AI systems. Our goal was to identify recurring patterns across prior work: what studies on conversational AI investigate, what they manipulate, how they are typically conducted, what technical demands those choices create, and what forms of infrastructure existing systems already provide. From these recurring patterns, we identified what researchers need to specify and control, and which details must remain inspectable for others to reproduce and build on a study. These requirements motivate five design desiderata for Griceaâs study representation and authoring environment. Analysis Procedure. We began by collecting 100 candidate papers using keyword combinations around conversational AI, agent, or LLM, together with terms related to users, humans, and studies, across venues and repositories such as CHI, UIST, CUI. We then filtered this set to 57 papers that centered participant-facing conversational or agentic AI systems and provided sufficient detail about the study design, system configuration, or evaluated interaction condition. For each paper, the research team coded the study type, focus area, participant count, independent and dependent variables, between- and within-subject structure, procedural stages, system or pipeline components, analysis methods, and the overall structure of the study procedure. The research team reviewed and clustered these codes to identify recurring outcome areas, manipulation dimensions, procedural structures, and infrastructural demands. As part of this analysis, we also examined how papers visually represented their study designs. Papers used staged diagrams, branching structures, and flowcharts to communicate condition assignment, task sequences, and follow-up measures. These representations make explicit how study logic structures the activities participants experience; motivating Griceaâs support for executable visual representations: researchers should be able to design, inspect, communicate, and run a study through the same representation, without reconstructing its logic manually in code. 3.1. What studies on conversational AI investigate Studies on conversational AI investigate how configured assistants shape human behavior, judgment, and experience within particular task settings. In our corpus, these settings included co-writing, conversational search, learning, dietary recommendation, and daily planning and reflection. Participants composed text with generated suggestions(Jakesch et al., 2023), explored information through dialogue, received personalized recommendations(Liang et al., 2025), or revisited plans across sessions(Abbas et al., 2025). Each task establishes what participants are trying to accomplish and the role the assistant plays in that activity. Within these settings, the outcomes of interest are similarly broad. Prior work examines trust, reliance, persuasion, misinformation response, privacy behavior, writing quality, learning, and collaboration (Jakesch et al., 2023; Sharma et al., 2024; Sharma et al., 2025; Sharma et al., 2026; Zhang et al., 2025; Shi et al., 2026). What links these studies is not a single application domain, but a common methodological concern: how a conversational system condition shapes what users believe, do, and produce over time. A platform for this area must therefore support both configuring the participant-facing interaction and collecting the evidence needed to examine its outcomes, including self-reports, behavioral traces, and task outputs (Baradari et al., 2025; Liu et al., 2026a; Li et al., 2024). 3.2. The manipulation space of conversational AI studies Our formative analysis shows that studies on conversational AI manipulate far more than prompts or underlying models. The true experimental object is a configured interaction condition: the combination of agent behavior, interface, context, and procedure that defines what participants experience. For example, a study of chatbot relationship framing varied both the agentâs self-description and the visibility of conversation history across sessions (Cox et al., 2025). Therefore, representation of studies requires specifying both what researchers manipulate and the surrounding configuration they hold constant. We synthesize recurring configurations into six interacting dimensions: Interface Condition, Agent Condition, Context & Grounding, Task & Modality, Domain & Audience, and Study Procedure. Table 1 summarizes their configurations and infrastructural implications. These dimensions connect what researchers configure, what participants experience, and how the study is conducted. Dimension Examples of study configuration Implications for study infrastructure Interface Condition Layout and workspace organization; citation and provenance display; highlighting; progress guides; verification prompts; structural overlays; editability of shared artifacts; time-synchronized guidance and conversation navigation (Chen et al., 2025; Cheng et al., 2024; Jeong et al., 2025; Chi et al., 2025; Nguyen et al., 2025). Represent what participants can see, inspect, and change. Preserve presentation and available actions as part of the study condition, including how evidence, guidance, and task progress are displayed. Agent Condition System instructions; persona or role framing; stance; response style; initiative; clarification behavior; trust-adaptive interventions; tool-use policy; response delays and waiting cues; coordination of model components (Jakesch et al., 2023; Dhillon et al., 2024; Li et al., 2025; Liu et al., 2025; Maslych et al., 2025; Liang and Luo, 2025). Make agent behavior, permitted actions, timing, and coordination explicit. Distinguish the roles participants encounter from the model components that implement those roles. Context & Grounding Retrieved documents; retrieval policy; context filtering; citation constraints; external evidence injection; opposing or balanced context; task context; interaction history; persistent memory and its inspection or editing (Lewis et al., 2020; Sharma et al., 2024; Sharma et al., 2025; Chi et al., 2025; Yan et al., 2025). Specify what information the agent receives, what participants can inspect or change, and how context persists across turns or sessions. Distinguish information available to the agent from history displayed to participants and records retained for research. Task & Modality Search, writing, tutoring, coding, annotation, voice, multimodal interaction, direct artifact manipulation, and conversational control of haptic feedback (Qian et al., 2025; Yao et al., 2026; Flores-Saviaga et al., 2025; Caetano et al., 2025; Mishra and Schneider, 2025). Connect different participant activities and modality-specific inputs and outputs to a shared study procedure. Support task-specific actions and outcome collection without rebuilding the surrounding study infrastructure. Domain & Audience Topic domain; expertise; prior beliefs and experience; participant population; targeted community; high-stakes versus low-stakes settings; language and dialect; accessibility and care needs (Pang et al., 2025; Rathod M.S., 2024; Ge et al., 2024; Flores-Saviaga et al., 2025; Elwahsh et al., 2025; Czech et al., 2025). Document the task setting, study materials, and relevant participant characteristics separately from assigned conditions. Preserve the context needed to interpret findings and adapt a study for another population or setting. Study Procedure Pre-task elicitation; condition assignment; branching; within- or between-subject structure; counterbalancing; post-task measures; longitudinal follow-up; scheduled check-ins and cross-session continuity (Jakesch et al., 2023; Qian et al., 2025; Cai et al., 2023; Abbas et al., 2025; Cox et al., 2025; Penney et al., 2025). Make assignment, task order, branching, measurement timing, and cross-session continuity explicit and authorable. Preserve the procedure as executable logic that can be inspected, reused, and extended. Table 1. Recurring dimensions of conversational AI study configuration, synthesized from the formative analysis. Examples include experimental manipulations, fixed settings, and participant characteristics. The infrastructure implications identify what must remain explicit to design, inspect, reproduce, and extend a study. Across these dimensions, understanding a study requires inspecting how its procedure, interface, agent behavior, and contextual information jointly produce the participant experience (Jakesch et al., 2023; Sharma et al., 2024; Cox et al., 2025). Researchers need to distinguish the choices that define a condition from those held constant, and to understand how those choices are implemented. A shared study representation should preserve these relationships so that both the original research team and subsequent researchers can inspect the design, reproduce its conditions, and make deliberate changes when extending it. 3.3. The Procedural Anatomy of Conversational AI Studies The analysis also shows that studies on conversational AI combine system configurations with structured research procedures. A study may begin with consent, instructions, and pre-task elicitation, proceed through condition assignment and an interactive task, and conclude with post-task measures, reflection, or interviews. Prior work combines these elements in co-writing tasks with pre- and post-task measures (Jakesch et al., 2023; Dhillon et al., 2024), conversational search with turn-level behavior and post-task attitudes (Sharma et al., 2024; Avula et al., 2022), and daily planning and reflection that link repeated conversations to daily surveys and an exit interview (Abbas et al., 2025). A study on conversational AI therefore couples a study procedure with an interactive runtime. The study procedure determines how participants move through the study: the instructions they receive, the condition they encounter, when branching occurs, and when measurements are collected. The interactive runtime determines what participants experience within a task: what they can see and do, what information the system receives, how it responds, how the interface adapts, and which actions and responses are logged. Together, these levels specify both how participants encounter a condition and how that condition operates during the interaction. Across these studies, condition assignment and information collected before the task can configure the agentâs behavior and available context, while participantsâ actions within the task can determine subsequent interaction paths and when post-task measures are collected (Jeong et al., 2025; Liang et al., 2025; Abbas et al., 2025). A research platform must therefore represent these dependencies explicitly, connecting the assigned condition, the interaction participants experience, and the evidence collected about its outcomes. This requirement motivates Griceaâs coupled support for Study Flow and Task Flow, allowing researchers to specify the surrounding procedure and within-task behavior as connected parts of the same study. 3.4. Synthesis of Infrastructural Challenges Executing studies across this broad manipulation space is challenging because the experimental condition is distributed across many components that must be built and controlled in tandem. Researchers often need to stitch together survey tools, custom interfaces, backend orchestration, model and retrieval pipelines, assignment logic, deployment infrastructure, and fine-grained behavioral logging (Qian et al., 2025; Yao et al., 2026). Even when the intended manipulation is conceptually straightforward, implementing it as a controlled, reproducible participant experience requires substantial engineering effort, creating barriers for researchers without the technical expertise or resources to build and maintain this infrastructure. Maintaining experimental control also requires researchers to specify how interface affordances, retrieval behavior, model configuration, and system defaults jointly produce the participant experience (Qian et al., 2025; Yao et al., 2026). Reliance on commercial systems adds dependencies whose behavior may not be fully exposed or preserved across versions, with documented changes in model behavior showing why the same prompt and model name do not establish an equivalent condition (Chen et al., 2024). Study infrastructure must therefore make the configuration and its dependencies inspectable alongside the record of what participants actually encountered. When instructions, prompts, task materials, interface behavior, and procedural logic remain embedded in one-off implementations, publishing the findings does not necessarily preserve the condition needed to reproduce or extend the study. Subsequent researchers must reconstruct how these elements were connected before they can determine whether a new implementation reproduces the original condition or introduces consequential differences. Preserving the configured study as an inspectable, executable artifact provides a shared frame of reference for comparing implementations, adapting procedures, and building on prior work (Qian et al., 2025; Yao et al., 2026). Hence, such infrastructures must reduce the effort of constructing studies along with preserving the configuration necessary for collective knowledge to accumulate. 4. Design Desiderata Building on the formative analysis (Section 3), we derive five design desiderata for infrastructure that supports the design, execution, and reuse of conversational AI studies. D1: Explicit Representation of Study Conditions. The platform must represent the configured interaction condition shown to participants rather than only isolated prompts, screens, or model calls. Prior studies manipulate agent behavior and participant-facing interaction support while also specifying shared interfaces, controls, and contextual information across conditions (Jakesch et al., 2023; Chi et al., 2025). The infrastructure must therefore make the relationships among procedure, interface, agent behavior, and context explicit, allowing researchers to distinguish what is manipulated from what is held constant. Researchers must be able to inspect how these choices shape what participants see, what actions they can take, and how the system responds, without reconstructing the condition from separate implementation details. This representation must support checking whether the implemented assignment, task behavior, and measures correspond to the intended experimental design before publication. D2: Coupled Support for Study Flow and Task Flow. The platform must support both the overall study procedure and the behavior of the interactive task, while keeping their roles distinct. Study Flow specifies how participants move through instructions, condition assignment, tasks, and measures, whereas Task Flow specifies how participant actions, agent responses, and branching shape progression within a task. Prior studies connect information collected before an interaction to the agentâs behavior and assess the resulting experience through subsequent measures (Liang et al., 2025). Researchers must therefore be able to author and modify each flow separately while specifying how information enters a task, when the task ends, and how its outputs connect to subsequent study stages. D3: Lower Technical Barriers to Controlled Study Authoring. The platform must reduce the amount of bespoke engineering required to build and deploy controlled studies on conversational AI (Qian et al., 2025). Our formative analysis identified staged diagrams, branches, and condition flows as recurring ways of representing study logic. The authoring model should build on these representations, enabling researchers without systems backgrounds to specify and connect study components through a no-code interface. Reusable support for participant interfaces, model integration, deployment, and data collection should allow researchers to move from a study concept to an executable artifact without reconstructing the surrounding infrastructure from scratch; Lowering the barriers for conducting these studies allowing for a broader range of researchers to contribute. D4: Reproducibility Through Inspectable and Reusable Artifacts. The platform must preserve authored studies as explicit research artifacts rather than leaving critical details embedded in transient setup steps or one-off implementations (Cutler et al., 2026). Each study version must retain its procedure, task logic, prompts, model and context settings, and participant-facing materials in an executable form. Recorded interactions must remain linked to the corresponding version so that researchers can inspect both the authored condition and the experience participants encountered. Another researcher should be able to inspect, redeploy, adapt, and build on the artifact, with changes made explicit across versions. Preserving this continuity provides a shared frame of reference for reproducing studies and building cumulative knowledge. D5: Extensibility Across Tasks, Modalities, and Study Settings. The platform must remain extensible as the design space of conversational AI continues to expand. Studies may involve text chat, coding, multimodal interaction, voice, longer-running procedures, or community-facing workflows. New task types, modalities, and study settings should be supported through extensions that reuse the platformâs mechanisms for study execution and data collection, rather than requiring reimplementation of the surrounding system (Yao et al., 2026). These extensions must remain configurable within the study representation, allowing researchers to inspect, version, and reuse the resulting studies through the same authoring environment. Figure 2. The Gricea authoring interface, illustrating how researchers can assemble multi-stage study procedures (left) and independently configure agent pipelines and interface scaffolds for specific conversational tasks (right).The Gricea authoring interface, illustrating how researchers can assemble multi-stage study procedures (left) and independently configure agent pipelines and interface scaffolds for specific conversational tasks (right). Figure 3. Participant-facing interfaces configured in Gricea: conversational AI with integrated surveys (top left), real-time voice interaction alongside a driving task (top right), conversational coding with an editable code workspace (bottom left), and a participant interacting with two conversational agents (bottom right).Four screenshots show participant-facing interfaces. The top-left interface combines survey questions, a text conversation, and a response-rating item. The top-right interface places a driving simulation beside voice interaction controls. The bottom-left interface places a code editor with proposed changes beside an AI assistant. The bottom-right interface displays separate panels for Agent A and Agent B, with indicators for participant and agent turns. 5. Gricea: A Platform for Configurable Conversational AI Studies Gricea represents controlled conversational AI studies as configurable, executable research artifacts. Its architecture consists of three decoupled layers that operate on this shared representation: a researcher-facing visual authoring environment, a participant-facing execution runtime, and a publication layer for versioning, reuse, and community sharing. Each artifact specifies the experimental procedure and the interactive conditions participants encounter within it. This representation allows complex study conditions to be authored, executed, preserved, and reused within a unified framework rather than reconstructed through ad hoc infrastructure. Gricea operationalizes study designs as directed graphs. A Study Flow graph governs participant progression through the experiment, while nested Task Flow graphs specify the behavior of each participant-facing interactive condition. Publication serializes these graphs, their configurations, and their assets into an immutable study version that the runtime engine executes directly. 5.1. Core System Abstractions Gricea represents authored studies through two connected levels of executable graphs. Study Nodes form the procedural Study Flow, while Task Nodes form Task Flows embedded in its interactive stages. Logic Nodes provide branching, variable assignment, and condition control within these flows. Publication validates the authored definition and makes a specific Study Version available as a Published Study. Templates and Community Document Collections provide reusable components and grounding materials, while Participant Analytics summarizes the evidence produced during execution. Study Flow. The Study Flow graph encodes the procedural structure through which participants navigate. Its nodes represent participant-facing stages, including instructions, surveys, task entry, annotations, and completion, alongside control operations for branching, randomization, and within-subject ordering. During execution, the engine follows the authored transitions using assignment results and participant-specific state, advancing through control operations until it reaches a stage requiring participant interaction or study completion. Task Flow. The Task Flow graph encodes the runtime behavior of the interactive condition. Its nodes represent operations such as LLM inference calls, retrieval and context injection, real-time voice interaction, loops, participant input, and updates to interface components. The engine executes these operations according to the graph, coordinating model calls and interface changes with participant actions. When the task completes, control returns to the surrounding Study Flow, where information produced during the interaction can inform subsequent stages and measures. Logic and Condition Control. Condition control is represented through graph structure, node configurations, and variables. Branching, variable assignment, weighted randomization, and within-subject routing determine which conditions participants encounter and in what order. Assignments and condition orders are retained in the participantâs execution state. Variables connect assigned conditions to task configurations, making explicit how experimental assignment shapes the participant-facing interaction. Researchers can inspect the parameters deliberately varied across conditions alongside the settings held constant within the same authored study. 5.2. Researcher Authoring Environment Researchers author studies visually through no-code, node-based canvas editors for both Study Flow and Task Flow (Figure 2). The authoring model builds on the staged procedures, condition branches, and task entry points identified in our formative analysis (Section 3), allowing researchers to express experimental logic in the same graphs that drive execution. Researchers can construct the procedure and interactive condition within one environment while retaining separate control over each. At the study level, researchers add and connect instructions, elicitation stages, branching logic, randomization, within-subject blocks, and task entry points. At the task level, they configure each taskâs Task Flow and participant interface, including system prompts, retrieved context, model parameters, supported tool settings, interface layout, scaffolds, and modality-specific settings. Study-level variables connect procedural decisions to the task configuration that participants encounter. The platform separates iterative authoring from participant deployment. Researchers can run a participant-facing preview and revise an editable draft before publication validates and locks the authored graphs, prompts, configuration states, and runtime bindings into a published study version. The participant runtime executes that version while further development proceeds in new drafts, preserving the exact authored condition deployed to participants independently of subsequent changes. Researchers can inspect assignment and routing in Study Flow, examine their effects on interaction in Task Flow, and preview the participant experience. Publication and execution use this same definition, avoiding a separate translation into bespoke software. Preview complements structural validation by letting researchers check the represented study against their intended design. 5.3. Participant Runtime and Instrumentation The participant runtime combines browser-based rendering with server-side execution of the published study. It resolves the Study Flow for procedural progression, invokes the associated Task Flow upon task entry, and binds participant interactions to the authored graph logic. The browser renders the interface defined for the current stage or task, submits participant actions to the runtime, and applies the resulting updates. The published artifact therefore governs both how participants progress through the study and how the interactive condition responds to their actions. The runtime supports chat, web-based search tasks, voice interaction, humanâAI coding, annotation workflows, image generation, and split-view configurations. These tasks use shared mechanisms for execution, state management, and event collection, allowing researchers to vary the participant-facing experience while preserving a consistent connection between the study configuration and the resulting data. The instrumentation layer records timestamps, clicks, focus events, scroll depth, input timing, text-edit counts, and task-specific interaction traces according to the studyâs collection settings. Where enabled, additional capture includes voice recordings, masked browser-session replay, screen recordings, file-upload traces, and eye-tracking data. These records are associated with the participant session and its immutable Study Version, allowing researchers to interpret behavior in relation to the exact configured condition participants encountered. 5.4. Publication, Reuse, and Community Workflows Gricea treats reproducibility as an intrinsic property of the study artifact itself. Each published study is preserved as an immutable version containing its prompts, task parameters, graph structures, assignment logic, and participant-facing materials. These configurations remain inspectable after deployment, and researchers can distribute published studies through shareable links, making the executable study available alongside its written description. The publication layer supports reuse of both complete studies and their constituent resources. Researchers can publish Community Templates containing whole studies or selected nodes and flow segments, share Community Document Collections used for RAG grounding, and release de-identified dataset assets. Sharing these resources preserves not only the studyâs outputs, but also its executable design and the materials used to construct the conversational condition. A peer researcher can therefore inspect the underlying graph logic and configurations, experience the study live as a participant, and fork the artifact to run an independent replication. The fork provides an editable draft for changing conditions, tasks, or measures without altering the published source. A researcher extending the study can change a selected parameter while retaining the surrounding procedure and materials, then publish and deploy the revised artifact. The source and revised artifacts provide a shared frame of reference for examining what was preserved and what changed, allowing subsequent research to build directly on existing experimental designs. 5.5. Architecture and Extensibility Gricea separates its visual authoring client, execution runtime, and data and publication services so that new capabilities can be added without rebuilding the surrounding study infrastructure. Developers extend task behavior through new node types, each with configuration definitions and an execution handler. Participant-facing components similarly declare the properties researchers can configure and the interaction events they produce. These additions integrate with the existing Study Flow and Task Flow model. The architectural objective is to preserve a common study representation as the supported tasks and modalities expand. Extensions for richer multimodal interactions, longitudinal deployments, and community-facing workflows can reuse the same mechanisms for procedural control, publication, and instrumentation, keeping the resulting studies configurable, inspectable, and reusable. 6. Evaluation 6.1. Evaluation Rationale and Questions Gricea is intended to support the replication of existing conversational AI studies and the construction of new studies through a shared, executable representation. We evaluated whether the platform provides the features needed to implement published study designs beyond those included in our formative analysis, and whether researchers from diverse backgrounds can use those features to construct their own studies. Combining replication and usability studies follows prior work on toolkit evaluation, which examines what systems enables and itâs usability (Ledo et al., 2018; Olsen Jr, 2007). Our evaluation addresses four questions. RQ1: Can Gricea replicate conversational AI study designs beyond those included in our formative analysis? RQ2: Can researchers from diverse backgrounds implement their intended study designs with Gricea? RQ3: Can Gricea represent and execute a diverse range of conversational AI studies? RQ4: What frictions remain in translating research intent into an executable study? We first conducted a replication study of CUI 2026 papers outside our formative corpus to examine whether Gricea could represent and execute full study configurations or their conversational portions, addressing RQ1 and providing evidence of expressivity for RQ3 (Section 6.2). We then conducted a usability study in which researchers and AI practitioners authored studies addressing their own research questions, examining their ability to implement intended designs (RQ2), the range of studies they constructed (RQ3), and the frictions they encountered during authoring (RQ4). 6.2. RQ1: Replication of Published CUI 2026 Studies To evaluate Griceaâs support for published conversational AI studies, we examined 37 CUI 2026 full papers, of which 29 reported participant-facing human-subject studies. The full corpus is listed in Appendix A.2. These included live conversations, prerecorded conversational stimuli, expert annotation and rating, multimodal interfaces, group interaction, and embodied agents (Bieberstein et al., 2026; Tschopp et al., 2026; Wazzan et al., 2026; Hata et al., 2026; Reyes-Cruz et al., 2026). The research team extracted study details from the papers and supplementary materials, attempted reconstruction in Gricea, and recorded replication failures. We assessed replication success by whether Gricea provided the features needed to represent and execute the study procedures, participant-facing interfaces, and conversational task behavior. Complete replication covered the full study configuration, whereas partial replication covered the conversational portions that could be implemented when missing source materials or external dependencies prevented reproduction of the full study. Studies with full replication can still have missing details from studies such as missing participant facing instructions but they do not block representation of the study configurations and CAI tasks. We recorded unavailable information separately from requirements for external hardware or systems to distinguish reporting gaps and scope boundaries from the study features supported by Gricea. Replication outcome: Using Gricea, we successfully replicated full study configurations or their conversational portions from 27 of the 29 eligible papers: 10 completely and 17 partially. Gricea provided the features needed to represent and execute the replicated procedures, interfaces, and conversational tasks, covering prompt-based manipulations, model and stance comparisons, role-play conversations, fixed-media evaluations, repeated sessions, and speech-versus-typing tasks. Missing source materials and external dependencies limited complete replication, while unavailable interview and survey materials prevented replication of the remaining two studies. These results demonstrate that Gricea supports executable study designs beyond those used to inform its representation. Reporting Gaps and Their Impact on Replication: We found incomplete reporting or unavailable materials in 28 of the 29 eligible papers, although not all omissions prevented replication. The most common gaps were questionnaire items, revisions or participant-facing interview wording (21 papers), complete agent prompts, grounding inputs, or configuration rules (15), and stimulus or exercise materials (11), with overlap across categories. When unavailable information was necessary to implement the study configuration, it limited which portions could be replicated. Gricea External Dependencies and Scope Boundaries Fourteen papers involved external requirements in at least one phase: robots or wearables, voice-cloning pipelines, specialized applications, or in-person group coordination. These should generally remain external systems connected to Gricea and hence were only partially replicated. 6.3. Participants We recruited N=10N=10 participants from diverse disciplinary backgrounds and roles, including researchers, students, faculty, and industry practitioners. The participant summary is provided in the Table 3. Of the 10 participants 5 self-identified as Male and 5 self-identified as female. The median age of participants was in the range of 25-34. There were 4 PhDs, 2 Master students, 2 professionals, 1 professor and 1 undergraduate student in our sample. Participants varied in their prior experience with controlled studies and in the kinds of conversational AI questions they had previously explored or hoped to explore. This diversity was intentional since Gricea is meant to onboard researchers from diverse backgrounds by lowering the barriers to conduct user studies. 6.4. Study Design and Procedure Each session was designed to assess both onboarding and open-ended study authoring. The procedure drew on a common pattern in toolkit and platform evaluation: a structured task that helps participants develop the systemâs basic mental model, followed by an open-ended task that reveals how they apply the platform to questions that matter to them [8, 98]. Sessions consisted of four phases: pre-task interview, assisted authoring walkthrough, open-ended think-aloud authoring, and post-task reflection. The total study lasted 90 minutes and participants were paid 30$ through amazon gift cards. Throughout, we recorded the participant screen and audio after obtaining their informed consent. We began each session with a brief introduction followed by a pre-task interview on their background, prior experience with human subject studies, challenges encountered in running human subject studies, and research questions they were interested in the area of Conversational AI. In the assisted authoring walkthrough, participants implemented a fixed research question: How does the stance of an AI assistant on controversial issues affect usersâ perceived trust? During this phase, the facilitator provided guidance on the platform features, while participants retained control of the interface and performed the authoring themselves. In the open-ended phase, participants were asked to design a conversational AI study of their own choosing using Gricea. They were instructed to think aloud as they worked. After the authoring tasks, participants completed a post-task survey and took part in a semi-structured interview. These instruments focused on perceived usability, expressivity, reproducibility, likely time savings relative to current workflows, and adoption potential. The interview further probed where participants felt confident, where they felt stuck, and what forms of support would make the platform more useful in their own work. 6.5. Analysis Approach We analyzed the survey, transcripts and screen recordings using a combination of descriptive summaries and qualitative analysis. Completion, timing, hints, and breakdowns were summarized descriptively across participants. Think-aloud transcripts, facilitator notes, and interview responses were analyzed thematically to identify recurring patterns related to learnability, expressivity, reproducibility, and unmet support needs. 6.6. Findings 6.6.1. Overcoming the Prototyping and Reproducibility Bottleneck In the pre-task interviews, participants consistently described current conversational AI workflows as bespoke, costly, and highly unstable. P7 noted, âThereâs no standardized way of conducting these studies⊠these models keep changing over time.â P6 described building custom interfaces from scratch, noting, âI manually coded all of that,â which took âa few weeks.â For participants with limited programming experience, this overhead was prohibitive. P9 stated, âI cannot code so I never thought I would be able to do this on my own, I would have to just pay someone to build it out for me.â Furthermore, reproducibility emerged as a primary concern. P7 articulated this explicitly: âIf I want to release my system, it is not clear to me how my systems can stand over time and how will my study be replicated and built upon.â Following the authoring tasks, participants overwhelmingly viewed Gricea as a structural solution to these bottlenecks. P8 estimated that utilizing Gricea would save âat least 3 to 6 months of effortâ and roughly â$25,000 worth of money,â noting that building their protocol manually âwould have been a nightmare.â This qualitative enthusiasm was supported by the post-task survey, where participants indicated a strong likelihood to recommend Gricea to colleagues (M=6.8,SâE=0.13M=6.8,SE=0.13) and expressed high confidence that studies authored in Gricea could be reliably reproduced by other researchers (M=5.9,SâE=0.37M=5.9,SE=0.37). 6.6.2. Participants from diverse backgrounds were able to author studies Participants across disciplines successfully mapped their conceptual study designs onto Griceaâs dual-flow abstractions, reporting high overall ease of use (M=6.1,SâE=0.31M=6.1,SE=0.31). Several explicitly stated that the node-based architecture mirrored their internal cognitive models of experimental design. P1 noted that âall those component building blocks make sense to meâ and could be used âintuitivelyâ to build procedures. P3 found that âthe plug, play, click and edit pipeline is really intuitive,â preserving their âchain of thought,â while P4 highlighted that âthe most intuitive part is when you connect those together.â Three platform strengths consistently emerged as critical for supporting researchers from such diverse backgrounds. First, the visual canvas provided necessary architectural clarity; P8 noted, âThe visual interface is quite easy to develop the protocol⊠The visual overlay made a lot of difference.â Second, participants valued the unified consolidation of procedure and runtime. P6 appreciated having surveys, AI tasks, and deployment âcontained into one system,â contrasting it with fragmented legacy workflows. Third, the localized validation and preview mechanisms were highly praised. P7 appreciated âbeing able to preview it⊠and seeing if what I thought is what is actually happening,â while P3 praised âthe ease with which a study could be validated,â calling the feature âsuper cool.â These accounts show why inspection matters beyond usability: participants used the representation and preview to connect their intended design to the procedure and interaction the platform would execute. Participant Open-ended research question P01 Under what conditions and tasks do users rely on different agentic tool calls, such as web search? P02 How does serendipity differ across web search, generative-AI search, and retrieval-augmented generation? P03 How do proactive versus reactive coding agents shape participantsâ coding practices? P04 How does AI-assisted creative writing compare to human-assisted creative writing? P05 If people cannot distinguish AI-written text from human-written text, where does the perceived âAI penaltyâ in evaluation come from? P06 How do different personas of AI chatbots affect engineers in mechanical design tasks? P07 How does AI assistance in creativity tasks affect subsequent creativity without AI? P08 How does an LLM-augmented debate with a human moderator affect discussion and decision-making? P09 How do people fact-check information when using generative AI? P10 How do differences in model tier exacerbate the digital divide? Table 2. Open-ended research questions participants implemented as valid, runnable Gricea studies. These studies varied across procedures, interfaces, models, surveys, and outcomes, illustrating the range of research designs expressible through the shared study representation. 6.6.3. Gricea supported RQs across the Conversational AI Design Space The open-ended phase provided robust evidence of Griceaâs expressive capacity. Participants successfully generated and operationalized a highly diverse set of research questions, confirming they were able to express the exact study designs they had in mind (M=5.5,SâE=0.40M=5.5,SE=0.40). Implemented studies spanned the breadth of the conversational AI manipulation space. P7 investigated how AI assistance affects subsequent non-AI creativity. P5 explored the âAI penaltyâ by testing evaluations of text with and without AI-disclosure. P2 operationalized a study on serendipity across standard web search, generative search, and RAG architectures. Other participants configured studies on proactive versus reactive coding agents (P3), debates involving two voice agents and a human moderator (P8), conditions prompting users to utilize agentic tool-calls (P1), human fact-checking behaviors with generative AI (P9), and how different tiers of models exacerbate the digital divide (P10). Table 2 lists the research questions. Participants explicitly confirmed that Gricea easily accommodated these designs. P3 stated, âAll of it was able to be done, and I have been trying to think of studies that I cannot run but I cannot think of one yet.â P6 reported Gricea supported âeven more than I thought was possible,â and P8 summarized the platformâs capacity as âall of it and more.â Even for highly domain-specific protocolsâsuch as P6âs mechanical design collaboration or P8âs clinical protocolsâparticipants found the platform broadly expressive enough to capture their required experimental manipulations and compliant with medical standards for PII data. 6.6.4. High Degrees of Freedom and the Need for additional scaffolding While Griceaâs flexibility enabled diverse RQs, the high degrees of freedom introduced new methodological frictions. Breakdowns occurred most often for variables and branching: P4 found the randomization logic âconfusing for me for like the branching and the randomized [nodes],â while P1 struggled with âhow to dynamically insert a promptâ using variables. On a similar note, several participants experienced a conceptual boundary when transitioning from the macro Study Flow to the micro Task Flow. P6 noted that entering the Task Builder âbrought us to another workflow that was a little confusing.â P2 found the task canvas overly granular, stating that âit offers too much detailâ and âfor social scientists who donât need complex configurations they might to abstract away the complexities.â These frictions suggest that while Griceaâs core abstractions are powerful, researchers, especially those from non-tehcnical background, require stronger scaffolding to manage the flow of variables across nested nodes. P1 and P4 also noted a âhigh-learning curveâ. To resolve these issues, participants requested additional support for methodological mapping. P7 desired an AI feature to provide âa basic starting templateâ to help structure the flow. P1 articulated an important boundary condition for the platform: Gricea is highly effective when the experimental design is already concrete, but early-stage ideation still requires âcollaborative brainstorming with AI agents.â All participants however echoed that once they were used to the platform and ran a few studies they could see themselves getting over these barriers. 7. Discussion 7.1. Conversational AI Studies as Research Artifacts Studies on conversational AI should remain useful beyond the team that created them, but a publication alone cannot preserve the participant experience when procedures, interfaces, prompts, grounding, and API orchestration remain fragmented across code and configuration. Gricea makes the executable study available for inspection, reproduction, and extension, giving the research community a shared basis for building on an individual study. Participants valued versioning, inspectability, templates, and shareable links as ways to understand what another researcher built and ran. Preserving the authored condition makes methods easier to audit and interpret, extending the value of these features beyond individual authoring convenience. Our replication study of CUI 2026 papers revealed a gap between reporting a study for publication and preserving the information needed to reproduce its implementation (Section 6.2). Although these papers had passed peer review, questionnaire wording, agent prompts and grounding inputs, and stimulus or exercise materials were frequently incomplete or unavailable. When those details were necessary to implement a study, their absence prevented replication of the corresponding portions. A published account can therefore communicate a studyâs rationale, procedure, and findings while leaving another research team unable to reconstruct the participant experience. Reporting and reproducibility frameworks call for making methodological choices explicit (Feuerriegel et al., 2026; Feger et al., 2019; Aguilar et al., 2024). Our findings motivate preserving executable study artifacts as part of the methodological record alongside the paper, so that the community can inspect and reuse the implementation underlying the reported method. Gricea connects study materials and configurations to the executable study version, preserving them during authoring and execution rather than requiring researchers to reconstruct those connections after publication. 7.2. Related Research Questions Require Inspectable Study Configurations Our CUI reconstructions show that studies can address a shared research question while implementing substantially different experimental conditions. Studies examining uncertainty and reliance implemented that relationship through different tasks and procedures: one compared model and prompting conditions during travel-planning conversations, whereas another manipulated linguistic hedging and decision friction in financial advice with explicit allocation decisions (Abdrabou et al., 2026; Spillner et al., 2026). Beyond the underlying model, the studies differ in condition assignment, conversational progression, available actions, and the operationalization of reliance. Preserving the configured conditions in a shared representation allows subsequent researchers to inspect an earlier study and identify which parts to retain and which to vary when investigating a related hypothesis. Studies of conversational contact with an outgroup likewise address related questions through different interaction designs. Prior work has compared individual dialogue with a chatbot expressing a vegan perspective against a static essay, and facilitator-mediated group dialogue with an agent grounded in outgroup membersâ discussions against a document containing that material (Falkner and Kammerer, 2026; Hata et al., 2026). Although both investigate the consequences of interactive exposure to an outgroup perspective, they differ in who participates, whose perspective grounds the agent, how messages are composed, and how attitude change is measured. Preserving the study configurations in a shared artifact allows researchers to compare how related questions have been operationalized and design subsequent studies that retain or vary specific aspects of the interaction. 7.3. Study Flow and Task Flow as a Methodological Abstraction In Gricea, the distinction between Study Flow and Task Flow clarifies the methodological structure of conversational AI studies as well as organizing the platform. Researchers design procedures that assign participants to conditions, collect pre- and post-task measures, and manage staged progression, while also configuring an interactive runtime in which participants encounter a conversational condition that unfolds turn by turn. The participant experience therefore depends on both the study procedure and the behavior of the interactive task. Across the papers reviewed in our formative analysis, researchers often communicated study procedures through staged diagrams, branching depictions, and flowchart-like structures. The diagrams externalized how researchers conceptualized and communicated experimental logic, beyond summarizing a paper after the fact. Participants in our evaluation repeatedly described Griceaâs visual canvas as matching how they thought about study design, and several highlighted that connecting stages and previewing the resulting flow helped them reason about the experiment more concretely. Visual authoring therefore matters both for usability and for its alignment with how researchers already represent study logic in practice. The process of developing Gricea suggests that research platforms should organize their abstractions around the choices researchers need to distinguish and control. Study Flow separates assignment, ordering, and measurement from the operations that implement the interactive task, while Task Flow makes those operations inspectable without hiding their dependencies on the surrounding procedure. Separating study procedure from interactive task behavior allows a researcher to change conversational behavior while retaining the study sequence, or to change the experimental design while retaining the task. Because Study Flow and Task Flow also drive execution, the relationships a researcher inspects are the relationships used to implement the study. Block-based authoring makes the methodological choices explicit and reusable, extending work on reusable experimental components and methodological specifications (de Leeuw et al., 2023; Jun et al., 2019). Separating study procedure from interactive task behavior also provides a basis for extending research platforms without fragmenting their methods. A new task component should expose its configurable properties, required inputs, completion conditions, and recorded outputs so that it can participate in the existing procedure and data collection mechanisms. In Gricea, a modality-specific extension can reuse the surrounding study infrastructure while making its methodological consequences visible to the researcher. The component and its connections become part of the executable study representation, preserving the relationship between a configuration change and the participant experience it produces when another research team reuses or extends the study. The distinction between procedure and interactive task also applies to online studies of decision aids, interactive visualizations, educational interfaces, and collaborative tools, which combine assignment and measurement with a task that responds to participant actions (Chen et al., 2016; Almaatouq et al., 2021; Cutler et al., 2026). The representation could extend to these settings by replacing or adding task components while retaining the surrounding Study Flow and its connection to execution records. Different participant-facing activities could therefore share the same mechanisms for specifying, running, and preserving a study. 7.4. Lowering Technical Barriers Requires Methodological Scaffolding The cost of building controlled conversational AI studies can constrain who conducts them and which questions they explore. Lowering technical barriers broadens participation in producing research, not only in using a platform. Researchers from different disciplinary backgrounds implemented valid, runnable studies across the research questions in Table 2. Gricea supported participantsâ study designs through a common representation while reducing barriers for researchers without systems backgrounds. Sharing the resulting artifacts also allows other teams to inspect and adapt the studies without rebuilding their infrastructure. Our evaluation also showed where researchers need methodological scaffolding: participants requested help translating research intent into variables, branching, randomization, and validation, and previewing how those choices shape the participant experience. Related systems likewise show the need for guidance alongside greater authoring control (Arawjo et al., 2024; Yao et al., 2026). In Gricea, methodological scaffolding should help researchers connect a research question to executable study logic while retaining visibility and control over the resulting design. 7.5. Making Design-to-Implementation Fidelity Inspectable With a shared study representation, researchers can examine assignment branches and task configurations, preview the participant experience, and relate execution records to the published Study Version to check design-to-implementation fidelity. P7 described the check as asking whether âwhat I thought is what is actually happeningâ. Inspecting the configured study and its execution lets researchers examine the experimental choices used during execution rather than infer them from a separate implementation. Even a structurally valid study can diverge from its intended design: a valid within-subject flow may still misrepresent an intended between-subject comparison. Researcher review checks whether the implemented study matches the intended design, while publication preserves the inspected definition and version-linked records expose its execution. Connecting researcher review, the published study definition, and execution records gives the original team and subsequent researchers a common basis for examining fidelity. 7.6. Open-Science Workflows for Cumulative Conversational-AI Research When we scale across a sequence of studies, the community value of Gricea becomes obvious: one team may publish a conversational task and its experimental conditions; another may retain that procedure while changing an agent behavior, interface feature, or participant population to examine a related hypothesis. Preserving the source and revised artifacts makes the methodological relationship between studies inspectable, giving researchers a basis for explaining which findings concern the same configured interaction and which concern a deliberate extension, rather than treating each implementation as an unrelated starting point. Connected artifacts could help a research community trace evolving hypotheses, identify unresolved comparisons, and synthesize findings across related studies (Figure 1). Synthesizing findings across studies requires examining differences in populations, measures, and contexts as well as shared configurations. Gricea provides the mechanisms for publishing, inspecting, and reusing artifacts, while the contribution of shared artifacts to cumulative knowledge will depend on sustained community use. 7.7. AI-Assisted Research Through a Shared Study Representation AI-assisted research can build on shared study representations: Liu et al. (2026b) argue that agents need executable research artifacts to understand, reproduce, and extend scientific work. In Gricea, authoring agents could propose or modify Study Flow and Task Flow while researchers inspect changes to assignment, conversational behavior, measurement, and the resulting participant experience. Integrating coding assistance with Study Flow and Task Flow would preserve a common method for reviewing and reusing agent-authored studies. Supporting AI-native research requires distinguishing agents that help conduct research from agents whose behavior is part of the experiment. An authoring agent might retrieve a prior study, propose a controlled variation, or explain a change to its configuration. An agent participating within a study instead operates under the roles, information access, and interaction rules specified by that study, as humanâAI research platforms have begun to support (Qian et al., 2025; Yao et al., 2026). A shared representation can make both authoring changes and within-study agent behavior inspectable, while keeping changes to the research design separate from actions taken within an experimental condition. Agent-proposed changes could retain a link to the source study and expose which conditions they alter, allowing researchers to review the proposed revisions before accepted changes enter the representation used for deployment and data collection. Human and AI contributions could then be inspected and reused through a common research workflow. 8. Limitations and Future Work Gricea is intended for a broad research community. The authoring study demonstrates how researchers from varied backgrounds implemented conversational AI studies, but its sample of ten participants does not capture the full range of prospective researchers and research practices. The evaluation covers a subspace of the methodologies and configurations that Gricea is intended to support. The reconstructed online components do not capture the facilitated group activities of participatory co-design workshops, situated observations of people using their own devices and assistive technologies, or the physical behavior of embodied agents (Rekkas et al., 2026; Choi and Fischer, 2026; Reyes-Cruz et al., 2026). These methods involve participant activities, physical settings, and researcher involvement beyond the procedures and interactions represented in the reconstructed components. The published-study evaluation is also limited to CUI 2026. The corpus provides varied conversational interfaces and study procedures, but does not cover the full range of social-science experiments or the complex combinations of task behavior, modalities, and study logic that Gricea permits. Future work: Our evaluation highlights three directions for future work. First, to help researchers navigate Griceaâs high degrees of freedom, we plan to integrate an AI copilot that proposes initial study configurations from high-level research questions and modifications to existing artifacts. Proposals would remain within the shared study representation and subject to researcher inspection, validation, and preview before publication. Second, addressing feedback that Gricea is currently âbest used when there is [a] clear RQâ (P1, P10), we will add methodological scaffolding to help researchers brainstorm and identify potential confounders during early-stage ideation. Third, Gricea allows support for running AI participant simulations that could support pre-deployment checks of study paths, interaction behavior and pilot studies, further lowering the cost of conducting human subject studies. (Park et al., 2023). References Abbas et al. (2025) A. Abbas, C. Wohn, D. Hu, E. H. Rho, and S. W. Lee PITCH: designing agentic conversational support for planning and self-reflection. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI â25, New York, NY, USA. External Links: ISBN 9798400715273, Link, Document Cited by: §3.1, §3.3, §3.3, Table 1. Abdrabou et al. (2026) Y. Abdrabou, Y. Abdelrahman, E. Bozkir, Y. Mazen, F. Alt, and E. Kasneci Beyond Benchmarks: A User-Centric Framework for Evaluating Large Language Models. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â17. External Links: Link, Document Cited by: Table 4, §7.2. Aguilar et al. (2024) L. Aguilar, M. Gath-Morad, J. GrĂŒbel, J. Ermatinger, H. Zhao, S. Wehrli, R. W. Sumner, C. Zhang, D. Helbing, and C. Hölscher Experiments as Code and its application to VR studies in human-building interaction. Scientific Reports 14 (1), p. 9883. External Links: ISSN 2045-2322, Document, Link Cited by: §1, §2.2, §7.1. Almaatouq et al. (2021) A. Almaatouq, J. Becker, J. P. Houghton, N. Paton, D. J. Watts, and M. E. Whiting Empirica: a virtual lab for high-throughput macro-level experiments. Behavior Research Methods 53 (5), p. 2158â2171. External Links: ISSN 1554-3528, Link, Document Cited by: §2.1, §7.3. Amershi et al. (2019) S. Amershi, D. S. Weld, M. Vorvoreanu, A. Fourney, B. Nushi, P. Collisson, J. Suh, S. T. Iqbal, P. N. Bennett, K. I. Quinn, J. Teevan, R. Kikin-Gil, and E. Horvitz Guidelines for human-ai interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. External Links: Link Cited by: §1. Amiri Besheli and Jasim (2026) M. Amiri Besheli and M. Jasim SPARC: exploring interaction, sensemaking, and engagement in ai-augmented news reading. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â15. External Links: Link, Document Cited by: Table 4. Aoyama et al. (2026) S. Aoyama, K. Suganuma, H. Jiang, and S. Kasahara Designing a feedback loop between a human and their ai clones for science communication in museums. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â18. External Links: Link, Document Cited by: Table 4. Arawjo et al. (2024) I. Arawjo, C. Swoopes, P. Vaithilingam, M. Wattenberg, and E. L. Glassman ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI â24, p. 1â18. External Links: Link, Document Cited by: §2.3, §7.4. Avula et al. (2022) S. Avula, B. Choi, and J. Arguello The effects of system initiative during conversational collaborative search. Proc. ACM Hum.-Comput. Interact. 6 (CSCW1). External Links: Link, Document Cited by: §3.3. Baradari et al. (2025) D. Baradari, N. Kosmyna, O. Petrov, R. Kaplun, and P. Maes NeuroChat: a neuroadaptive ai chatbot for customizing learning experiences. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI â25, New York, NY, USA. External Links: ISBN 9798400715273, Link, Document Cited by: §3.1. Bieberstein et al. (2026) A. Bieberstein, B. L. Schnitzer, S. Gampe, and O. Korn Do prompt-level empathy instructions influence user experience? evidence from a controlled chatbot study. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, New York, NY, USA, p. 3:1â3:15. External Links: Document, Link Cited by: Table 4, §6.2. Caetano et al. (2025) A. Caetano, K. Verma, A. Taheri, R. Kumaran, Z. Chen, J. Chen, T. Höllerer, and M. Sra Agentic workflows for conversational human-ai interaction design. External Links: 2501.18002, Link Cited by: Table 1. Cai et al. (2023) W. Cai, Y. Jin, X. Zhao, and L. Chen âListen to music, listen to yourselfâ: design of a conversational agent to support self-awareness while listening to music. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI â23, New York, NY, USA. External Links: ISBN 9781450394215, Link, Document Cited by: Table 1. Cai et al. (2024) Y. Cai, S. Mao, W. Wu, Z. Wang, Y. Liang, T. Ge, C. Wu, W. You, T. Song, Y. Xia, J. Tien, N. Duan, and F. Wei Low-code LLM: Graphical User Interface over Large Language Models. External Links: 2304.08103, Link Cited by: §2.3. Chatterji et al. (2025) A. Chatterji, T. Cunningham, D. J. Deming, Z. Hitzig, C. Ong, C. Y. Shan, and K. Wadman How people use chatgpt. Technical report National Bureau of Economic Research. Cited by: §1. Chen et al. (2016) D. L. Chen, M. Schonger, and C. Wickens oTreeâAn open-source platform for laboratory, online, and field experiments. Journal of Behavioral and Experimental Finance 9, p. 88â97. External Links: ISSN 2214-6350, Link, Document Cited by: §2.1, §7.3. Chen et al. (2025) J. Chen, Y. Zhang, Y. Zhang, Y. Shao, and D. Yang Generative interfaces for language models. External Links: 2508.19227, Link Cited by: Table 1. Chen et al. (2024) L. Chen, M. Zaharia, and J. Zou How is chatgptâs behavior changing over time?. Harvard Data Science Review 6 (2). External Links: Link, Document Cited by: §3.4. Cheng et al. (2024) R. Cheng, T. Barik, A. Leung, F. Hohman, and J. Nichols BISCUIT: scaffolding llm-generated code with ephemeral uis in computational notebooks. In IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), External Links: Link Cited by: Table 1. Chi et al. (2025) P. Chi, S. Hu, L. Shi, T. Kraljic, J. Secor, T. Dong, I. Essa, and M. Cleron WatchWithMe: llm-based interactive guided watching of review videos. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI â25, p. 1â15. External Links: Link, Document Cited by: Table 1, Table 1, §4. Choi and Fischer (2026) S. Choi and J. E. Fischer âIâM BLIND, ChatGPTâ: Interactional Breakdown and Repair in LLM-based Conversational AI for Visually Impaired Users. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â14. External Links: Link, Document Cited by: Table 4, §8. Cooney et al. (2026) O. Cooney, S. J. Leonard, P. R. Peña, J. Hannah, and B. R. Cowan Exploring perceptions of robo-advisors for personal investing. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â12. External Links: Link, Document Cited by: Table 4. Cox et al. (2025) S. R. Cox, R. M. Jacobsen, and N. van Berkel The impact of a chatbotâs ephemerality-framing on self-disclosure perceptions. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI â25, New York, NY, USA. External Links: ISBN 9798400715273, Link, Document Cited by: §3.2, §3.2, Table 1. Cross et al. (2026) E. K. Cross, L. Santos, L. Denoue, S. Carter, N. S. Bravo, and K. Sieck ConvoDojo: structured llm-based sparring partners for difficult workplace conversations.. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â15. External Links: Link, Document Cited by: Table 4. Cutler et al. (2026) Z. Cutler, J. Wilburn, H. Shrestha, Y. Ding, B. Bollen, K. A. Nadib, T. He, A. McNutt, L. Harrison, and A. Lex ReVISit 2: A Full Experiment Life Cycle User Study Framework. IEEE Transactions on Visualization and Computer Graphics 32 (1), p. 13â23. External Links: ISSN 2160-9306, Link, Document Cited by: §2.2, §4, §7.3. Czech et al. (2025) E. Czech, E. Soubutts, I. Craddock, and A. A. OâKane Understanding the multimodal voice assistant as an informal and social care support tool in the uk. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI â25, p. 1â14. External Links: Link, Document Cited by: Table 1. Dam and Glomann (2026) N. Dam and L. Glomann AI interviews the interviewers: practitioner experience and evaluation of conversational ai interviewing. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â14. External Links: Link, Document Cited by: Table 4. de Leeuw et al. (2023) J. R. de Leeuw, R. A. Gilbert, and B. Luchterhandt jsPsych: Enabling an Open-Source Collaborative Ecosystem of Behavioral Experiments. Journal of Open Source Software 8 (85), p. 5351. External Links: ISSN 2475-9066, Link, Document Cited by: §2.1, §7.3. Desai et al. (2026) S. Desai, J. Chin, D. Wang, B. R. Cowan, and M. Twidale Toward metaphor-fluid conversation design for voice user interfaces. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â21. External Links: Link, Document Cited by: Table 4. Dhillon et al. (2024) P. S. Dhillon, S. Molaei, J. Li, M. Golub, S. Zheng, and L. P. Robert Shaping human-ai collaboration: varied scaffolding levels in co-writing with language models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI â24, New York, NY, USA. External Links: ISBN 9798400703300, Link, Document Cited by: §3.3, Table 1. Elwahsh et al. (2025) S. Elwahsh, N. Stern, A. Singh, and A. Ayobi Linguistic diversity and mental well-being: co-designing custom ai chatbots with multilingual mothers. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI â25, p. 1â17. External Links: Link, Document Cited by: Table 1. Falkner and Kammerer (2026) J. Falkner and Y. Kammerer More Interactivity, More Open-Mindedness? The Influence of AI-based Chatbot Interactivity on Attitudes Toward Vegans. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â11. External Links: Link, Document Cited by: Table 4, §7.2. Feger et al. (2019) S. S. Feger, S. Dallmeier-Tiessen, P. W. WoĆșniak, and A. Schmidt The role of hci in reproducible science: understanding, supporting and motivating core practices. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, p. 1â6. Cited by: §7.1. Feng et al. (2025) K. J. K. Feng, Q. V. Liao, Z. Xiao, J. W. Vaughan, A. X. Zhang, and D. W. McDonald Canvil: Designerly Adaptation for LLM-Powered User Experiences. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI â25, p. 1â22. External Links: Link, Document Cited by: §2.3. Feuerriegel et al. (2026) S. Feuerriegel, C. Barrie, M. J. Crockett, L. K. Globig, K. L. McLoughlin, D. Mirea, A. Spirling, D. Yang, T. Althoff, M. Antoniak, L. P. Argyle, A. Ashokkumar, M. Atari, H. Bailey, K. Bauer, U. Bhatt, Y. Chai, T. Chakraborty, Y. Chandra, H. Chen, H. DaumĂ© I, G. De Francisci Morales, M. Dehghani, D. Dillion, J. C. Eichstaedt, K. Forster, D. Geissler, K. Gray, T. L. Griffiths, J. Hartmann, O. P. Hauser, J. K. He, R. Hemrajani, F. Holzmeister, A. H. Hwang, T. Hu, A. A. Ivanova, N. Köbis, Y. Kyrychenko, H. Lakkaraju, J. Liu, A. Maarouf, S. Maier, L. Meincke, R. Mihalcea, B. Mittelstadt, S. M. Mohammad, M. Naaman, O. Netzer, A. Oh, D. C. Ong, F. Pierri, B. Plank, I. Rahwan, T. Rahwan, P. S. B. Rao, C. E. Robertson, D. M. Rothschild, M. J. Salganik, E. Schulz, C. Shah, Y. R. Shrestha, E. Shutova, A. A. Siegel, A. Simchon, H. Sun, M. Toetzke, J. J. Van Bavel, M. Vaccaro, J. W. Vaughan, E. Vayena, P. O. S. Vaz-de-Melo, B. Vecchione, A. Wang, R. West, R. Willer, D. U. Wulff, R. Zhang, S. Zhang, S. Rathje, and M. H. Ribeiro A reporting checklist for large language models in behavioural science. Nature Human Behaviour 10 (7), p. 1182â1186. External Links: ISSN 2397-3374, Link, Document Cited by: §2.2, §7.1. Flores-Saviaga et al. (2025) C. Flores-Saviaga, B. V. Hanrahan, K. Imteyaz, S. Clarke, and S. Savage* The impact of generative ai coding assistants on developers who are visually impaired. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI â25, p. 1â17. External Links: Link, Document Cited by: Table 1, Table 1. Foster and Deardorff (2017) E. D. Foster and A. Deardorff Open Science Framework (OSF). Journal of the Medical Library Association 105 (2). External Links: ISSN 1536-5050, Link, Document Cited by: §1, §2.2. Ge et al. (2024) X. Ge, C. Xu, D. Misaki, H. R. Markus, and J. L. Tsai How culture shapes what people want from ai. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI â24, p. 1â15. External Links: Link, Document Cited by: Table 1. Google Cloud (2026a) Google Cloud Dialogflow CX: Experiments. Note: Official documentation; updated September 3, 2026; accessed September 10, 2026 External Links: Link Cited by: §2.3. Google Cloud (2026b) Google Cloud Dialogflow CX: Playbooks. Note: Official documentation; updated September 3, 2026; accessed September 10, 2026 External Links: Link Cited by: §2.3. Google Cloud (2026c) Google Cloud Dialogflow CX: Versions and Environments. Note: Official documentation; updated September 3, 2026; accessed September 10, 2026 External Links: Link Cited by: §2.3. Hata et al. (2026) K. Hata, R. Chujo, R. Takamatsu, W. Xu, and Y. Baba GroupEnvoy: a conversational agent speaking for the outgroup to foster intergroup relations. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, New York, NY, USA, p. 31:1â31:18. External Links: Document, Link Cited by: Table 4, §6.2, §7.2. Hautasaari et al. (2026) A. Hautasaari, Y. Kibayashi, R. Chujo, Y. Hatada, and T. Naemura Butlerliezer: context- and receiver-aware appropriately deceptive auto-reply system based on egocentric video. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â15. External Links: Link, Document Cited by: Table 4. He and Lex (2026) T. He and A. Lex Integrating an LLM-Based Chatbot into a reVISit Study. Note: reVISit project tutorialPublished April 30, 2026; accessed September 7, 2026 External Links: Link Cited by: §2.2. He and Fussell (2026) W. P. He and S. R. Fussell XPLAIN: a proactive scaffold across speech processing stagesâsupporting non-native speakers in real-time ai-mediated turn-taking. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â17. External Links: Link, Document Cited by: Table 4. Iarygina et al. (2026) O. Iarygina, K. HornbĂŠk, and A. Mottelson On the Computational Reproducibility of Human-Computer Interaction. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, CHI â26, p. 1â16. External Links: Link, Document Cited by: §2.2. Imam and Sin (2026) H. Imam and J. Sin âMore like a personâs voiceâ: exploring the design of empathetic virtual agents for older adults. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â11. External Links: Link, Document Cited by: Table 4. Jakesch et al. (2023) M. Jakesch, A. Bhat, D. Buschek, L. Zalmanson, and M. Naaman Co-writing with opinionated language models affects usersâ views. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI â23, New York, NY, USA. External Links: ISBN 9781450394215, Link, Document Cited by: §1, §3.1, §3.1, §3.2, §3.3, Table 1, Table 1, §4. Jeong et al. (2025) D. Jeong, S. Shin, and J. Jeong Conversation progress guide: ui system for enhancing self-efficacy in conversational ai. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI â25, New York, NY, USA. External Links: ISBN 9798400713941, Link, Document Cited by: §3.3, Table 1. Jun et al. (2019) E. Jun, M. Daum, J. Roesch, S. Chasins, E. Berger, R. Just, and K. Reinecke Tea: A High-level Language and Runtime System for Automating Statistical Analysis. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, UIST â19, p. 591â603. External Links: Link, Document Cited by: §2.3, §7.3. Kirk et al. (2025) H. R. Kirk, I. Gabriel, C. Summerfield, B. Vidgen, and S. A. Hale Why humanâai relationships need socioaffective alignment. Humanities and Social Sciences Communications 12, p. 728. External Links: Document, Link Cited by: §1, §1. Kittur et al. (2008) A. Kittur, E. H. Chi, and B. Suh Crowdsourcing user studies with Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI â08, p. 453â456. External Links: Link, Document Cited by: §2.1. Kulkarni et al. (2026) A. M. Kulkarni, S. A. Brown, N. Rani, and S. Lynn Chu Towards pedagogy-grounded conversational ai tutors for interest-based learning. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â16. External Links: Link, Document Cited by: Table 4. Ledo et al. (2018) D. Ledo, S. Houben, J. Vermeulen, N. Marquardt, L. Oehlberg, and S. Greenberg Evaluation strategies for hci toolkit research. In Proceedings of the 2018 CHI conference on human factors in computing systems, p. 1â17. Cited by: §6.1. Lewis et al. (2020) P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. KĂŒttler, M. Lewis, W. Yih, T. RocktĂ€schel, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. nips 33, p. 9459â9474. External Links: Link Cited by: Table 1. Li et al. (2025) Y. Li, X. Shen, X. Yao, X. Ding, Y. Miao, R. Krishnan, and R. Padman Beyond single-turn: a survey on multi-turn interactions with large language models. External Links: 2504.04717, Link Cited by: Table 1. Li et al. (2024) Z. Li, C. Liang, J. Peng, and M. Yin The value, benefits, and concerns of generative ai-powered assistance in writing. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI â24, New York, NY, USA. External Links: ISBN 9798400703300, Link, Document Cited by: §3.1. Liang and Luo (2025) M. Liang and Y. Luo Exploring multi-llm collaboration to power conversational recommender system: a case study of dietary recommendation. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI â25, p. 1â6. External Links: Link, Document Cited by: Table 1. Liang et al. (2025) M. Liang, J. Wang, and Y. Luo SmartEats: investigating the effects of customizable conversational agent in dietary recommendations. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI â25, New York, NY, USA. External Links: ISBN 9798400715273, Link, Document Cited by: §3.1, §3.3, §4. Liao and Xiao (2025) Q. V. Liao and Z. Xiao Rethinking model evaluation as narrowing the socio-technical gap. External Links: 2306.03100, Link Cited by: §1. Liu et al. (2026a) C. Liu, Q. Zhou, X. Shen, X. B. Liu, T. Wu, and X. Chen Behavioral indicators of overreliance during interaction with conversational language models. arXiv preprint arXiv:2602.11567. Cited by: §3.1. Liu et al. (2026b) J. Liu, J. Pei, J. Huang, C. Si, A. Qu, X. Tang, R. Lu, L. Chen, X. Bai, H. Zheng, C. Chen, Z. Chen, H. Ye, Y. Fu, Z. He, Z. Jin, Z. Zhang, S. Sun, M. Harmon, J. D. Wang, J. Zeng, J. Sun, M. Wu, B. Zhou, C. You, S. Lu, Y. Qiu, F. Lai, Y. Yuan, Y. Li, J. Hong, R. Zhu, B. Chen, A. Pentland, A. Chen, M. Chowdhury, and Z. Zhang The Last Human-Written Paper: Agent-Native Research Artifacts. External Links: 2604.24658, Link Cited by: §2.2, §7.7. Liu and Law (2026) J. Liu and E. L. Law Beyond emotional mirroring: understanding affective alignment in artist-in-the-loop art chatbots. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â14. External Links: Link, Document Cited by: Table 4. Liu et al. (2026c) J. M. Liu, C. C. Weng, Y. Lin, P. Hsiao, and Y. T. Hou AI echoing in the backstage: private ai consulting may strengthens confidence and limits depolarization. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â17. External Links: Link, Document Cited by: Table 4. Liu et al. (2025) X. B. Liu, S. Fang, W. Shi, C. Wu, T. Igarashi, and X. â. Chen Proactive conversational agents with inner thoughts. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI â25, New York, NY, USA. External Links: ISBN 9798400713941, Link, Document Cited by: Table 1. Maslych et al. (2025) M. Maslych, M. Katebi, C. Lee, Y. Hmaiti, A. Ghasemaghaei, C. Pumarada, J. Palmer, E. Segarra Martinez, M. Emporio, W. Snipes, R. P. McMahan, and J. J. LaViola Jr. Mitigating response delays in free-form conversations with llm-powered intelligent virtual agents. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI â25, p. 1â15. External Links: Link, Document Cited by: Table 1. Mishra and Schneider (2025) A. Mishra and O. Schneider TacTalk: personalizing haptics through conversation. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI â25, p. 1â19. External Links: Link, Document Cited by: Table 1. Nguyen et al. (2025) H. V. Nguyen, G. Y. Yen, O. Shakir, H. Huynh, S. Gutierrez, J. A. Smith, S. Jimenez, S. E. Abdelgelil, and S. MacNeil Feedstack: layering structured representations over unstructured feedback to scaffold humanâai conversation. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI â25, p. 1â6. External Links: Link, Document Cited by: Table 1. Nobre et al. (2021) C. Nobre, D. Wootton, Z. Cutler, L. Harrison, H. Pfister, and A. Lex reVISit: Looking Under the Hood of Interactive Visualization Studies. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI â21, p. 1â13. External Links: Link, Document Cited by: §2.2. Olsen Jr (2007) D. R. Olsen Jr Evaluating user interface systems research. In Proceedings of the 20th annual ACM symposium on User interface software and technology, p. 251â258. Cited by: §6.1. Pang et al. (2025) R. Y. Pang, H. Schroeder, K. S. Smith, S. Barocas, Z. Xiao, E. Tseng, and D. Bragg Understanding the llm-ification of chi: unpacking the impact of llms at chi through a systematic literature review. (arXiv:2501.12557). Note: arXiv:2501.12557 [cs] External Links: Link, Document Cited by: Table 1. Park et al. (2023) J. S. Park, J. C. OâBrien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein Generative agents: interactive simulacra of human behavior. External Links: 2304.03442, Link Cited by: §8. Peer et al. (2017) E. Peer, L. Brandimarte, S. Samat, and A. Acquisti Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology 70, p. 153â163. External Links: ISSN 0022-1031, Link, Document Cited by: §2.1. Peer et al. (2022) E. Peer, D. Rothschild, A. Gordon, Z. Evernden, and E. Damer Data quality of platforms and panels for online behavioral research. Behavior Research Methods 54 (4), p. 1643â1662. External Links: ISSN 1554-3528, Document, Link Cited by: §2.1. Penney et al. (2025) J. Penney, P. Acharya, P. Hilbert, P. Parekh, A. Sarma, I. Steinmacher, and M. A. Gerosa Outcomes, perceptions, and interaction strategies of novice programmers studying with chatgpt. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI â25, p. 1â15. External Links: Link, Document Cited by: Table 1. Qian et al. (2025) C. Qian, V. Tsai, M. Behr, N. Hussein, L. Laugier, N. Thain, and L. Dixon Deliberate Lab: A Platform for Real-Time Human-AI Social Experiments. External Links: 2510.13011, Link Cited by: §2.1, §3.4, §3.4, §3.4, Table 1, Table 1, §4, §7.7. Rathod M.S. (2024) G. Rathod M.S. Information diet: understanding its role in a digital age with statistical insights. International Journal of Research in Library Science 10, p. 67â77. External Links: Document Cited by: §1, Table 1. Rathore et al. (2026) N. Rathore, T. Billakanti, J. Ceha, and J. Henderson A comparison of speech and typing input for creative generative ai tasks. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â10. External Links: Link, Document Cited by: Table 4. Reinecke and Gajos (2015) K. Reinecke and K. Z. Gajos LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, CSCW â15, p. 1364â1378. External Links: Link, Document Cited by: §2.1. Rekkas et al. (2026) P. Rekkas, K. Verbert, and V. Vanden Abeele Design and Evaluation of ChatBlend: A Framework and Card Deck for the Human-Centered Design of Mental Health Chatbots in Blended Care. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â21. External Links: Link, Document Cited by: Table 4, §8. Reyes-Cruz et al. (2026) G. Reyes-Cruz, M. Romeo, D. J. Price, M. P. Aylett, C. Greenhalgh, and J. E. Fischer Embodied active listening: how non-verbal backchannel behaviour influences trust and perception of a social robot. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, New York, NY, USA, p. 29:1â29:11. External Links: Document, Link Cited by: Table 4, §6.2, §8. Salous et al. (2026) M. Salous, M. P. WoĆșniak, T. von Reeken, M. Kramer, W. Heuten, S. Boll, and L. Abdenebaoui Beyond Captions: Shaping Imagery Models for Blind or Visually Impaired People During Interacting with AI-powered Conversational Visual Assistant. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â15. External Links: Link, Document Cited by: Table 4. Sanchez et al. (2026) A. Sanchez, J. Zhong, A. Deligianni, A. Ross, and S. King When text-to-speech speaks in your voice: a study on public perception. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â17. External Links: Link, Document Cited by: Table 4. Seymour et al. (2026) W. Seymour, A. D. G. Jenkins, M. Cote, and J. Such Beliefs and misconceptions around integrated conversational ai. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â10. External Links: Link, Document Cited by: Table 4. Sharma et al. (2024) N. Sharma, Q. V. Liao, and Z. Xiao Generative echo chamber? effect of llm-powered search systems on diverse information seeking. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI â24, New York, NY, USA. External Links: ISBN 9798400703300, Link, Document Cited by: §1, §3.1, §3.2, §3.3, Table 1. Sharma et al. (2025) N. Sharma, K. Murray, and Z. Xiao Faux polyglot: a study on information disparity in multilingual large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 8090â8107. External Links: Link, Document Cited by: §1, §3.1, Table 1. Sharma et al. (2026) N. Sharma, Z. Zhang, D. Lee, N. Krishnan, G. Ren, Z. Xiao, and Y. Li Feedback by design: understanding and overcoming user feedback barriers in conversational agents. arXiv preprint arXiv:2602.01405. Cited by: §1, §3.1. Sheshadri et al. (2026) S. Sheshadri, M. B. Pillai, R. Kumar, P. P. M. Nair, A. Gupta, M. Tm, S. S. Nair, B. M. Menon, U. Radhakrishnan, and B. Rao R Sruthi: what becomes sayable through peer-mode reflective scaffolding in fieldwork communication. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â15. External Links: Link, Document Cited by: Table 4. Shi et al. (2026) Y. Shi, Q. Xiao, Q. Hu, H. Shen, and H. Shen The siren song of llms: how users perceive and respond to dark patterns in large language models. External Links: 2509.10830, Link Cited by: §1, §3.1. Spillner et al. (2026) L. Spillner, J. Rockstroh, N. Wenig, N. Zargham, L. Kiefner, R. Porzel, and R. Malaka Linguistic Uncertainty Markers for Trust Calibration in AI-Assisted Decision-Making. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â15. External Links: Link, Document Cited by: Table 4, §7.2. Steeds et al. (2026) M. Steeds, K. Seaborn, T. Fujii, O. Cooney, and I. Gessinger Is chatgpt gender-neutral? implicit stereotyping persists over time and with experience. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â16. External Links: Link, Document Cited by: Table 4. Tschopp et al. (2026) M. V. Tschopp, C. Chen, M. Wischnewski, M. Gieselmann, J. Schöning, S. S. Sundar, and K. Sassenberg Does humanizing chatbots promote or hinder voice shopping?. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, New York, NY, USA, p. 28:1â28:14. External Links: Document, Link Cited by: Table 4, §6.2. Valcamonica et al. (2026) G. Valcamonica, G. Caleffi, F. Piferi, M. Tagliani, M. Vender, D. D. Delfitto, and F. Garzotto LLM-based conversational agents for dyslexia treatment: From chatbots to structured tutors. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, p. 1â12. External Links: Link, Document Cited by: Table 4. Wazzan et al. (2026) A. Wazzan, I. Qaiser, S. MacNeil, and R. Souvenir Comparing intent communication modes for instruction-based image editing. In Proceedings of the 8th ACM Conference on Conversational User Interfaces, CUI â26, New York, NY, USA, p. 17:1â17:15. External Links: Document, Link Cited by: Table 4, §6.2. Wu et al. (2022) T. Wu, E. Jiang, A. Donsbach, J. Gray, A. Molina, M. Terry, and C. J. Cai PromptChainer: Chaining Large Language Model Prompts through Visual Programming. In CHI Conference on Human Factors in Computing Systems Extended Abstracts, CHI â22, p. 1â10. External Links: Link, Document Cited by: §2.3. Yan et al. (2025) Z. Yan, J. E. Fischer, and J. Clos Remembering things makes chatbots sound smarter, but less trustworthy - a pilot study. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI â25, p. 1â8. External Links: Link, Document Cited by: Table 1. Yao et al. (2026) B. Yao, J. Chen, C. Chen, A. Wang, T. J. Li, and D. Wang Through the Lens of Human-Human Collaboration: A Configurable Research Platform for Exploring Human-Agent Collaboration. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, CHI â26, p. 1â30. External Links: Link, Document Cited by: §2.3, §3.4, §3.4, §3.4, Table 1, §4, §7.4, §7.7. Zamfirescu-Pereira et al. (2023) J.D. Zamfirescu-Pereira, R. Y. Wong, B. Hartmann, and Q. Yang Why johnny canât prompt: how non-ai experts try (and fail) to design llm prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI â23, New York, NY, USA. External Links: ISBN 9781450394215, Link, Document Cited by: §1. Zhang et al. (2024) L. Zhang, Z. Ji, and B. Chen CREW: Facilitating Human-AI Teaming Research. Transactions on Machine Learning Research. External Links: ISSN 2835-8856, Link Cited by: §2.3. Zhang et al. (2025) Z. Zhang, R. A. Rossi, B. Kveton, Y. Shao, D. Yang, H. Zamani, F. Dernoncourt, J. Barrow, T. Yu, S. Kim, R. Zhang, J. Gu, T. Derr, H. Chen, J. Wu, X. Chen, Z. Wang, S. Mitra, N. Lipka, N. Ahmed, and Y. Wang Personalization of large language models: a survey. External Links: 2411.00027, Link Cited by: §1, §3.1. Appendix A Appendix A.1. Participant Details The participant summary is provided in Table 3. ID Role Conducted studies? Study count Primary research interests / intended study P01 Researcher Yes 5â10 Capability boundary of conversational AI; what tasks can and cannot be automated; humanâhuman / humanâagent boundaries. P02 Master Student No N/A Serendipity in search; comparing web search, GenAI search, and RAG search; detailed search-session logging. P03 Master Student No N.A. Legal AI and hallucinated precedent; coding agents; mental-health advice and human influence. P04 PhD Yes 1â5 Social consequences of conversational agents; tutor-like interaction; transfer from humanâAI to humanâhuman interaction. P05 PhD Yes 1â5 AI penalty: how judgments change when people learn a text was AI-generated versus human-generated. P06 PhD Yes 1â5 Utility of AI systems in design / mechanical collaboration; AI personas in design tasks. P07 PhD Yes 1â5 Long-term creativity after AI assistance; comparing no-AI, AI-guidance, and answer-like AI support. P08 Professor Yes 10+ Surgeon knowledge assessment; using LLMs to identify knowledge deficits and support dialogue around experiential expertise. P09 Professional Yes 10+ How users use physical devices related to healthcare; P10 Undergraduate Student No N/A N/A Table 3. Completed participant records used in the analysis (n=10n=10 usable completed records in the attached JSON export). Participants spanned multiple roles and brought diverse conversational-AI study ideas to the authoring task. A.2. CUI 2026 Replication Corpus Table 4 lists all 29 eligible papers included in the published-study replication evaluation (Section 6.2). Table 4. The complete corpus of 29 eligible CUI 2026 papers that we attempted to recreate. Reference Paper (Bieberstein et al., 2026) Do Prompt-Level Empathy Instructions Influence User Experience? Evidence From A Controlled Chatbot Study (Liu et al., 2026c) AI Echoing in the Backstage: Private AI Consulting May Strengthens Confidence and Limits Depolarization (Abdrabou et al., 2026) Beyond Benchmarks: A User-Centric Framework for Evaluating Large Language Models (Spillner et al., 2026) Linguistic Uncertainty Markers for Trust Calibration in AI-Assisted Decision-Making (Seymour et al., 2026) Beliefs and Misconceptions around Integrated Conversational AI (Rekkas et al., 2026) Design and Evaluation of ChatBlend: A Framework and Card Deck for the Human-Centered Design of Mental Health Chatbots in Blended Care (Imam and Sin, 2026) âMore Like a Personâs Voiceâ: Exploring the Design of Empathetic Virtual Agents for Older Adults (Salous et al., 2026) Beyond Captions: Shaping Imagery Models for Blind or Visually Impaired People During Interacting with AI-powered Conversational Visual Assistant (Choi and Fischer, 2026) âIâM BLIND, ChatGPTâ: Interactional Breakdown and Repair in LLM-based Conversational AI for Visually Impaired Users (Valcamonica et al., 2026) LLM-based conversational agents for dyslexia treatment: From chatbots to structured tutors (Wazzan et al., 2026) Comparing Intent Communication Modes for Instruction-based Image Editing (Hautasaari et al., 2026) Butlerliezer: Context- and Receiver-Aware Appropriately Deceptive Auto-Reply System based on Egocentric Video (Desai et al., 2026) Toward Metaphor-Fluid Conversation Design for Voice User Interfaces (Cross et al., 2026) ConvoDojo: Structured LLM-based Sparring Partners for Difficult Workplace Conversations (Dam and Glomann, 2026) AI Interviews the Interviewers: Practitioner Experience and Evaluation of Conversational AI Interviewing (Sheshadri et al., 2026) Sruthi: What Becomes Sayable Through Peer-Mode Reflective Scaffolding in Fieldwork Communication (Amiri Besheli and Jasim, 2026) SPARC: Exploring Interaction, Sensemaking, and Engagement in AI-Augmented News Reading (Rathore et al., 2026) A Comparison of Speech and Typing Input for Creative Generative AI Tasks (He and Fussell, 2026) XPLAIN: A Proactive Scaffold Across Speech Processing StagesâSupporting Non-Native Speakers in Real-Time AI-Mediated Turn-Taking (Tschopp et al., 2026) Does Humanizing Chatbots Promote or Hinder Voice Shopping? (Reyes-Cruz et al., 2026) Embodied Active Listening: How Non-Verbal Backchannel Behaviour Influences Trust and Perception of a Social Robot (Cooney et al., 2026) Exploring Perceptions of Robo-advisors for Personal Investing (Hata et al., 2026) GroupEnvoy: A Conversational Agent Speaking for the Outgroup to Foster Intergroup Relations (Liu and Law, 2026) Beyond Emotional Mirroring: Understanding Affective Alignment in Artist-in-the-Loop Art Chatbots (Kulkarni et al., 2026) Towards Pedagogy-Grounded Conversational AI Tutors for Interest-Based Learning (Falkner and Kammerer, 2026) More Interactivity, More Open-Mindedness? The Influence of AI-based Chatbot Interactivity on Attitudes Toward Vegans (Steeds et al., 2026) Is ChatGPT Gender-Neutral? Implicit Stereotyping Persists Over Time and With Experience (Sanchez et al., 2026) When Text-to-Speech Speaks in Your Voice: A Study on Public Perception (Aoyama et al., 2026) Designing a Feedback Loop Between a Human and Their AI Clones for Science Communication in Museums