Paper deep dive
MIVAIS: A Study Environment for Multi-Agent Mixed-Initiative Visual Analytics Applications
Tobias Stähle, Simon Schneider, Rita Sevastjanova, Mennatallah El-Assady
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/7/2026, 4:14:19 AM
Summary
The paper introduces MIVAIS, a dual-layered research platform for Multi-Agent Mixed-Initiative Visual Analytics (VA). It consists of a computational Infrastructure that standardizes human-software agent interaction, state synchronization, and communication, and a declarative Study Environment that automatically logs multimodal telemetry (state, screen, audio, sensors) for rigorous evaluation. The framework is validated by replicating three state-of-the-art systems (Podium, Voyager 2, ProactiveVA) and evaluated through expert case studies, demonstrating its ability to lower the barrier for prototyping and evaluating intelligent, co-adaptive interfaces.
Entities (10)
Relation Signals (8)
MIVAIS → hasauthor → Tobias Stähle
confidence 95% · Tobias Stähle email: tobias.staehle@inf.ethz.ch
MIVAIS → isdevelopedby → ETH Zürich
confidence 95% · Affiliation: ETH Zürich, Zürich, Switzerland
MIVAIS → replicates → ProactiveVA
confidence 90% · We technically validate our infrastructure by replicating three state-of-the-art systems (Podium, Voyager 2, and ProactiveVA).
MIVAIS → replicates → Podium
confidence 90% · We technically validate our infrastructure by replicating three state-of-the-art systems (Podium, Voyager 2, and ProactiveVA).
MIVAIS → replicates → Voyager 2
confidence 90% · We technically validate our infrastructure by replicating three state-of-the-art systems (Podium, Voyager 2, and ProactiveVA).
MIVAIS → supports → Mixed-Initiative Visual Analytics
confidence 90% · MIVAIS: A Study Environment for Multi-Agent Mixed-Initiative Visual Analytics Applications
MIVAIS → containscomponent → World State
confidence 85% · Human and software agents interact with the application’s World State and each other, strictly mediated by a central Permission Guard.
MIVAIS → containscomponent → Permission Guard
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mixed-initiative Visual Analytics (VA) systems empower human users by interleaving human intuition with software agents and their machine intelligence. However, the development and rigorous evaluation of such systems remain constrained by engineering overhead. Developers must, e.g., implement complex, low-level state synchronization to manage asynchronous agent behaviors, while researchers struggle to capture the multimodal provenance required to study and evaluate human-AI collaboration. We present MIVAIS, a dual-layered research platform designed to abstract the structural complexities of mixed-initiative VA. First, it contributes a computational Infrastructure that standardizes human-software agent interaction, state synchronization, and communication between the agents. Second, it provides a declarative Study Environment that automatically logs multimodal human-AI telemetry - including application/system state, screen capture, audio, and additional sensor data - enabling seamless, in-situ user studies and post-session analysis. We technically validate our infrastructure by replicating three state-of-the-art systems (Podium, Voyager 2, and ProactiveVA). Furthermore, we evaluate the framework's expressiveness and efficiency through expert case studies with HCI and VA researchers, demonstrating how MIVAIS effectively lowers the barrier to prototyping and evaluating intelligent, co-adaptive interfaces.
Tags
Links
- Source: https://arxiv.org/abs/2609.04983v1
- Canonical: https://arxiv.org/abs/2609.04983v1
Trouble viewing inline? Open PDF directly →
Full Text
123,644 characters extracted from source content.
Expand or collapse full text
_set:Ne kwColorBox _set:Ne _set:Ne _set:Ne MIVAIS: A Study Environment for Multi-Agent Mixed-Initiative Visual Analytics Applications CCS: Human-centered computing Visual analyticsCCS: Computing methodologies Multi-agent systemsCCS: Human-centered computing User studiesCCS: Human-centered computing Empirical studies in HCICCS: Human-centered computing Interactive systems and tools Tobias Stähle email: tobias.staehle@inf.ethz.ch Affiliation: ETH Zürich, Zürich, Switzerland , Simon Schneider email: schneiders@student.ethz.ch Affiliation: ETH Zürich, Zürich, Switzerland , Rita Sevastjanova email: rita.sevastjanova@inf.ethz.ch Affiliation: ETH Zürich, Zürich, Switzerland and Mennatallah El-Assady email: menna.elassady@inf.ethz.ch Affiliation: ETH Zürich, Zürich, Switzerland Figure 1. Overview of the MIVAIS framework. The MIVAIS Infrastructure (left) supports the implementation of modular, mixed-initiative Visual Analytics (VA) systems where software and human agents collaborate. The MIVAIS Study Environment (right) enables researchers to easily configure experiments, collect multi-modal data during participant sessions, and perform comprehensive post-analysis. Overview of the . The (left) supports the implementation of modular, mixed-initiative Visual Analytics (VA) systems where software and human agents collaborate. The Environment (right) enables researchers to easily configure experiments, collect multi-modal data during participant sessions, and perform comprehensive post-analysis. Abstract. Mixed-initiative Visual Analytics (VA) systems empower human users by interleaving human intuition with software agents and their machine intelligence. However, the development and rigorous evaluation of such systems remain constrained by engineering overhead. Developers must, e.g., implement complex, low-level state synchronization to manage asynchronous agent behaviors, while researchers struggle to capture the multimodal provenance required to study and evaluate human-AI collaboration. We present MIVAIS, a dual-layered research platform designed to abstract the structural complexities of mixed-initiative VA. First, it contributes a computational Infrastructure that standardizes human-software agent interaction, state synchronization, and communication between the agents. Second, it provides a declarative Study Environment that automatically logs multimodal human-AI telemetry — including application/system state, screen capture, audio, and additional sensor data — enabling seamless, in-situ user studies and post-session analysis. We technically validate our infrastructure by replicating three state-of-the-art systems (Podium, Voyager 2, and ProactiveVA). Furthermore, we evaluate the framework’s expressiveness and efficiency through expert case studies with HCI and VA researchers, demonstrating how MIVAIS effectively lowers the barrier to prototyping and evaluating intelligent, co-adaptive interfaces. 1. Introduction The integration of (proactive) software agents into Visual Analytics (VA) has shifted the paradigm of data analytics from human-driven querying and exploration to human-AI collaboration. In these multi-agent setups, both humans and software agents can have equal agency. In mixed-initiative VA systems, both sides share analytical tasks such as identifying data insights (Dhanoa et al., 2025), generating forecasts (Sivaraman et al., 2025), recommending visualizations (Wu et al., 2022), and steering underlying models (Sperrle et al., 2021b; Endert et al., 2014), while taking into account the intents of other stakeholders (Horvitz, 1999). Mixed-initiative and multi-agent VA systems consist of data visualizations, interface interactions, and communication among agents. Rapid progress in agent capabilities is driving an increase in the complexity of the architectures required to support their interactions. Thus, system developers must ensure continuous synchronization of a highly dynamic interface with asynchronous agents and resolve conflicts when human and software agents attempt to update the application state simultaneously. While theoretical taxonomies on human-AI collaboration and intelligent agents in VA are established (Monadjemi et al., 2023; Stähle et al., 2025), there is a distinct lack of structural toolkits that abstract these engineering complexities. Consequently, Human-Computer Interaction (HCI) researchers spend a disproportionate amount of time building bespoke middleware rather than investigating the actual mixed-initiative and intelligent user interface phenomena, such as user trust, cognitive load, or multi-agent interplay. Recently, in the visualization community, there has been an effort to develop evaluation frameworks that systematize the research evaluation process for visualization research, e.g., reVISit (Ding et al., 2023; Cutler et al., 2026). However, the existing evaluation systems are tailored for single-user sessions with isolated visual stimuli and cannot capture the intricate dynamics of mixed-initiative collaboration. In multi-agent setups, human and AI agents continuously influence each other’s behavior, introducing additional analytical and cognitive burdens that must be explicitly measured. Evaluation requires tracking of inter-agent communication to capture dynamic agency distribution and handovers, user trust, and co-adaptiveness, alongside agent behavior and feedback loops. This tracking is essential because participants’ cognitive load stems not only from processing the underlying data but also from managing interactions with proactive agents. To effectively assess this multi-agent interplay, researchers must go beyond standard HCI logging mechanisms and capture synchronized, multimodal provenance. This includes: underlying application state to understand the exact context of collaborative actions; provenance logs of all interacting agents to identify correlations and behavioral patterns; the (visual) stimulus of each agent to reveal the specific information that triggered decisions; the human agents verbalized reasoning (think-aloud audio) that exposes decision-making processes, trust, and mental models; and inter-agent communication logs that capture the actual dialogue and task delegation between agents. Capturing additional biometric sensor data can further provide insights into cognitive load and stress – a capability that current study frameworks like reVISit do not support. To address this research gap, we introduce MIVAIS, a comprehensive research framework that streamlines the prototyping and evaluation of mixed-initiative VA applications. Bridging the gap between conceptual design, software realization, and conducting studies, MIVAIS operationalizes theoretical frameworks on intelligent agents and mixed-initiative VA human-AI collaboration into a pragmatic, research-centric toolkit. Our contribution is threefold: (1) A Mixed-Initiative Infrastructure Model: A scalable software toolkit that allows structured implementation and configuration of agents and provides for high-level APIs for VA environment observation, interaction, state synchronization, communication channels among the agents, and action provenance, abstracting the low-level complexities of mixed-initiative VA systems. (2) An Integrated Study Environment: Along the Infrastructure Model, we provide a multimodal logging architecture that allows researchers to seamlessly configure and deploy user studies. By automatically logging all agents’ observations, interactions, and communications and synchronizing these states with screen and audio captures, the environment enables deep post-hoc analysis of human-AI collaborative behaviors in mixed-initiative VA systems. (3) Demonstration and Evaluation: We demonstrate the capabilities of MIVAIS by replicating state-of-the-art systems (Podium (Wall et al., 2018), Voyager 2 (Wongsuphasawat et al., 2017), and ProactiveVA (Zhao et al., 2026)). In addition, we assess its expressiveness through technical dimensions of programming systems. We also evaluate the effectiveness of MIVAIS through expert case studies with 3 researchers showing how HCI and VA researchers and developers use MIVAIS to effectively implement and evaluate novel mixed-initiative VA interfaces. MIVAIS is available as open-source software at : github.com/ETH-IVIA-Lab/MIVAIS to support the research community with a modular, open-source foundation for future human-AI collaboration research. 2. Background & Related Work 2.1. Mixed-Initiative in Visual Analytics Systems Research in HCI explores how humans and computers can tackle complex problems (Dellermann et al., 2019; Xu and Gao, 2024), make informed decisions (Lee et al., 2021; Reverberi et al., 2022; Schemmer et al., 2023), and learn how data visualization and interactive exploration help humans derive insights (Keim et al., 2008). The introduction of mixed-initiative interfaces (Horvitz, 1999) has further advanced the domains of decision-making (Jiang and Odom, 2018; Ju et al., 2022) and visual analytics (Chen et al., 2025; Cook et al., 2015; Sperrle et al., 2021c; Zhao et al., 2025a; Zhao et al., 2025b). Several works by Gotz et al. (2006), Cook et al. (2015), Liu et al. (2007), and Wen et al. (2007), have established the mixed-initiative concept into VA applications, introducing software agents that support humans in navigating complex analytical tasks. Since then, researchers and developers have implemented a wide range of VA applications (e.g., (Cook et al., 2015; El-Assady et al., 2019; Wall et al., 2018; Wongsuphasawat et al., 2016)) and many of these applications aim to generally balance automation with human steering. Nevertheless, all of these applications are implemented differently and differ in their structure and modularization. Further, many works do not provide their research prototypes or applications openly accessible to the research community, making it hard for other researchers to replicate system behavior and studies, and most publications do not provide enough details for substantial replication (Fekete et al., 2020; Cockburn et al., 2020). 2.2. Intelligent Agents and Human-AI Collaboration While the literature considers both human and AI agents as actors in mixed-initiative VA systems (Dhanoa et al., 2025; Monadjemi et al., 2023; Monadjemi et al., 2025), the implementation and operational dynamics of these intelligent agents vary widely. Recent enhancements in Artificial Intelligence (AI) enable transforming these agents from passive to proactive actors, e.g., (Zhao et al., 2026; Huang and Narechania, 2026). To identify different agent roles, behaviors, and agency distribution, work by Monadjemi et al. (2023), and Dhanoa et al. (2025) make use of an agent-based conceptual model for mixed-initiative VA. Holter and El-Assady (2024) note that the agent-centric view allows for capturing the complexity of mixed-initiative VA. Further, Stähle et al. (2025) characterize intelligent agents’ logic and behavior in mixed-initiative VA through six dimensions (Configuration + Logic, World Model, Observations, Communication, Actions, and Infrastructure) by decomposing mixed-initiative VA systems into agents, an infrastructure layer, and the VA environment. The agent-centric view (Monadjemi et al., 2023) and the decomposition (Stähle et al., 2025) inspired the conceptual architecture of MIVAIS. 2.3. Mixed-Initiative Studies and Study Environments When conducting studies for generic VA, there exist approaches that allow for easier interactive data analysis (Cutler et al., 2020; Blascheck et al., 2016). VA2 by Blascheck et al. (2016) focuses on identifying common study participants’ interaction strategies after conducting a human subject study with a VA interface. To further equip VA systems with interaction tracking to capture user intent and provide provenance logging, Cutler et al. (2020) provide the software library Trrack. ReVISit (Ding et al., 2023; Cutler et al., 2026) provides a study environment that allows for defining web-based visualization studies through a declarative format. The platform has been used for testing, e.g., users’ visualization literacy (Nobre et al., 2021). Further, researchers can run the defined studies in different settings (lab or crowdsourced study) while tracking user interactions using Trrack (Cutler et al., 2020) and then allowing further analysis by replaying user sessions. The tool also enables researchers to share existing studies for transparency and replication. Another web-based framework for conducting behavioral experiments is jsPsych, introduced by de Leeuw et al. (2023) and currently used by the HCI community (Cumbal et al., 2025; Masai et al., 2021). It allows researchers to define experiments in self-contained modules that can be reused in other experiments later. Additional community plugins allow capturing participants’ interaction, and e.g., support eye tracking. ReVISit’s setup and jsPsych are primarily tailored for single-participant sessions with isolated visual stimuli and thus do not, in principle, meet the requirements and infrastructure for evaluating and tracking fully interactive mixed-initiative VA situations and systems. Nevertheless, these approaches inspired the study setup configuration (using a modular structure and declarative format) and the initial session replay feature of MIVAIS’s Study Environment. 3. Design Goals and Requirements We derived 10 design goals based on the analysis of prior mixed-initiative VA applications (Zhou et al., 2025; Huang et al., 2026; Pérez-Messina et al., 2024; Sevastjanova et al., 2021), studies (Narechania et al., 2025; Setlur and Tory, 2022; Ha et al., 2024), design spaces (Stähle et al., 2025; Monadjemi et al., 2023; Holter and El-Assady, 2024), other VA and HCI study frameworks (Cutler et al., 2026; de Leeuw et al., 2023), and our own assessment of the capabilities we aim to support in MIVAIS. Our overarching goal was to provide a framework for web-based, mixed-initiative VA applications that supports rigorous scientific research, study sharing, and reproducibility. We structure the design goals into goals for General VA Applications and Study Environments and Mixed-Initiative Multi-Agent Frameworks, which we subdivide into Infrastructure and Study Environment related design goals. [colorGeneral]G1 – Provenance, Logging and Transparency by Construction Every mixed-initiative VA application requires logging of state updates, agent interactions (provenance tracking), and communication to provide transparency. Consequently, the Infrastructure must automatically capture all state updates, agent communications, and interactions with the VA environment to establish complete provenance. These logs should be detailed enough to identify misconfigurations or errors, and to reconstruct a full session (Narechania et al., 2025; Cutler et al., 2026). The Study Environment should directly map the logs to the study session and to additional session recordings across all agents, to allow for replay and post-session analysis. [colorGeneral]G2 – Flexible Study Design The technical burden of setting up a study upon an implemented VA environment should be as minimal as possible. Researchers should be provided with an interface for uploading and deploying study configurations. To support studies that combine quantitative and qualitative methodologies, study configurations should be flexible enough to include questionnaires in addition to VA system-related tasks (de Leeuw et al., 2023; Cutler et al., 2026). The configuration should support task definition, including embedding VA interfaces built on the Infrastructure, randomizing the task sequence, implementing attention checks, and handling invalid operations. [colorGeneral]G3 – Sharing of Studies for Replication of Studies and More Scientific Rigor The scientific community is embracing a culture of openness, actively advancing replicable research and shared study designs (Hornbæk et al., 2014; Iarygina et al., 2026). Researchers should be able to use the Study Environment to replicate or extend other researchers’ work and share study session analysis results with their stakeholders. [colorGeneral]G4 – Interface for Post-Study Analysis Once a study session is completed and data is collected, researchers should be able to see study results, replay the session, explore and analyze the collected data by being provided with an interface that interlinks the data and allows accessing the data for further analysis (Cutler et al., 2026). Beyond general requirements for study environments, we derived specific goals to enable effective human and agent collaboration, study agent behavior, and advance mixed-initiative systems: [colorInfrastructure]G5 – Decoupling Agents from the VA Environment Following Stähle et al. (2025), the Infrastructure should be modular and decouple agents from the VA environment, which would allow researchers to add or replace agents without updating the environment itself. Further, a layer in the Infrastructure should handle environment-state access and manipulation, as well as inter-agent communication, to enable mixed-initiative collaboration (Horvitz, 1999). All modules should support logging. [colorInfrastructure]G6 – Standardized Access for Humans and Software Agents Given the modular structure, the Infrastructure should provide a structured and standardized approach for adding and implementing agents. This should provide an interface and an infrastructure layer for human and software agents to access the VA environment uniformly (Stähle et al., 2025). The infrastructure should allow agents to observe states and perform actions on the VA environment. In addition, if required, a communication module should allow agents to share information directly among each other. [colorInfrastructure]G7 – Centralized and Declarative Configuration of Roles and Permissions To define and steer agent behavior and permissions, i.e., which parts of the VA environment they can observe, and act on, which actions they can execute, and with which agents they can communicate, it should be possible to configure these permissions and roles through declarative formats to make permission rules easier to audit and update without changing core application code (similar to contributions in Lotse (Sperrle et al., 2023), ReVISit (Ding et al., 2023; Cutler et al., 2026), Vega-lite (Satyanarayan et al., 2017), and Mosaic (Heer and Moritz, 2024)). To support dynamic environments (Stähle et al., 2025), the Infrastructure should allow configuration and permission updates at runtime. [colorInfrastructure]G8 – State Synchronization Across Agents and Environment As VA application comprises the visual interface, the underlying data, machine learning models, and the users – the agents – the state among all components of the application needs to be synchronized at runtime (Stähle et al., 2025). I.e., if one agent performs an action that updates the VA environment state, the new state needs to be propagated for updating the underlying data, the visual interface, and possibly the machine learning model. It must be ensured that other agents retrieve the updated state when they observe the VA environment. Further, agents should be able to follow other agents by retrieving their interaction trajectories to react to their actions. [colorStudyEnvironment]G9 – Allow Complex Collaborative Study Setups To study agentic mixed-initiative VA systems, researchers apply a diverse set of study scenarios (Wobbrock and Kientz, 2016), such as general user studies for collaboration and interaction logging (Zhao et al., 2026), controlled user studies with multiple conditions to measure the impact of different agent types or behaviors (Narechania et al., 2025), or different VA interfaces on the collaborations (Kuhlman et al., 2019), or Wizard-of-Oz (WoZ) scenarios (Sperrle et al., 2025). Further, common methods for data collection in HCI and VA are lab-based or crowdsourced studies. The Study Environment should support flexible collaborative study-scenario design and settings that allow scaling the number of collaborators in a study effortlessly. [colorStudyEnvironment]G10 – Advanced Data Collection Besides participants’ interaction and behavior data, researchers are more and more interested in collecting additional data, such as biometric sensor data (heart rate, etc.) (Ellemose et al., 2026; Panwar and Collins, 2018; Krisam et al., 2026). The Study Environment should support researchers in collecting additional sensor data during a study session, e.g., to later analyze participants’ stress and cognitive load. 4. The MIVAIS Architecture MIVAIS builds on the frameworks of agent-centric views (Monadjemi et al., 2023; Holter and El-Assady, 2024): VA systems consist of a VA environment in which human and software agents operate ( [colorInfrastructure]G5). This allows for deconstructing mixed-initiative VA systems into subparts that build the infrastructure and agents, enabling a modular approach to standardize the design of mixed-initiative systems and facilitating easier system comparison, configuration, and evaluation. In the following, we describe the architecture and components of MIVAIS provided in Figure 1, explicitly mapping each component to its corresponding design goal [colorGeneral]Gx (defined in Section 3). Figure 2. Architecture and data flow of a VA system built on the MIVAIS Infrastructure. Human and software agents interact with the application’s World State and each other, strictly mediated by a central Permission Guard. The backend handles state management and comprehensive audit logging, while an optional Study Gateway facilitates seamless integration with the broader Study Environment.In a system built on the , agents observe and act on the World State through the frontend or directly through the backend. Agents can send and receive messages through Communication Channels. The Permission Guard validates interactions with the World State and inter-agent communication based on the permission defined in the Agent Registry. The visual interface (frontend) and the backend are connected via a WebSocket communication that updates the state across the Provider (frontend) and World State in the backend. At runtime, Audit Logs are captured by collecting provenance and communication data from the agents and recording World State and permission updates. If the VA system is used with the Study Environment, the Study Gateway connects with the Study Environment and can be used to initiate a study session by setting agent roles, permissions, and initial World State. Further is collects the logs and can automatically validate participant tasks based on interactions or state updates. 4.1. The Mixed-Initiative Infrastructure Model To enable the researcher to build mixed-initiative VA systems with reduced effort, the MIVAIS Infrastructure (Figure 2) comprises the six modular components, explained in the following: VA Environment (World State), Agents, Agent Registry, Permission Guard, VA Visual Interface, and Agent Communication Channels. 4.1.1. [colorComponentWorldState]World State and VA Environment Similar to other works (Monadjemi et al., 2023; Abel et al., 2023; Stähle et al., 2025), the VA environment contains components, such as data management, analytical models, visualizations, knowledge representation, and potentially data collected through evaluation or feedback loops. The environment is built on the World State. The World State represents the current state of the VA environment at any given time ( [colorInfrastructure]G8). Therefore, all data and entities that can be updated through actions on the environment are stored in the World State. E.g., if an agent can select a dataset and specify a chart for further exploration, the dataset and the chart specification are stored in the World State. Every agent synchronizes with the World State to consume (observe) and update (act) the latest state. This is crucial for mixed-initiative applications where multiple agents can operate in the environment independently on their own schedule, influencing the World State. Further, a synchronized World State is important, as agents might join or leave the session during runtime (either through dynamic updates to the application configuration or through human agents quitting the session). If an agent joins an ongoing session, only consuming new events is not sufficient without full access to the underlying World State, as it needs the relevant context to evaluate the effects of those events. We operationalize the World State through a central and shared “blackboard” where every agent can read and update the state. Without any permissions, every agent interacting with the VA environment can subscribe to and update the World State. Although the World State could also be stored in a decentralized manner, the centralized approach offers multiple advantages such as more straightforward permission management, data collection, and provenance logging when conducting research studies ( [colorGeneral]G1). In addition to tracking the current World State, a provenance log of all manipulations is created. 4.1.2. [colorComponentAgents]Agents Based on the definition by Stähle et al. (2025), in mixed-initiative (VA) systems, an agent is any entity, human or software, “capable of perceiving its environment and proactively performing goal-driven actions. An agent must be capable of autonomous decision-making by interpreting its environment and applying internal logic, allowing it to take or cede initiative.” In line with this definition, in MIVAIS every agent is modeled uniformly. Agents can perceive the full or parts of the VA environment through accessing the World State or listening to its updates ( [colorInfrastructure]G5). Further, any logic within the agents can derive a decision and act on the environment by updating parts of the World State. When implementing an agent, it is important to define which parts of the environment (World State) the agent needs to observe, how it computes and derives its next actions, and which parts of the environment (World State) are affected by these actions ( [colorInfrastructure]G6). Agents can communicate with other agents by sending and receiving messages in subscribed channels (Section 4.1.3). This allows agents to notify each other, request actions, or share information. Each software agent can be implemented in a separate code file that defines its logic and is activated once registered in the Agent Registry. 4.1.3. [colorComponentCommunication]Agent Communication Channels To allow agent-to-agent interactions, MIVAIS provides Communication Channels. As messages across agents do not update the World State, by design, these signals do not pass through the environment ( [colorInfrastructure]G6). Instead, agents can subscribe to and publish to communication channels. This allows us to design any kind of agent communication, e.g., distributed, centralized, or hierarchical networks. Further, these networks can be dynamically updated at runtime as agents can dynamically (un)subscribe to channels. The body of messages is text-based, allowing human and software agents to communicate in natural language. At the same time, this also supports structured language for messages only sent between software agents. Because the central infrastructure of MIVAIS has access to all channels, it is possible to log all messages exchanged between agents ( [colorGeneral]G1). 4.1.4. [colorComponentPermissionGuard]Agent Registry and Permission Guard The Agent Registry uses a domain-specific language (DSL) to define which agents are present in the system, specifying their identifiers and roles (Appendix C.1). Additionally, it defines the permissions, i.e., which agent can access and update which parts of the World State ( [colorInfrastructure]G7). Agents and Permissions can be dynamically added or removed from the registry. This allows researchers and developers to manipulate the behavior and scenarios of their applications at runtime using scheduled conditions or updates on demand. The Permission Guard centralizes the validation of permissions defined in the Agent Registry. Thus, before agents can perceive or update parts of the World State, the operation is validated. If an agent tries to access or update a part of the VA environment for which it has no permission, the permission guard denies the request and informs the agent. If the agent has permission, the Permission Guard lets the agent’s observation or state update pass. Further, the Permission Guard ensures that agents can send and receive messages only through the communication channels they are authorized to use. This centralized access control enables auditing of all access, update, and communication attempts by agents ( [colorGeneral]G1). 4.1.5. [colorComponentInterface]Visual Interface for Agents Relying on the UI Human visual perception is particularly effective at recognizing complex patterns that can be difficult for algorithms to detect. VA environments therefore use graphical interfaces to represent their World State. To interact effectively with these environments, both human and software agents (e.g., (Stähle et al., 2026)) require direct visual access to this representation. In this work, we focus on web-based interfaces that allow us to use technologies such as WebSockets for state synchronization between the application’s backend and frontend. When an agent executes an interaction to act upon the VA environment in the visual interface, this interaction is first validated by the Permission Guard, and if valid, updates the World State. When another agent updates the World State, the visual interface updates the visual representation to present the updated state ( [colorInfrastructure]G8). As UI agents can perform interactions in the visual interface, such as mouse interactions that contain rich information (Nguyen et al., 2016; Holter et al., 2026), these interactions can be recorded, displayed in the interface of another agent, and passed to the central audit log of MIVAIS ( [colorGeneral]G1). 4.1.6. Summary Grounded in the design goals, these components simplify mixed-initiative VA system development, allowing researchers and developers to focus on agent logic while abstracting away state synchronization, communication, and permission management. We show its applicability in Section 5.1. 4.2. The Study Environment Once researchers have implemented new systems or approaches, the main goal is to evaluate these. Similar to many other research areas in HCI and VA, mixed-initiative approaches are usually evaluated based on a mix of quantitative and qualitative methods. Researchers, therefore, use multiple tools in a queue to set up studies and enable the necessary data collection (e.g., by tracking user interactions and final application states). Furthermore, they calculate metrics on application logs, record the screen of the participant(s), capture think-aloud sessions, and collect questionnaire results, commonly through a separate tool, such as Google Forms11 1 https://docs.google.com/forms. This creates significant overhead, while leaving collected data fragmented across multiple sources, complicating data aggregation and alignment for downstream analysis. It becomes even more challenging when testing multiple applications, running multi-user sessions, or conducting WoZ experiments. When designing MIVAIS, we address this gap by following best practices from prior work and introducing new features that are especially relevant for evaluating multi-agent mixed-initiative systems. For each feature, we first outline its similarities to existing study frameworks, followed by the unique properties MIVAIS provides for mixed-initiative setups. 4.2.1. Study Setup Studying mixed-initiative approaches in VA can have many different facets (Wobbrock and Kientz, 2016; Sperrle et al., 2021a). Therefore, before conducting a study, researchers need to specify the experiment. Like existing frameworks (Cutler et al., 2026), MIVAIS uses a DSL to define study structures, participants, and tasks ( [colorGeneral]G2), although in MIVAIS, we use YAML instead of JSON to support inline comments for documentation. Researchers can edit study and task files directly in the Study Environment interface or upload them from an IDE. Studies include ID tags, names, optional versioning for identification, and task sequences, explained in Section 4.2.2. Beyond interaction and response tracking ( [colorGeneral]G1, Sections 4.2.3 and 4.2.4), MIVAIS provides built-in multi-modal collection, including screen recording, audio capture, and automatic transcription ( [colorStudyEnvironment]G10). Finally, researchers can configure an integrated consent form shown before the experiment to address ethics and privacy ( [colorGeneral]G1). Additionally, MIVAIS provides several unique properties for study configuration. For instance, it allows researchers to conduct both single- or multi-system evaluations ( [colorStudyEnvironment]G9) simply by referencing the target systems in the DSL during setup. Systems can be hosted internally or externally, requiring a reference name, system URL, and health endpoint. The URL embeds the VA interface into the participant view and enables backend communication with the MIVAIS Study Gateway (Figure 2), while the health endpoint tracks availability for logging and crash recovery. To support single- or multi-participant studies, researchers specify the participant mode, total capacity, and specific roles (e.g., user vs. wizard in WoZ studies) ( [colorStudyEnvironment]G9). MIVAIS captures multi-modal data from all collaborating participants simultaneously and aligns it along a shared timeline ( [colorStudyEnvironment]G10). 4.2.2. Task Definition Tasks define the study’s subparts and can vary in structure. For modularity and reuse, tasks are stored in stand-alone YAML files identified by a task ID and a relative path (optional), which allows reusing them multiple times (e.g., providing participants with the NASA-TLX (Hart and Staveland, 1988) or System Usability Scale (Brooke, 1996)). In MIVAIS, task sequences are grouped into blocks and subgroups, which are processed sequentially or shuffled to minimize carry-over effects (Lazar et al., 2017; Brooks, 2012) ( [colorGeneral]G2). Similar to reVISit and jsPsych, MIVAIS allows researchers to display the stimulus, i.e., a target mixed-initiative VA application, alongside task descriptions (see Figure 3), interactive forms (e.g., subtask lists, finding reports, or Likert scales), questionnaires, or attention tests ( [colorGeneral]G2). If required, tasks can also be timed to enforce completion deadlines. Additionally, we allow storing tasks in a central library for reuse across multiple studies, eliminating redundant configuration ( [colorGeneral]G3). The DSL grammar for a task definition is provided in Appendix C.3. To address mixed-initiative setups, MIVAIS uniquely enables researchers to either reconfigure agent permissions and initialize the VA system’s World State for individual tasks (e.g., disabling an agent from offering proactive visualization recommendations during an initial exploratory task to establish a baseline for unassisted user discovery) or maintain a continuous system session across sequential tasks. Additionally, tasks in MIVAIS can include subtasks that must be completed within the linked mixed-initiative VA application ( [colorStudyEnvironment]G9), which are validated automatically by tracking the application’s event logs received through the Study Gateway and monitoring for a certain event or accumulation of events (Figure 3). Figure 3. In MIVAIS, a study and its tasks are defined in DSL YAML files. This example shows an excerpt from the Study Definition and one of the Task Definitions for a study on the reimplemented ProactiveVA system (with additional configuration details elided). The right part shows the deployed study and task, as well as the resulting study interface provided to participants.In , a study and its tasks are defined in DSL YAML files. This example shows an excerpt from the Study Definition and one of the Task Definitions for a study on the reimplemented ProactiveVA system. The right part shows the deployed study and task, as well as the resulting study interface provided to participants. 4.2.3. Provenance Tracking As with other study frameworks, MIVAIS records mouse interactions and cursor paths, and it continuously collects provenance trace logs, storing them together with participant responses and telemetry ( [colorGeneral]G1). This setup enables both high-level summaries and detailed, on-demand inspection during post-session analysis. In addition, the Study Environment’s provenance tracking of all agents is built on centralized audit logs created in each application that uses the MIVAIS Infrastructure. Once a study is deployed and participants begin a task using a MIVAIS-based VA application, the Study Environment connects directly to the application’s Study Gateway via WebSocket and collects the data. In addition, it allows researchers to capture custom agent-interaction events. Beyond agent provenance, the system tracks World State changes and inter-agent communication. Any additional data recorded by researchers, like biometric sensor data, is automatically aligned with these provenance traces ( [colorStudyEnvironment]G10). 4.2.4. Tracking of Interface and Audio HCI and VA research frequently incorporates screen and audio recordings. Capturing these media streams allows researchers to record the exact visual stimuli perceived by participants alongside think-aloud protocols during task execution. When configuring a study with MIVAIS, researchers can enable screen and audio recording. As our framework is web-based, this feature uses the built-in browser’s APIs to record the screen and audio. The batched recording chunks are send from the participant’s client to the host of the Study Environment. If participants decline the recording, the study can be terminated immediately or continued without recoding the data. Similar to reVISit, MIVAIS provides automatic transcription of recorded audio. Transcripts are aligned with the other tracking data such that timestamps match and can be used for coding (Braun and Clarke, 2006) or further analysis of participants’ thoughts (Ericsson and Simon, 1993). To address mixed-initiative needs, MIVAIS can capture the interface and audio of all participants in the same session simultaneously and align them on a timeline, allowing switching point of view at any time during post-analysis ( [colorStudyEnvironment]G10). 4.2.5. Tracking of Additional Sensor Data To afford maximum flexibility in data collection ( [colorStudyEnvironment]G10) in mixed-initiative setups, MIVAIS offers a REST API endpoint capable of ingesting external data streams into active study sessions. Researchers can utilize this mechanism to capture additional metrics, such as biometric sensor data or eye-tracking logs gathered from secondary devices, and consolidate them for offline analysis. The endpoint can handle single and batched data. This allows for reducing data traffic while still enabling high-frequency data collection. Additional data tracking can be enabled in the study configuration. If activated, a unique token is provided at the beginning of the study session that researchers add to their sending devices. The token takes care of authentication and study session mapping. Data can be added both during an active session and after the session is completed to add data from offline sensors. The data is aligned to the other data through the ISO-8601 wall-clock timestamp. 4.2.6. Deploying and Executing Studies Addressing our design goal [colorGeneral]G2, the Study Environment provides researchers with necessary infrastructure to deploy their studies. Once a study is designed, configured, and the configuration files are uploaded to the MIVAIS Study Environment, researchers register a study for deployment in the admin visual interface. All related study and task configurations are copied to a read-only directory to preserve the underlying files from being updated while the study is deployed. MIVAIS covers hosting the study, providing each participant with a unique access code. Participants access the study by visiting the base URL of MIVAIS and entering their code (this forwards the participant to the linked study). Studies can be conducted through lab studies or crowdsourcing ( [colorStudyEnvironment]G9). For lab studies, researchers can manually generate and share the access codes with the participants. Participants then access the study via a web browser on a computer in the lab setup. To run crowd-sourced studies, MIVAIS currently provides a connector for Prolific22 2 w.prolific.com/ that lets researchers source participants through the Prolific Platform by lodging a study-specific URL. The connector handles assigning participants to the correct study session, storing their participant IDs and other metadata, and, at the end, forwarding them to the Prolific Platform. As there are multiple more providers for crowd-sourced studies, we plan to provide additional connectors soon. An experiment can be terminated either when a predefined number of participants is reached, which prevents other participants from signing up, or when researchers manually terminate the study by archiving it. For investigation and analysis of a single completed study session, it is not required to terminate and archive the whole study. As an additional feature, MIVAIS validates the syntax of the study and task configuration files and, if correct, deploys the study immediately. If a study contains multiple participants who should collaborate in the same session, we also provide mini-cohorts ( [colorStudyEnvironment]G9): access codes directly forward participants into a shared session. MIVAIS isolates each session and propagates its context to the involved VA systems. During tasks, the MIVAIS Study Environment connects directly to the active VA session to collect and store provenance and event data for post-session analysis. Researchers can also passively monitor active sessions in real time via the Live view. 4.2.7. Post-Study Session Replay and Evaluation Following study completion, MIVAIS’s Study Environment provides several features to assist researchers with data analysis and evaluation. Session Overview and Participants’ Responses Consistent with existing study platforms, selecting a study with completed sessions displays high-level statistics in the Study Overview and Summary View (Figure 4), including session counts, task and subgroup summaries, completion times, and aggregated participant responses ( [colorGeneral]G4). Selecting an individual task opens the Single Task Result Overview (Figure 4), showing task-specific participant responses or performance metrics. Interaction Timeline and VA Environment State Analysis To support multi-agent collaboration analysis, the timeline visualizes all human and software agent interactions, communications ( [colorInfrastructure]G8), and recorded sensor data during VA system tasks ( [colorStudyEnvironment]G10). Researchers can define custom interaction or event sequences to be highlighted on the timeline ( [colorGeneral]G3), revealing individual agent behaviors or multi-agent interaction patterns. Session Replay Additionally, researchers can review individual study sessions using the Replay feature (Figure 4, I.). To analyze how agent actions transform the World State, the Replay View displays the state at any timestep, highlighting updated values and allowing researchers to click variables to compare previous and current values. In multi-participant sessions, researchers can toggle between participant viewpoints, with the active view indicated by a camera icon next to the participant’s ID. Think-Aloud Analysis and Annotation When audio recording is enabled, MIVAIS automatically transcribes input audio with timestamp annotations using Whisper small (Radford et al., 2023) ( [colorGeneral]G4). As Cutler et al. (2025) note, transcripts in think-aloud studies aid in insight elicitation, detecting participant surprise during agent interactions, and usability testing. Researchers can highlight and annotate specific transcript sections, which are then reflected on the timeline visualization. Similar to Cutler et al. (2026), this captures user thought processes during tasks rather than post-interaction. Further Analysis To enable deeper analysis of the collected data, MIVAIS offers the option to export the data as JSON lists, which can then be imported into any analysis tool or notebook the researcher prefers. Figure 4. The visual interface of MIVAIS for post-study analysis. The dashboard provides researchers with three primary analytical views: (I.) a Study Overview summarizing aggregate metrics and session progress, (I.) a Task Result Overview for evaluating specific task outcomes (e.g., questionnaires), and (I.) a comprehensive Replay View that synchronizes screen recordings, interaction timelines, and transcripts for deep inspection of individual sessions.The visual interface of MIVAIS for post-study analysis. The dashboard provides researchers with three primary analytical views: (I) a Study Overview summarizing aggregate metrics and session progress, (I) a Task Result Overview for evaluating specific task outcomes (e.g., questionnaires), and (I) a comprehensive Replay View that synchronizes screen recordings, interaction timelines, and transcripts for deep inspection of individual sessions. 4.3. Library Implementation and API The Infrastructure of MIVAIS is implemented as a Python library that provides the base infrastructure for the described architecture and a WebSocket interface for the frontend. We further provide a TypeScript library that leverages backend communication in the frontend of the application, leaving the choice of frontend framework to the developer of the mixed-initiative VA system. The Study Environment is based on a FastAPI backend, providing a RESTful API and storing data in a MongoDB. The visual interface is implemented in React TypeScript. For deployment, we provide Docker containers that can be hosted or run locally. The source code is provided in the supplemental material. 5. Technical Evaluation A typical MIVAIS workflow begins with conceptualizing the system by outlining the core data model, machine learning components, and VA interface. Next, researchers implement this draft in the backend by defining the central World State, agent configurations, permissions, and communication channels. After coding the agents’ logic and integrating the frontend via the WebSocket API, the system is complete. Finally, researchers use the Study Environment to configure, deploy, and analyze their study before sharing their system and results with the community for replication. A more detailed description is provided in Appendix A. We evaluate MIVAIS’s applicability by adapting three prior mixed-initiative VA interfaces to our Infrastructure and assess our design using the Technical Dimensions of Programming Systems (TDPS) (Jakubovic et al., 2023). 5.1. Replicating Prior Work We utilize MIVAIS to replicate three mixed-initiative prior works in informing data-driven decisions (Podium (Wall et al., 2018)), data exploration and targeted question answering (Voyager 2 (Wongsuphasawat et al., 2017)), and context-aware proactive assistance in VA (ProactiveVA (Zhao et al., 2026)). The source code is provided in supplemental material. 5.1.1. Podium Wall et al. (2018) developed Podium, a mixed-initiative VA application that helps users identify attributes relevant to ranking multivariate data. Users drag table rows to express their preferred ranking, and a Ranking-SVM–based agent infers an attribute-weighting model that best fits these preferences. We reimplement Podium’s interface and functionality so human users can rank cars by their attributes, given a dataset of automobile statistics (Quinlan, 1993). Our implementation maintains a [colorComponentWorldState]World State that stores the initial order of data points and attribute weights. It also includes two [colorComponentAgents]agents (one human, one software), both with [colorComponentPermissionGuard]permissions to update the ranking in the World State, while only the human can add priorities to specific attribute weights and only the software agent can change the attribute weights in the World State. The human agent can request the software agent to update attribute weights or rank all data points using the current weights by sending a message through the [colorComponentCommunication]communication channel (triggered by a button click). In the [colorComponentInterface]VA interface, we bind the World State ranking to the main table and map attribute weights and priorities to the control panel visualization. 5.1.2. Voyager 2 By combining manual and automated chart specification, Voyager 2 (Wongsuphasawat et al., 2017) enables analysts explore data and drill down to specific questions. Users create charts to find insights, while a software agent simultaneously suggests additional visualizations relevant to the current chart. In our MIVAIS-based reimplementation of Voyager 2, the [colorComponentInterface]user interface provides the same views and functions as the original. The views synchronize relevant data with the [colorComponentWorldState]World State, which stores uploaded datasets, current chart specifications, alternative encodings, related summaries, wildcard results, field suggestions, and bookmarked charts. We have two [colorComponentAgents]agents (one human, one software). [colorComponentPermissionGuard]Permissions match the original: the human agent can manually select attributes and wildcards to define charts and bookmark specifications. The software agent generates insights and recommends alternative encodings and field suggestions based on the human’s current specification. Recommendations update the World State and thus appear in the user interface as selectable, bookmarkable options, whereas generated insights are sent directly to the human analyst via the [colorComponentCommunication]communication channel. To assess the feasibility of multiple software [colorComponentAgents]agents in this environment, we implemented a variant that splits insight and recommendation generation between two agents (recommendation and insight) with different [colorComponentPermissionGuard]permissions. The recommendation agent suggests related views by observing the [colorComponentWorldState]World State (focus view and chart specification) and updating alternative encodings, field suggestions, and related summaries. The insight agent operates only on the dataset, updating focus encodings and [colorComponentCommunication]communicating identified insights to the human user, including possible actions. If the human agent selects an action, it is sent to the insight agent, which then instructs the recommendation agent to apply it in subsequent suggested chart generations. 5.1.3. ProactiveVA Zhao et al. (2026) introduce ProactiveVA, a framework using an LLM-powered software agent that proactively offers context-aware assistance by monitoring human interactions. We reimplement their social event analysis system using the MIVAIS infrastructure. To provide a similar [colorComponentInterface]interface and identical functionality to the human user as the original system, we implemented ProactiveVA with a [colorComponentWorldState]World State that stores the static dataset, selected entities and hexagons on the map, time range, filter keywords, current focused view, notes, pending suggestions, and staged evidence. In the original design, a single software agent detects when help is needed, infers user intent, provides suggestions, and, once confirmed, reasons over and acts on the data. This agent perceives the environment (observation), reasons for decision-making (logic), and acts (actions) on the environment. Although MIVAIS could replicate this monolithic agent, we intentionally adopted a multi-agent architecture as an illustrative case study: five [colorComponentAgents]agents execute the subtasks in (Zhao et al., 2026) and [colorComponentCommunication]communicate via shared channels. We do not claim this decomposition is superior for ProactiveVA; instead, we split the logic to explicitly demonstrate MIVAIS’s agent-to-agent communication and decentralized coordination – core framework features a monolithic design would not validate. Our implementation uses three intent-perception agents (help-needed detector, user-analysis checker, possible-solution detector), each [colorComponentPermissionGuard]provided access to different parts of the [colorComponentWorldState]World State to detect when the user needs assistance. Two additional agents (planner and actor) generate suggestions and intents, reason over the data, and adjust data selections. As the human [colorComponentAgents]user explores the data, the perception agents analyze the user’s behavior. When they detect a need for assistance, they notify the planner via the assistance-required communication channel. Receiving the message, the planner generates suggestions and sends them to the user via the pending-suggestion channel. After the human confirms a suggestion, the actor executes it. The user can also directly message the actor via chat to request help. Unlike the original ProactiveVA, we use Gemini 2.5 flash (Gemini Team, 2023) for LLM inference. 5.1.4. Summary and Lessons Learned The original systems we replicated differ in nature and functionality, yet all can be implemented on the same modular Infrastructure. The replication process shaped the final synchronization mechanism and permission guard: we moved from broadcasting all state updates to all agents without restriction to a guarded, selective update scheme. It also refined the structure of agent configuration files, revealing additional useful features, such as hyperparameter injection to steer agent behavior. Although first decomposing an existing application into the core components needed for the MIVAIS Infrastructure is cognitively demanding, we could quickly adapt agents in an existing system (Section 5.1.2) by leaving most of the code unchanged and focusing on agent implementation and reconfiguration. 5.2. Technical Dimensions of Programming Systems (TDPS) Similar to Cutler et al. (2026), we self-assess MIVAIS via close reading utilizing the “Technical Dimensions of Programming Systems” framework (Jakubovic et al., 2023) and the cluster descriptions by McNutt et al. (2025). To provide analytical clarity, we decouple our assessment into the Infrastructure and Study Environment. The first author assessed MIVAIS for each of the 7 dimensions, and two other authors reviewed it. 5.2.1. Interaction Which loops in the system are overlapping and how far apart are the corresponding gulfs of evaluation? Infrastructure: MIVAIS narrows the traditionally wide feedback loops inherent in mixed-initiative VA development. The modular architecture enables software agents to be implemented and tested sequentially, independent of the VA interface and the study setup. Study Environment: MIVAIS mitigates the wide, iterative feedback loops in study design by enabling researchers to test isolated subtasks without executing a full study walk-through. The architecture manages session isolation and data storage without requiring the VA application to be explicitly deployed within the Study Environment. 5.2.2. Notation What notations are present and how do they interrelate? We utilize a YAML-based DSL to define agent and study configurations. The MIVAIS DSL is based on a common abstraction to avoid the need to maintain a full programming language, which might result in a steep learning curve and increased difficulty in generating configurations. We explicitly decided on YAML as, unlike JSON, it allows for adding comments within the files to share additional information with other stakeholders or researchers. Infrastructure: Developers configure modular agent permissions using highly composable YAML files, maintaining the freedom to build VA interfaces using their preferred frameworks. Study Environment: The DSL enables researchers to sequence tasks and dynamically overwrite agent parameters for ablations without recompiling the underlying application. While YAML is structurally prone to indentation errors, MIVAIS actively mitigates this risk through strict pre-deployment schema validation. 5.2.3. Conceptual Structure What is the shape of the notations at play and how do they relate to user goals? Infrastructure: MIVAIS enforces a modular conceptual paradigm that programmatically unifies human and software agents. To manage the inherent temporal mismatches and race conditions when software agents and humans interact simultaneously, all updates are routed through a centralized World State and validated by a Permission Guard. Study Environment: Leveraging declarative configurations for permissions, tasks, and studies aligns with previous works (Sperrle et al., 2023; Ding et al., 2023). Further, it centralizes study execution and multimodal data tracking for a diverse set of study setups common in HCI and VA, such as system testing, human-software collaboration, multi-agent interaction, Wizard of Oz studies, etc. Nevertheless, relying on the infrastructure and specification of the study through a declarative format might not satisfy all users. 5.2.4. Customizability How can programs be modified? The entire codebase will be open-source, but we explicitly avoid relying on “forking” as a primary extension mechanism to maintain structural standardization. Nevertheless, in exceptional cases, if required, users can investigate and change the full source-code. Infrastructure: The MIVAIS architecture is designed for modular extension; developers can dynamically add custom agents, update permissions, or define tracking events without modifying the core infrastructure. Study Environment: Researchers can easily share, replicate, and modify studies by customizing task orders in the configuration files. Furthermore, an API supports integrating supplementary data streams, such as biometric sensors, directly into the session log. 5.2.5. Complexity How is complexity dealt with through design and automation? MIVAIS explicitly separates application logic from study execution to reduce cognitive overload. Infrastructure: The strict modularity isolates agent definition, state synchronization, and inter-agent communication, allowing developers to focus entirely on agent and environment logic. Study Environment: The declarative setup lowers study implementation effort but introduces an integration trade-off: researchers must maintain two separate mental models (application versus study setup), requiring careful alignment when configuring cross-system dependencies. This duality is inherent to mixed-initiative research: the engineering model of application differs from the experimental model of study tasks and metrics. MIVAIS preserves this separation with a strict architectural boundary, preventing study-specific task logic or logging from contaminating the application code. 5.2.6. Errors What are they and how are they handled? Error handling depends on the error that occurs and whether it is related to the Infrastructure or the Study Environment. Infrastructure: The modularity isolates malfunctioning agent behaviors, while the type-safe MIVAIS libraries utilize linters to expose invalid mappings. Built-in audit logs can expose agent misconfigurations, unauthorized update attempts, and connection failures. Study Environment: Prior to deployment of studies, MIVAIS automatically validates the YAML schema and verifies the reachability of linked VA systems. While syntactic and connection errors are caught systematically, MIVAIS does not highlight semantic experimental design errors; researchers must manually verify logical linkages, such as cross-system task accomplishment notifications. 5.2.7. Adaptability What socio-technical (e.g., learnability) dimensions are considered? MIVAIS is tailored for mixed-initiative (VA) researchers who have experience in developing mixed-initiative tools. It provides a high architectural ceiling and a low experimental threshold. To ease building systems and studies, we provide tutorials and example implementations, allowing less experienced users to use MIVAIS for their research. Infrastructure: Designing mixed-initiative systems requires full-stack programming proficiency to leverage our framework’s capabilities. Nevertheless, MIVAIS already handles complex structures, such as state synchronization and inter-agent communication. Study Environment: Once the core VA system is deployed, the YAML-based configuration provides a low-threshold entry point, allowing less technically versed researchers to design, modify, and execute complex multimodal studies without altering the underlying application code. 6. User Evaluation: Expert Case Studies In the following, we present three case studies of projects built and evaluated using MIVAIS, demonstrating its utility and ease of integration into both new and existing applications. 6.1. Methodology & Participants Methodology The case studies were carried out over two weeks and started with an introduction session (∼ 1 hour) in which we introduced the theoretical framework to the VA researchers, provided a tutorial for an example system, showed two example implementations of previous works (see section 5.1), and shared the codebase. In the following two weeks, participants implemented or retrofitted their systems in the backend and frontend, set up, and conducted pilot user studies. During this period, we kept in touch with participants to check on their progress and answer clarifying questions. Because our evaluation focuses on validating the framework’s capability to streamline development and automate multi-modal data collection, the experts were instructed to conduct a focused pilot study with their system (three study sessions per expert tool). This scope was selected to thoroughly stress-test the functionality of MIVAIS and validate the multi-modal logging pipeline and post-analysis capabilities across diverse mixed-initiative VA scenarios, without placing a prohibitive administrative burden on the experts participating in our case study. We concluded the case studies with semi-structured interviews to capture in-depth, formative feedback from experts on the utility and experience of using MIVAIS (see Appendix B). Interviews were transcribed verbatim. Two authors iteratively reviewed the interview transcripts alongside the experts’ notes taken during the two-week use of MIVAIS. The feedback was grouped into key areas of interest: usability, workflows, development and integration effort, development strategies, and enhancement and feature requests. We report these findings as descriptive observations. Participants We recruited three researchers (P1−3P_1-3) who conduct research in HCI or VA, currently designing or implementing mixed-initiative VA systems. They have rich experience (2-5 years) in design and development of multi-agent mixed-initiative systems, and have previously conducted at least three user studies. Participants were not paid, as they could use the built applications and pilot studies for their own research. This study was approved by ETH Zurich’s Ethics Commission (Project 26 ETHICS-244). 6.2. Developed Prototypes Figure 5. Case study results implemented by the experts. UserRAG (P1P_1) and TopoAlign (P2P_2) are shown in deployed study mode, while PleaSQLarify (P3P_3) is shown in the Replay View of a study session. 6.2.1. UserRAG UserRAG is an interactive mixed-initiative VA system where human and software agents collaborate to steer the retrieval and augmentation stages in a Retrieval Augmented Generation pipeline. Besides the generic retrieval, a software agent generates additional context on request of the human agent that can be further used to augment the retrieval. The researcher aims to evaluate their mixed-initiative approach by letting participants interact with the system and measure system usability and whether participants achieve better responses when the software agent augments the retrieval pipeline. 6.2.2. TopoAlign TopoAlign (Yan et al., 2026) is a topology-aware mixed-initiative VA framework that leverages mapper graph agents to help machine learning experts compare and analyze the structural alignment of neural network representations across different models, layers, or training conditions. The study evaluates whether mapper graph agents, compared to traditional embedding approaches, can uncover additional similarities or differences between two representation spaces and how machine learning experts interact with them. 6.2.3. PleaSQLarify PleaSQLarify (Chan et al., 2026) is a mixed-initiative VA system designed for natural language database querying (text-to-SQL). It re-frames input ambiguity through the lens of pragmatic repair – an incremental clarification process between human agent and the SQL software agent that helps align user intent with system interpretations. The authors’ study aimed to evaluate pragmatic repair in text-to-SQL disambiguation by observing how agents navigate uncertain query spaces and assessing their overall control, interaction workflows, and system usability. Specifically, they investigated whether prioritized decision variables allow agents to efficiently and understandably filter the query space. Retrofitting the existing system with the MIVAIS Infrastructure and replicating the study in the Study Environment allows assessing MIVAIS’s effectiveness along the full research pipeline. 6.3. Expert and Developer Experience We refer to the experts as P1P_1 – P3P_3, following the prototype order of Section 6.2. 6.3.1. General Feedback All experts characterized the agent-centric abstraction intuitive and quick to learn, since it reuses familiar engineering concepts: agents are “very logical units” like classes in object-oriented programming, and the World State maps to the model in model–view–controller (P1P_1). For participants, learning the underlying concepts of the MIVAIS took from minutes (P2P_2) to an afternoon (P3P_3), supported by the introduction session, tutorial, and documentation (P2,3P_2,3). All three experts implemented their intended collaboration behaviors without reporting an expressiveness ceiling, as “all the power of a programming language” stays available, “just placed at a different level” (P1P_1) ( [colorInfrastructure]G5, [colorInfrastructure]G6). Friction arose mainly in the declarative layer: getting used to the DSL grammar for agents, permissions, and study/task definitions took time (“time-consuming” for P2P_2), and separating task and study files led to “back and forth” (P1P_1), though P1P_1 also noted “great benefits in reusability of tasks.” One expert needed our support to fix a defect when injecting a World State update within a study due to a variable naming error. MIVAIS required no toolchain changes: experts used their familiar tools with the Python/TypeScript stack (P2P_2), browser-based deployment (P1P_1), and edited YAML files in their preferred editors. 6.3.2. Development Strategies All experts followed the same implementation sequence: agent decomposition and World State in the backend first, visual interface last. P1P_1 defined agents, their communication, and the World State, then built the frontend; P2P_2 and P3P_3 extracted main components from an existing system, defined agents and World State, then adjusted the frontend. The decomposition served as a design lens: P1P_1 noted they “would not have looked at the problem on this agentic level” otherwise and found it useful “to look at the roles and abilities instead of just the functions”, which also made adding another agent mid-development straightforward ( [colorStudyEnvironment]G9). Making permissions explicit was seen as helpful rather than overhead (P2P_2, [colorInfrastructure]G7), and routing all updates through the World State was considered “better practice” because all variables are centralized (P1P_1, [colorInfrastructure]G8). For study design, P2P_2 and P3P_3 derived additional study ideas from the theoretical concept and technical options offered by MIVAIS. They first configured the core study setup and modalities to track, then defined the tasks. As tasks and metrics reside outside the application ( [colorGeneral]G2), P2P_2 explored different study setups once the system was deployed without the need to edit their VA application and captured additional metrics “on the fly.” 6.3.3. Development and Integration Effort Integration costs depended on the starting point: P2P_2 and P3P_3 retrofitted existing systems, while P1P_1 built on the MIVAIS Infrastructure from scratch. All experts observed a shift and reduction in effort: implementing the mixed-initiative VA system itself is “about the same” as without MIVAIS (P2P_2), with “more focus on the agent behavior as MIVAIS Infrastructure takes care of state synchronization, communication channels, etc.” (P3P_3). In contrast, configuring, deploying, and iterating studies became much faster (P1,2,3P_1,2,3): complete setups were prepared within hours, and ablations created by copying tasks (P1,3P_1,3). Because these are self-assessments, we treat them as perceptions: the pilots covered the full pipeline (system implementation plus study definition, deployment, and execution) but are too small to quantify analysis savings. P3P_3 offered the clearest comparison: their previous workflow for a user study spanned five sources—recordings, transcripts, notes, plus external questionnaire and consent forms—requiring about half a day per transcript pass plus extra hours revisiting recordings to contextualize user quotas. This was replaced by a single aligned record ( [colorGeneral]G1, [colorGeneral]G4): sessions can be replayed to locate relevant events (P2,3P_2,3), and interlinking preserves the context of qualitative statements (P3P_3). P3P_3 emphasized this primarily as a quality gain: “you can measure everything from the start.” P1P_1 and P2P_2 appreciated that additional sensor streams only need to be enabled in the study setup ( [colorStudyEnvironment]G10). Contrary to our expectation (Section 5.2.5), no expert found maintaining separate application and study models burdensome. Instead, P3P_3 noted that mapping application variables to study events adds a layer that “can create confusion and errors” and must be validated—an effort absorbed by a pilot run, which is good scientific practice (Olson and Kellogg, 2014). 6.4. Enhancement and Feature Requests Our expert participants suggested enhancements to MIVAIS, including authoring support for the study DSL: a task-type selector with skeleton code (P1P_1), plus LLM agents to help generate task and study configurations (P2P_2). P2P_2 also requested video integration at the start of a study to replace live introductions, which we added in the latest version. For post-analysis, P1P_1 suggested filters to faster find specific study sessions, and P3P_3 recommended built-in verification of variable-to-event mappings to increase confidence in the data, a feature that we consider for future work. 7. Discussion MIVAIS allows developers and researchers to build web-based mixed-initiative VA systems through a modular Infrastructure. Further, they can configure and deploy (multi-agent) user studies along their requirements that use the systems as stimuli. Our evaluation shows that the Infrastructure is flexible and modular enough to implement new or retrofit existing systems. Further, our expert case studies show that once implemented, researchers can configure and deploy their experiments and user studies with low effort. In addition, researchers like the diversity of data that can be collected through the Study Environment, and the optionality to add additional data streams from external sources, such as biometric sensors. 7.1. Limitations & Future Work While establishing a solid study framework, MIVAIS ’s current design operates under several inherent conceptual and technical limits. Adoption Effort and Cognitive Load While MIVAIS provides an infrastructure for implementing mixed-initiative VA applications, it requires researchers and developers to align their application design and architecture with the theoretical concepts of MIVAIS. While this improves standardization and application modularization, this might lead to additional effort and development overhead required to adapt to the provided infrastructure. Further, they need to maintain “two separate mental models” for the application state and the study setup, which might be cognitively demanding. Nevertheless, this burden comes to the benefit of being able to quickly set up and run studies with the implemented applications. Post-Analysis Enrichment Researchers can use the MIVAIS Study Environment to collect a diverse set of data during a study session, also including sensor data. At the moment, data is visualized per data stream in the visual interface for session replay, but the interface does not allow joining data streams. Currently, e.g., this does not support visualizing eye tracking data, such as area of interest, on the captured screen replay. Such features would enrich the post-analysis, but also depend on individual requirements, which we defined to be out of scope for this work. Nevertheless, researchers can adapt the core implementation as the source code is open-sourced. Desktop-First Currently, MIVAIS is limited to desktop web-based mixed-initiative applications, and does not support VR/AR applications, nor is it tailored for mobile devices. Future work should address these modalities, as these are becoming more prominent over the years. Browser Compatibility Some features, such as our heart rate collector, use the Web Bluetooth API for data collection, which is not supported by all browsers. A workaround is to use our generic Sensor API to push additional sensor data. We are aware of this limitation, and future work will address broader browser support for all integrated features. 7.2. Opportunities & Next Steps Our efforts to provide a modular study framework for multi-agent, mixed-initiative VA systems reveal several directions for future work. Sharing Systems and Studies MIVAIS can be run by any individual locally or hosted. Researchers can use the framework to share their own research and make their results accessible and replicable by making use of MIVAIS’s modular structure. Automated Testing of User Studies With the recent advancements in technology and the framework of MIVAIS, we see great potential for researchers in reducing efforts for conducting pilot studies, as protocols like VACP (Stähle et al., 2026) might allow replacing human participants with AI agents to act on the stimulus in pilot studies, collect first data, and optimize the setup. Flexible Study Design MIVAIS allows future studies to directly manipulate VA environment states and agents without heavy modification of the underlying VA system. E.g., an AI agent can be swapped to a human WoZ agent by adapting the agent permissions and roles, allowing for easier comparison of wizard agents and real software agents. Human Behavior Cloning The large multi-variant data that can be collected through MIVAIS allows for capturing human behavior and interactions. The data can be used for behavioral cloning into software agents and then testing the learned behavior in the same or a different environment, as systems and tasks can easily be changed. 8. Conclusion In this work, we presented MIVAIS, a multi-agent Study Environment for mixed-initiative VA systems. By providing a modular Infrastructure to build mixed-initiative VA systems and a Study Environment that eases the setup, configuration, and deployment of multi-agent experiments, MIVAIS enables researchers to collect the relevant data for post-session analysis that captures insights on multi-agent collaboration and interaction in VA environments as well as user behavior data. This leverages evaluation of their implemented systems and approaches. We show that MIVAIS’s Infrastructure architecture allows building diverse mixed-initiative VA applications. Further, our expert case studies reveal that once a system is implemented on the MIVAIS Infrastructure, researchers can configure, deploy, and run studies in a short time, providing a comprehensive set of data collection methods supporting mixing of qualitative and quantitative studies. To foster its use within the community, our framework is available as open source software at : github.com/ETH-IVIA-Lab/MIVAIS. Acknowledgements. We are grateful to our expert case study participants for their valuable time and feedback. GenAI Usage Disclosure GenAI (Claude - Sonnet 5 and Opus 4.8) was partially used as a coding assistant during the implementation of MIVAIS. All system architecture design, expert case study protocols, analysis, and manuscript writing were conducted by the authors. References Abel et al. (2023) D. Abel, A. Barreto, B. V. Roy, D. Precup, H. P. van Hasselt, and S. Singh A definition of continual reinforcement learning. In Advances in Neural Information Processing Systems, NIPS ’23. Cited by: §4.1.1. Blascheck et al. (2016) T. Blascheck, M. John, K. Kurzhals, S. Koch, and T. Ertl VA2 2: A visual analytics approach for evaluating visual analytics applications. IEEE Trans. Vis. Comput. Graphics 22 (1), p. 61–70. External Links: Link, Document Cited by: §2.3. Braun and Clarke (2006) V. Braun and V. Clarke Using thematic analysis in psychology. Qualitative Research in Psychology 3 (2), p. 77–101. External Links: Document, Link Cited by: §4.2.4. Brooke (1996) J. Brooke SUS: A ’quick’ and ’dirty’ usability scale. In Usability Evaluation in Industry, p. 189–194. External Links: ISBN 9780748404605 Cited by: §4.2.2. Brooks (2012) J. Brooks Counterbalancing for serial order carryover effects in experimental condition orders. Psychological Methods 17 (4), p. 600–614. External Links: Document Cited by: §4.2.2. Chan et al. (2026) R. S. M. Chan, R. Sevastjanova, and M. El-Assady PleaSQLarify: visual pragmatic repair for natural language database querying. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, p. 1163:1–1163:18. External Links: Link, Document Cited by: §6.2.3. Chen et al. (2025) W. Chen, W. Tong, A. Case, and T. Zhang Dango: A mixed-initiative data wrangling system using large language model. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, p. 389:1–389:28. External Links: Link, Document Cited by: §2.1. Cockburn et al. (2020) A. Cockburn, P. Dragicevic, L. Besançon, and C. Gutwin Threats of a replication crisis in empirical computer science. Commun. ACM 63 (8), p. 70–79. External Links: Link, Document Cited by: §2.1. Cook et al. (2015) K. A. Cook, N. Cramer, D. J. Israel, M. Wolverton, J. Bruce, R. Burtner, and A. Endert Mixed-initiative visual analytics using task-driven recommendations. In IEEE Conf. on Visual Analytics Science and Technology, p. 9–16. External Links: Link, Document Cited by: §2.1. Cumbal et al. (2025) R. Cumbal, N. Calvo-Barajas, M. Escobar-Planas, A. Rouchitsas, and G. Castellano Visualizing confidence in delivery robots: insights from two online studies. In Proc. of the Extended Abstracts of the CHI Conf. on Human Factors in Computing Systems, p. 578:1–578:7. External Links: Link, Document Cited by: §2.3. Cutler et al. (2020) Z. Cutler, K. Gadhave, and A. Lex Trrack: A library for provenance-tracking in web-based visualizations. In IEEE Trans. Vis. Comput. Graphics, p. 116–120. External Links: Link, Document Cited by: §2.3. Cutler et al. (2025) Z. Cutler, L. Harrison, C. Nobre, and A. Lex Crowdsourced think-aloud studies. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, p. 207:1–207:23. External Links: Link, Document Cited by: §4.2.7. Cutler et al. (2026) Z. Cutler, J. Wilburn, H. Shrestha, Y. Ding, B. C. Bollen, K. A. Nadib, T. He, A. M. McNutt, L. Harrison, and A. Lex ReVISit 2: A full experiment life cycle user study framework. IEEE Trans. Vis. Comput. Graphics 32 (1), p. 13–23. External Links: Link, Document Cited by: §1, §2.3, §3, §3, §3, §3, §3, §4.2.1, §4.2.7, §5.2. de Leeuw et al. (2023) J. R. de Leeuw, R. A. Gilbert, and B. Luchterhandt JsPsych: enabling an open-source collaborative ecosystem of behavioral experiments. J. Open Source Softw. 8 (87), p. 5351. External Links: Link, Document Cited by: §2.3, §3, §3. Dellermann et al. (2019) D. Dellermann, P. Ebel, M. Söllner, and J. M. Leimeister Hybrid intelligence. 61 (5), p. 637–643. External Links: ISSN 1867-0202, Link, Document Cited by: §2.1. Dhanoa et al. (2025) V. Dhanoa, A. Wolter, G. M. León, H. Schulz, and N. Elmqvist Agentic visualization: extracting agent-based design patterns from visualization systems. IEEE Computer Graphics and Applications 45 (6), p. 89–100. External Links: Link, Document Cited by: §1, §2.2. Ding et al. (2023) Y. Ding, J. Wilburn, H. Shrestha, A. Ndlovu, K. Gadhave, C. Nobre, A. Lex, and L. Harrison ReVISit: supporting scalable evaluation of interactive visualizations. In IEEE Visualization and Visual Analytics (VIS), p. 31–35. External Links: Link, Document Cited by: §1, §2.3, §3, §5.2.3. El-Assady et al. (2019) M. El-Assady, F. Sperrle, O. Deussen, D. A. Keim, and C. Collins Visual analytics for topic model optimization based on user-steerable speculative execution. IEEE Trans. Vis. Comput. Graphics 25 (1), p. 374–384. External Links: Link, Document Cited by: §2.1. Ellemose et al. (2026) J. Ellemose, S. Wanner, D. Slijepcevic, L. Cesar, V. Y. F. Leung, W. Aigner, and N. Elmqvist The frustrometer: detecting user frustration in data visualization tasks using biomarkers and interaction patterns. CoRR abs/2606.13687. External Links: Link, Document, 2606.13687 Cited by: §3. Endert et al. (2014) A. Endert, M. S. Hossain, N. Ramakrishnan, C. North, P. Fiaux, and C. Andrews The human is the loop: new directions for visual analytics. J. Intell. Inf. Syst. 43 (3), p. 411–435. External Links: ISSN 0925-9902, Link, Document Cited by: §1. Ericsson and Simon (1993) K. A. Ericsson and H. A. Simon Protocol analysis: verbal reports as data. The MIT Press. External Links: ISBN 9780262272391, Document, Link Cited by: §4.2.4. Fekete et al. (2020) J. Fekete, J. Freire, and T. Rhyne Exploring reproducibility in visualization. IEEE Computer Graphics and Applications 40 (5), p. 108–119. External Links: Link, Document Cited by: §2.1. Gemini Team (2023) Gemini Team Gemini: A Family of Highly Capable Multimodal Models. CoRR abs/2312.11805. External Links: Document, Link, 2312.11805 Cited by: §5.1.3. Gotz et al. (2006) D. Gotz, M. X. Zhou, and V. Aggarwal Interactive visual synthesis of analytic knowledge. In IEEE Conf. on Visual Analytics Science and Technology, p. 51–58. External Links: Link, Document Cited by: §2.1. Ha et al. (2024) S. Ha, S. Monadjemi, and A. Ottley Guided by AI: navigating trust, bias, and data exploration in ai-guided visual analytics. Comput. Graph. Forum 43, p. 3:1–3:12. External Links: Link, Document Cited by: §3. Hart and Staveland (1988) S. G. Hart and L. E. Staveland Development of nasa-tlx (task load index): results of empirical and theoretical research. In Human Mental Workload, Advances in Psychology, Vol. 52, p. 139–183. External Links: ISSN 0166-4115, Document, Link Cited by: §4.2.2. Heer and Moritz (2024) J. Heer and D. Moritz Mosaic: An Architecture for Scalable & Interoperable Data Views. IEEE Trans. Vis. Comput. Graphics 30 (1), p. 436–446. External Links: Document, Link Cited by: §3. Holter and El-Assady (2024) S. Holter and M. El-Assady Deconstructing human-ai collaboration: agency, interaction, and adaptation. Comput. Graph. Forum 43 (3). External Links: Link, Document Cited by: §2.2, §3, §4. Holter et al. (2026) S. Holter, T. Stähle, A. Narechania, and M. El-Assady From interaction to intent: inferring user objectives from provenance logs. CoRR abs/2607.04501. External Links: Link, Document, 2607.04501 Cited by: §4.1.5. Hornbæk et al. (2014) K. Hornbæk, S. S. Sander, J. A. Bargas-Avila, and J. G. Simonsen Is once enough?: on the extent and content of replications in human-computer interaction. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, p. 3523–3532. External Links: Link, Document Cited by: §3. Horvitz (1999) E. Horvitz Principles of mixed-initiative user interfaces. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, p. 159–166. External Links: Link, Document Cited by: §1, §2.1, §3. Huang et al. (2026) O. Huang, M. Fatir, T. Luo, S. Suh, H. Subramonyam, and C. Nobre Narrative scaffolding: A narrative-first framework for data-driven sensemaking. In iui, p. 1112–1140. External Links: Link, Document Cited by: §3. Huang and Narechania (2026) Y. Huang and A. Narechania Facilitating proactive and reactive guidance for decision making on the web: A design probe with webseek. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, p. 800:1–800:20. External Links: Link, Document Cited by: §2.2. Iarygina et al. (2026) O. Iarygina, K. Hornbæk, and A. Mottelson On the computational reproducibility of human-computer interaction. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, p. 1449:1–1449:16. External Links: Link, Document Cited by: §3. Jakubovic et al. (2023) J. Jakubovic, J. Edwards, and T. Petricek Technical dimensions of programming systems. Art Sci. Eng. Program. 7 (3). External Links: Link, Document Cited by: §5.2, §5. Jiang and Odom (2018) S. D. Jiang and J. L. Odom Toward initiative decision-making for distributed human-robot teams. In Proc. Int. Conf. on Human-Agent Interaction, p. 286–292. External Links: Link, Document Cited by: §2.1. Ju et al. (2022) S. Ju, X. Yang, T. Barnes, and M. Chi Student-tutor mixed-initiative decision-making supported by deep reinforcement learning. In Artificial Intelligence in Education, Lecture Notes in Computer Science, Vol. 13355, p. 440–452. External Links: Link, Document Cited by: §2.1. Keim et al. (2008) D. A. Keim, G. L. Andrienko, J. Fekete, C. Görg, J. Kohlhammer, and G. Melançon Visual analytics: definition, process, and challenges. In Information Visualization - Human-Centered Issues and Perspectives, Lecture Notes in Computer Science, Vol. 4950, p. 154–175. External Links: Link, Document Cited by: §2.1. Krisam et al. (2026) C. Krisam, L. Schick, B. Tadson, G. Serafino, and M. Feick MindTide: designing a stress-adaptive to-do list. In Adjunct Proc. of the CHIWORK Symp. on Human-Computer Interaction for Work, p. 17:1–17:7. External Links: Link, Document Cited by: §3. Kuhlman et al. (2019) C. Kuhlman, D. Doherty, M. Nurbekova, G. Deva, Z. Phyo, P. Schoenhagen, M. V. Valkenburg, E. A. Rundensteiner, and L. Harrison Evaluating preference collection methods for interactive ranking analytics. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, p. 512. External Links: Link, Document Cited by: §3. Lazar et al. (2017) J. Lazar, J. Feng, and H. Hochheiser Research methods in human-computer interaction, 2nd edition. Morgan Kaufmann. External Links: Link, ISBN 978-0-12-805390-4 Cited by: §4.2.2. Lee et al. (2021) M. H. Lee, D. P. Siewiorek, A. Smailagic, A. Bernardino, and S. B. i Badia A human-ai collaborative approach for clinical decision making on rehabilitation assessment. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, p. 392:1–392:14. External Links: Link, Document Cited by: §2.1. Liu et al. (2007) D. Liu, A. Raja, and J. Vaidyanath TIBOR: A resource-bounded information foraging agent for visual analytics. In iat, p. 349–355. External Links: Link, Document Cited by: §2.1. Masai et al. (2021) K. Masai, A. Kobayashi, and T. Kimura Online study reveals the multimodal effects of discrete auditory cues in moving target estimation task. In Int. Conf. on Multimodal Interaction (ICMI), p. 740–744. External Links: Link, Document Cited by: §2.3. McNutt et al. (2025) A. M. McNutt, M. C. Stone, and J. Heer Mixing linters with guis: A color palette design probe. IEEE Trans. Vis. Comput. Graphics 31 (1), p. 327–337. External Links: Link, Document Cited by: §5.2. Monadjemi et al. (2023) S. Monadjemi, M. Guo, D. Gotz, R. Garnett, and A. Ottley Human-computer collaboration for visual analytics: an agent-based framework. Comput. Graph. Forum 42 (3), p. 199–210. External Links: Link, Document Cited by: §1, §2.2, §3, §4.1.1, §4. Monadjemi et al. (2025) S. Monadjemi, Y. Guo, K. Xu, A. Endert, and A. Crisan A scoping review of mixed initiative visual analytics in the automation renaissance. CoRR abs/2509.19152. External Links: Link, Document, 2509.19152 Cited by: §2.2. Narechania et al. (2025) A. Narechania, A. Endert, and A. R. Sinha Guidance source matters: how guidance from ai, expert, or a group of analysts impacts visual data preparation and analysis. In iui, p. 789–809. External Links: Link, Document Cited by: §3, §3, §3. Nguyen et al. (2016) P. H. Nguyen, K. Xu, A. Bardill, B. Salman, K. Herd, and B. L. W. Wong SenseMap: supporting browser-based online sensemaking through analytic provenance. In IEEE Conf. on Visual Analytics Science and Technology, p. 91–100. External Links: Link, Document Cited by: §4.1.5. Nobre et al. (2021) C. Nobre, D. Wootton, Z. Cutler, L. Harrison, H. Pfister, and A. Lex ReVISit: looking under the hood of interactive visualization studies. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, p. 25:1–25:13. External Links: Link, Document Cited by: §2.3. J. S. Olson and W. A. Kellogg (Eds.) (2014) J. S. Olson and W. A. Kellogg (Eds.) Ways of knowing in HCI. Springer. External Links: Link, Document, ISBN 978-1-4939-0377-1 Cited by: §6.3.3. Panwar and Collins (2018) P. Panwar and C. M. Collins Detecting negative emotion for mixed initiative visual analytics. In Proc. of the Extended Abstracts of the CHI Conf. on Human Factors in Computing Systems, External Links: Link, Document Cited by: §3. Pérez-Messina et al. (2024) I. Pérez-Messina, D. Ceneda, and S. Miksch Guided visual analytics for image selection in time and space. IEEE Trans. Vis. Comput. Graphics 30 (1), p. 66–75. External Links: Link, Document Cited by: §3. Quinlan (1993) J. R. Quinlan Combining Instance-Based and Model-Based Learning. In Machine Learning, Proc. Int. Conf., p. 236–243. External Links: Document, Link Cited by: §5.1.1. Radford et al. (2023) A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever Robust speech recognition via large-scale weak supervision. In Int. Conf. on Machine Learning, ICML, Proc. of Machine Learning Research, Vol. 202, p. 28492–28518. External Links: Link Cited by: §4.2.7. Reverberi et al. (2022) C. Reverberi, T. Rigon, A. Solari, C. Hassan, P. Cherubini, G. Antonelli, H. Awadie, S. Bernhofer, S. Carballal, M. Dinis-Ribeiro, A. Fernández-Clotett, G. F. Esparrach, I. Gralnek, Y. Higasa, T. Hirabayashi, T. Hirai, M. Iwatate, M. Kawano, M. Mader, A. Maieron, S. Mattes, T. Nakai, I. Ordas, R. Ortigão, O. O. Zúñiga, M. Pellisé, C. Pinto, F. Riedl, A. Sánchez, E. Steiner, Y. Tanaka, A. Cherubini, and GI Genius CADx Study Group Experimental evidence of effective human–AI collaboration in medical decision-making. Nature 12 (1). External Links: ISSN 2045-2322, Document Cited by: §2.1. Satyanarayan et al. (2017) A. Satyanarayan, D. Moritz, K. Wongsuphasawat, and J. Heer Vega-Lite: A Grammar of Interactive Graphics. IEEE Trans. Vis. Comput. Graphics 23 (1), p. 341–350. External Links: Document, Link Cited by: §3. Schemmer et al. (2023) M. Schemmer, A. Bartos, P. Spitzer, P. Hemmer, N. Kühl, J. Liebschner, and G. Satzger Towards effective human-ai decision-making: the role of human learning in appropriate reliance on AI advice. In Proc. Int. Conf. on Information Systems, ICIS, External Links: Link Cited by: §2.1. Setlur and Tory (2022) V. Setlur and M. Tory How do you converse with an analytical chatbot? revisiting gricean maxims for designing analytical conversational behavior. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, p. 29:1–29:17. External Links: Link, Document Cited by: §3. Sevastjanova et al. (2021) R. Sevastjanova, W. Jentner, F. Sperrle, R. Kehlbeck, J. Bernard, and M. El-Assady QuestionComb: A gamification approach for the visual explanation of linguistic phenomena through interactive labeling. ACM Trans. Interact. Intell. Syst. 11 (3-4), p. 19:1–19:38. External Links: Link, Document Cited by: §3. Sivaraman et al. (2025) V. Sivaraman, Z. Li, and A. Perer Divisi: interactive search and visualization for scalable exploratory subgroup analysis. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, p. 845:1–845:17. External Links: Link, Document Cited by: §1. Sperrle et al. (2023) F. Sperrle, D. Ceneda, and M. El-Assady Lotse: A Practical Framework for Guidance in Visual Analytics. IEEE Trans. Vis. Comput. Graphics 29 (1), p. 1124–1134. External Links: Document, Link Cited by: §3, §5.2.3. Sperrle et al. (2025) F. Sperrle, M. El-Assady, A. Arleo, and D. Ceneda A wizard of oz study of guidance strategies and dynamics. IEEE Trans. Vis. Comput. Graphics 31 (9), p. 4968–4982. External Links: Link, Document Cited by: §3. Sperrle et al. (2021a) F. Sperrle, M. El-Assady, G. Guo, R. Borgo, D. H. Chau, A. Endert, and D. A. Keim A survey of human-centered evaluations in human-centered machine learning. Comput. Graph. Forum 40 (3), p. 543–567. External Links: Link, Document Cited by: §4.2.1. Sperrle et al. (2021b) F. Sperrle, H. Schäfer, D. A. Keim, and M. El-Assady Learning contextualized user preferences for co-adaptive guidance in mixed-initiative topic model refinement. Comput. Graph. Forum 40 (3), p. 215–226. External Links: Link, Document Cited by: §1. Sperrle et al. (2021c) F. Sperrle, H. Schäfer, D. A. Keim, and M. El-Assady Learning contextualized user preferences for co-adaptive guidance in mixed-initiative topic model refinement. Comput. Graph. Forum 40 (3), p. 215–226. External Links: Link, Document Cited by: §2.1. Stähle et al. (2026) T. Stähle, P. F. Gyarmati, T. Spinner, R. Sevastjanova, D. Moritz, and M. El-Assady VACP: visual analytics context protocol. CoRR abs/2603.29322. External Links: Link, Document, 2603.29322 Cited by: §4.1.5, §7.2. Stähle et al. (2025) T. Stähle, M. J. op de Haar, S. Boyer, R. Sevastjanova, A. Narechania, and M. El-Assady A design space for intelligent agents in mixed-initiative visual analytics. CoRR abs/2512.23372. External Links: Link, Document, 2512.23372 Cited by: §1, §2.2, §3, §3, §3, §3, §3, §4.1.1, §4.1.2. Wall et al. (2018) E. Wall, S. Das, R. Chawla, B. Kalidindi, E. T. Brown, and A. Endert Podium: ranking data using mixed-initiative visual analytics. IEEE Trans. Vis. Comput. Graphics 24 (1), p. 288–297. External Links: Link, Document Cited by: item (3), §2.1, §5.1.1, §5.1. Wen et al. (2007) Z. Wen, M. X. Zhou, and V. Aggarwal Context-aware, adaptive information retrieval for investigative tasks. In iui, p. 122–131. External Links: Link, Document Cited by: §2.1. Wobbrock and Kientz (2016) J. O. Wobbrock and J. A. Kientz Research contributions in human-computer interaction. Interactions 23 (3), p. 38–44. External Links: Link, Document Cited by: §3, §4.2.1. Wongsuphasawat et al. (2016) K. Wongsuphasawat, D. Moritz, A. Anand, J. D. Mackinlay, B. Howe, and J. Heer Voyager: exploratory analysis via faceted browsing of visualization recommendations. IEEE Trans. Vis. Comput. Graphics 22 (1), p. 649–658. External Links: Link, Document Cited by: §2.1. Wongsuphasawat et al. (2017) K. Wongsuphasawat, Z. Qu, D. Moritz, R. Chang, F. Ouk, A. Anand, J. D. Mackinlay, B. Howe, and J. Heer Voyager 2: augmenting visual analysis with partial view specifications. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems, p. 2648–2659. External Links: Link, Document Cited by: item (3), §5.1.2, §5.1. Wu et al. (2022) A. Wu, Y. Wang, M. Zhou, X. He, H. Zhang, H. Qu, and D. Zhang MultiVision: designing analytical dashboards with deep learning based recommendation. IEEE Trans. Vis. Comput. Graphics 28 (1), p. 162–172. External Links: Link, Document Cited by: §1. Xu and Gao (2024) W. Xu and Z. Gao Applying HCAI in developing effective human-ai teaming: A perspective from human-ai joint cognitive systems. Interactions 31 (1), p. 32–37. External Links: Link, Document Cited by: §2.1. Yan et al. (2026) X. Yan, R. Sevastjanova, M. El-Assady, and B. Wang TopoAlign: topology-aware visual representation alignment. CoRR abs/2605.25541. External Links: Link, Document, 2605.25541 Cited by: §6.2.2. Zhao et al. (2026) Y. Zhao, X. Shu, L. Fan, L. Gao, Y. Zhang, and S. Chen ProactiveVA: proactive visual analytics with llm-based UI agent. IEEE Trans. Vis. Comput. Graphics 32 (1), p. 451–461. External Links: Link, Document Cited by: item (3), §2.2, §3, §5.1.3, §5.1.3, §5.1. Zhao et al. (2025a) Y. Zhao, J. Wang, L. Xiang, X. Zhang, Z. Guo, C. Turkay, Y. Zhang, and S. Chen LightVA: lightweight visual analytics with llm agent-based task planning and execution. IEEE Trans. Vis. Comput. Graphics 31 (9), p. 6162–6177. External Links: Document, ISSN 1941-0506 Cited by: §2.1. Zhao et al. (2025b) Y. Zhao, Y. Zhang, Y. Zhang, X. Zhao, J. Wang, Z. Shao, C. Turkay, and S. Chen LEVA: using large language models to enhance visual analytics. IEEE Trans. Vis. Comput. Graphics 31 (3), p. 1830–1847. External Links: Link, Document Cited by: §2.1. Zhou et al. (2025) S. Zhou, B. Li, X. Chen, Z. Tu, Y. Wang, Y. Xiang, and T. Zhang HEPHA: A mixed-initiative image labeling tool for specialized domains. In iui, p. 810–826. External Links: Link, Document Cited by: §3. Appendix A Usage Scenario To demonstrate our framework’s practical value, we outline a typical workflow for designing, implementing, and evaluating a mixed-initiative VA application/approach. The implementation pipeline then follows a multi-stage process: System Conceptualization: The developer starts by outlining the core data model, relevant machine learning components, and drafting the VA interface. Crucially, this stage involves defining the agents and mapping their specific capabilities and action space to establish a comprehensive system prototype. Backend Orchestration: The conceptual draft is implemented in the MIVAIS backend. The developer defines the central World State by creating the environment schema, specifying state keys, and setting their initial values. Next, the agent configuration is declared by assigning unique identifiers, defining human and software agents, and setting strict observation and action permissions. The developer also configures event listeners, subscribes agents to communication channels, and sets hyperparameters for their behavior. Finally, the software agents’ logic is implemented by extending the MIVAIS base-agent class, operationalizing observations and actions, and structuring inter-agent communication. Frontend Integration: The visual interface is wired through a dedicated context using the MIVAIS Context Provider. The frontend communicates with the backend via the standardized MIVAIS WebSocket API. At runtime, user identities are resolved and retrieved through the visual interface or URL parameters. Study Deployment and Post-Session Analysis: After completing the system, the researcher uses MIVAIS’s Study Environment to configure the study sequence and specify analytical task parameters, then runs the setup in a controlled lab study. The framework then supports detailed post-session analysis, enabling systematic review of multimodal interaction trajectories and inter-agent communication, with direct annotation of recorded sessions for easy sharing of findings with stakeholders. Sharing Application and Study Setup for Replication: Finally, the researcher can open-source the mixed-initiative VA application and share the study setup so the research community can replicate the work using the MIVAIS Study Environment. Researchers can also share specific session replays via public links with reviewers and the community. Appendix B Semi-Structured Interview Protocol B.1. Introduction & Consent • Briefly restate case study purpose, confidentiality, and recording consent. • Confirmation of the expert’s role, background, and prior experience with Visual Analytics, mixed-initiative systems, or AI-human collaboration tooling. (1) Could you briefly describe your background in Visual Analytics, HCI, or AI systems research/development? B.2. Mixed-Initiative System-Building Experience (2) Can you walk me through the system you built with our MIVAIS framework? What was its purpose, and what AI/human collaboration scenario did it support? (3) What was your overall process for implementing it — what did you start with, and how did you proceed? (4) Were there parts of the implementation that felt straightforward? Which parts, and why? (5) Were there parts that felt difficult, confusing, or frustrating? Could you describe a specific moment where you got stuck? (6) How did our MIVAIS framework’s approach address state synchronization (between AI agents and human users), compared to how you would normally implement this from scratch? B.3. Criteria-Based Assessment For each criterion below we ask the participant to give answers to the questions, and then probe the reasoning behind their response. Expressiveness (7) To what extent did the MIVAIS framework allow you to implement the interaction patterns and AI-human collaboration behaviors you envisioned? (8) Were there any application and study designs or behaviors you wanted to implement but felt constrained or unable to express using MIVAIS? Creativity Support (9) Did working with MIVAIS influence the design decisions you made, positively or negatively? Did it open up ideas you wouldn’t have otherwise considered, or did it push you toward more conventional solutions? (10) Can you think of an instance where the MIVAIS framework either enabled or limited a creative design choice? Flexibility (11) How well did our software accommodate adapting or modifying your system and study design as your ideas evolved during development? Efficiency (12) Compared to building a similar application and user study without MIVAIS, how would you describe the development time and effort? (13) Which specific features, if any, saved you the most time or effort? Which, if any, slowed you down? Usability (14) How would you describe your overall experience interacting with the MIVAIS’s API, and functionalities? Learnability (15) How long did it take you to understand the core concepts and get a basic system and study running? (16) What aspects of MIVAIS were easy to learn, and what aspects required more effort or external help (e.g., documentation, trial and error)? Integration into Existing Workflows (17) How well did our MIVAIS framework fit into your existing development environment, tools, and practices (e.g., version control, testing, deployment)? (18) Would you be able to integrate the MIVAIS framework into an existing project, or does it require building from scratch? Please explain. B.4. Logging & State-Recording Feature (19) Did the logged data and analysis meet your needs for analyzing application state and user/agent interactions? What, if anything, was missing or required manual post-processing? (20) Compared to how you would normally implement logging in a research prototype, how does this compare in terms of effort and data quality? B.5. Comparative & Reflective Questions (21) If you had to build the same system and study again, would you choose to use MIVAIS again? Why or why not? (22) How does MIVAIS compare to other tools or frameworks you’ve used for similar purposes (if any)? (23) Are there scenarios or application types you think the MIVAIS framework would NOT be well-suited for? B.6. Improvement Suggestions & Closing (24) If you could change or add one thing to MIVAIS, what would it be? (25) Is there anything about your experience that we haven’t covered but that you think is important to share? Closing: • Thank you for participating in our expert case study. • Are there any additional questions on your side for our research team? Appendix C MIVAIS Domain Specific Language - Grammar C.1. Agent Registry - Role and Permission Definition Note on grammar notation: The ellipsis notation (…StructureName) is used as a spread operator to denote the inline expansion and inclusion of all fields defined within the referenced StructureName. AgentsConfig = agents::=AgentEntry[]users::=UserRole[]roles::=roleName→RoleDef \ array[]rclagents&::=&AgentEntry[]\\ users&::=&UserRole[]\\ roles&::=&\\,roleName \,\ array . AgentEntry = agent_id::=identifierclass::=stringenabled::=booleanrole::=string…PermissionFieldsparams::=string→JSONobservation::=Observationdescription::=string \ array[]rclagent\_id&::=&identifier\\ class&::=&string\\ enabled&::=&boolean\\ role&::=&string\\ …\ PermissionFields&&\\ params&::=&\\,string \,\\\ observation&::=&Observation\\ description&::=&string array . Observation = strategy::=watch∣poll∣subscribewatches::=string[] \ array[]rclstrategy&::=&watch \\ watches&::=&string[] array . UserRole = role::=identifier…PermissionFields \ array[]rclrole&::=&identifier\\ …\ PermissionFields&& array . RoleDef = description::=string…PermissionFields \ array[]rcldescription&::=&string\\ …\ PermissionFields&& array . PermissionFields = can_read::=string[]can_write::=string[]communication_bus_publish_topics::=string[]communication_bus_subscribe_topics::=string[] \ array[]rclcan\_read&::=&string[]\\ can\_write&::=&string[]\\ communication\_bus\_publish\_topics&::=&string[]\\ communication\_bus\_subscribe\_topics&::=&string[] array . Semantics not expressible above: agent_id must be unique across the list; class must resolve in the application’s agent-class map or the entry is skipped with a warning C.2. Study Configuration Note on grammar notation: The ellipsis notation (…StructureName) is used as a spread operator to denote the inline expansion and inclusion of all fields defined within the referenced StructureName. Study = id::=identifiername::=stringversion::=integermode::=single-participant|multi-participantparticipants_required::=integerconsent_text_md::=markdownva::=VASystemsroles::=Role[]recording::=Recordingui::=UiConfigadvance_policy::=all∣first∣adminblock_order::=declared|random∣round_robin∣latin_squareblock_orders::=identifier[][]parameters::=string→JSONcompletion_redirect_url::=stringcompletion_code::=stringmetrics::=Metric[]blocks::=Block[]tasks::=TaskRef[] \ array[]rclid&::=&identifier\\ name&::=&string\\ version&::=&integer\\ mode&::=&single-participant -participant\\ participants\_required&::=&integer\\ consent\_text\_md&::=&markdown\\ va&::=&VASystems\\ roles&::=&Role[]\\ recording&::=&Recording\\ ui&::=&UiConfig\\ advance\_policy&::=&all \\ block\_order&::=&declared \_robin \_square\\ block\_orders&::=&identifier[\,][\,]\\ parameters&::=&\\,string \,\\\ completion\_redirect\_url&::=&string\\ completion\_code&::=&string\\ metrics&::=&Metric[]\\ blocks&::=&Block[]\\ tasks&::=&TaskRef[] array . blocks and tasks are mutually exclusive – give one or the other; a bare tasks list folds into one implicit block. VASystems = va_systems::=identifier→VAConfigprimary_va_system::=identifier \ array[]rclva\_systems&::=&\\,identifier \,\\\ primary\_va\_system&::=&identifier array . VAConfig = system::=identifierexternal::=trueiframe_url::=stringhealth_check_url::=stringenv::=string→stringvariants::=identifier→VAVariantrecording_dir_override::=stringtoken_env::=stringtoken_param::=stringroom_per_session::=booleanroom_param::=string \ array[]rclsystem&::=&identifier\\ external&::=&true\\ iframe\_url&::=&string\\ health\_check\_url&::=&string\\ env&::=&\\,string \,\\\ variants&::=&\\,identifier \,\\\ recording\_dir\_override&::=&string\\ token\_env&::=&string\\ token\_param&::=&string\\ room\_per\_session&::=&boolean\\ room\_param&::=&string array . VAVariant = description::=stringfrontend_id::=stringbackend_id::=stringspawn_extra_args::=stringmivais_config::=string→JSON \ array[]rcldescription&::=&string\\ frontend\_id&::=&string\\ backend\_id&::=&string\\ spawn\_extra\_args&::=&string\\ mivais\_config&::=&\\,string \,\ array . Role = id::=identifiername::=stringdescription::=stringcapacity::=integerselectable::=booleancolor::=stringagent_id::=identifierwizard_panel::=boolean \ array[]rclid&::=&identifier\\ name&::=&string\\ description&::=&string\\ capacity&::=&integer\\ selectable&::=&boolean\\ color&::=&string\\ agent\_id&::=&identifier\\ wizard\_panel&::=&boolean array . agent_id names the MIVAIS Infrastructure role. wizard_panel enables an optional Wizard-of-Oz control panel. Recording = audio::=none∣per_participant∣single_micvideo::=none|screentranscribe::=booleanwhisper_model::=stringwhisper_language::=stringretention_days::=integerbiometric::=booleanexternal_sensor::=boolean \ array[]rclaudio&::=&none \_participant \_mic\\ video&::=&none \\ transcribe&::=&boolean\\ whisper\_model&::=&string\\ whisper\_language&::=&string\\ retention\_days&::=&integer\\ biometric&::=&boolean\\ external\_sensor&::=&boolean array . biometric: BLE heart-rate strap paired in the participant’s own browser via Web Bluetooth. external_sensor: enables POST /ingest/sensor-chunk for non-browser sources (token-authenticated). UiConfig = show_progress_bar::=booleanshow_timer::=booleantimer_position::=top_right|top_left∣sidebar∣hiddentask_description_position::=top∣left∣overlaytask_steps_visible::=booleantask_steps_position::=right∣left∣hiddenshow_presence::=booleantheme::=light∣dark∣auto \ array[]rclshow\_progress\_bar&::=&boolean\\ show\_timer&::=&boolean\\ timer\_position&::=&top\_right \_left \\ task\_description\_position&::=&top \\ task\_steps\_visible&::=&boolean\\ task\_steps\_position&::=&right \\ show\_presence&::=&boolean\\ theme&::=&light array . Block = id::=identifiername::=stringdescription_md::=markdownrandomize_within::=booleantasks::=TaskRef[] \ array[]rclid&::=&identifier\\ name&::=&string\\ description\_md&::=&markdown\\ randomize\_within&::=&boolean\\ tasks&::=&TaskRef[] array . TaskRef ::=::= identifier∣RefWithOverrides∣Taskidentifier A bare identifier resolves to tasks/<identifier>.yaml; RefWithOverrides is a shared fragment with a shallow merge; Task is an inline task object. RefWithOverrides = $ref::=string…TaskFieldOverride \ array[]rcl\$ref&::=&string\\ …\ TaskFieldOverride&& array . Metric = id::=identifierlabel::=stringkind::=count∣rate∣latency∣ratio∣pythontask::=identifier…MetricKindFields \ array[]rclid&::=&identifier\\ label&::=&string\\ kind&::=&count \\ task&::=&identifier\\ …\ MetricKindFields&& array . Metric allows to measure additional (behavioral) metrics when agents interact with a VA system. MetricKindFields ::=::= MatchFields|LatencyFields∣RatioFields∣PythonFieldsMatchFields MatchFields = match::=EventMatch \ array[]rclmatch&::=&EventMatch array . (kind: count | rate) LatencyFields = from::=EventMatchto::=EventMatchaggregate::=mean|median∣min∣maxfirst_only::=boolean \ array[]rclfrom&::=&EventMatch\\ to&::=&EventMatch\\ aggregate&::=&mean \\ first\_only&::=&boolean array . (kind: latency) RatioFields = numerator::=EventMatchdenominator::=EventMatch \ array[]rclnumerator&::=&EventMatch\\ denominator&::=&EventMatch array . (kind: ratio; value = n/(n+d)) PythonFields = module::=stringparams::=string→JSON \ array[]rclmodule&::=&string\\ params&::=&\\,string \,\ array . (kind: python script) EventMatch = event_type::=stringsource::=vasystem|studiometa::=string→JSON \ array[]rclevent\_type&::=&string\\ source&::=&vasystem \\ meta&::=&\\,string \,\ array . C.3. Task Configuration Note on grammar notation: The ellipsis notation (…StructureName) is used as a spread operator to denote the inline expansion and inclusion of all fields defined within the referenced StructureName. Task = id::=identifiertype::=TaskType…BaseFields…TypeFields \ array[]rclid&::=&identifier\\ type&::=&TaskType\\ …\ BaseFields&&\\ …\ TypeFields&& array . id defaults to the filename. TaskType::=info_screenθ|single_choiceθ∣multi_choiceθ∣likertθ∣sliderθ|number_inputθ∣free_textθ∣VA_interactionθ array[]rclTaskType&::=&info\_screen_θ \_choice_θ \_choice_θ _θ\\ & &slider_θ \_input_θ \_text_θ \_interaction_θ array θ marks that task type carries its own extra fields, listed under “Per-type fields” below. BaseFields = prompt_md::=markdownprompt_md_by_role::=identifier→markdowntimer_seconds::=integeroptional::=booleantags::=string[]task_steps::=TaskStep[]ground_truth::=GroundTruthwith_va::=booleanva_system::=identifierworld_state::=WorldStateadvance::=AdvanceOverride \ array[]rclprompt\_md&::=&markdown\\ prompt\_md\_by\_role&::=&\\,identifier \,\\\ timer\_seconds&::=&integer\\ optional&::=&boolean\\ tags&::=&string[]\\ task\_steps&::=&TaskStep[]\\ ground\_truth&::=&GroundTruth\\ with\_va&::=&boolean\\ va\_system&::=&identifier\\ world\_state&::=&WorldState\\ advance&::=&AdvanceOverride array . TaskStep = id::=identifierlabel::=stringrequired::=booleanmode::=manual∣auto∣eitherauto_check_on::=AutoCheck \ array[]rclid&::=&identifier\\ label&::=&string\\ required&::=&boolean\\ mode&::=&manual \\ auto\_check\_on&::=&AutoCheck array . AutoCheck = event_type::=stringcount::=integermeta::=string→JSON \ array[]rclevent\_type&::=&string\\ count&::=&integer\\ meta&::=&\\,string \,\ array . GroundTruth = type::=exact∣set_match∣ordered_match∣regex∣customvalue::=JSONpartial_credit::=boolean \ array[]rcltype&::=&exact \_match \_match \\ value&::=&JSON\\ partial\_credit&::=&boolean array . WorldState = mode::=continue|replaceinitial::=string→JSONset::=string→JSONby_role::=identifier→WorldState \ array[]rclmode&::=&continue \\ initial&::=&\\,string \,\\\ set&::=&\\,string \,\\\ by\_role&::=&\\,identifier \,\ array . initial is the full replace-mode scenario. set is a delta on top of the current state. Both are written through the Study Gateway at task start. AdvanceOverride = policy::=all∣first∣admintimeout_seconds::=integer \ array[]rclpolicy&::=&all \\ timeout\_seconds&::=&integer array . InfoFields = body_md::=markdowncontinue_label::=stringmin_view_seconds::=integer \ array[]rclbody\_md&::=&markdown\\ continue\_label&::=&string\\ min\_view\_seconds&::=&integer array . SingleChoiceFields = options::=Option[] \ array[]rcloptions&::=&Option[] array . MultiChoiceFields = options::=Option[]min_selected::=integermax_selected::=integer \ array[]rcloptions&::=&Option[]\\ min\_selected&::=&integer\\ max\_selected&::=&integer array . Option = id::=identifierlabel::=string \ array[]rclid&::=&identifier\\ label&::=&string array . LikertFields = scale::=LikertScaleitems::=LikertItem[]randomize_items::=boolean \ array[]rclscale&::=&LikertScale\\ items&::=&LikertItem[]\\ randomize\_items&::=&boolean array . LikertScale = min::=integermax::=integermin_label::=stringmax_label::=stringstep::=integer \ array[]rclmin&::=&integer\\ max&::=&integer\\ min\_label&::=&string\\ max\_label&::=&string\\ step&::=&integer array . LikertItem = id::=identifierlabel::=stringdescription::=stringreverse::=boolean \ array[]rclid&::=&identifier\\ label&::=&string\\ description&::=&string\\ reverse&::=&boolean array . SliderFields = min::=numbermax::=numberstep::=numbermin_label::=stringmax_label::=stringunit::=stringdefault_value::=numbershow_value::=boolean \ array[]rclmin&::=&number\\ max&::=&number\\ step&::=&number\\ min\_label&::=&string\\ max\_label&::=&string\\ unit&::=&string\\ default\_value&::=&number\\ show\_value&::=&boolean array . NumberFields = min::=numbermax::=numberstep::=numberunit::=stringplaceholder::=string \ array[]rclmin&::=&number\\ max&::=&number\\ step&::=&number\\ unit&::=&string\\ placeholder&::=&string array . FreeTextFields = min_chars::=integermax_chars::=integerplaceholder::=stringrows::=integer \ array[]rclmin\_chars&::=&integer\\ max\_chars&::=&integer\\ placeholder&::=&string\\ rows&::=&integer array . VAInteractionFields = variant::=identifierparameters::=string→JSONmivais_config::=string→JSONanswer::=VAAnswer \ array[]rclvariant&::=&identifier\\ parameters&::=&\\,string \,\\\ mivais\_config&::=&\\,string \,\\\ answer&::=&VAAnswer array . (variant default "default") VAAnswer = type::=capture_state|text∣choice∣noneworld_state_key::=stringrequired::=boolean \ array[]rcltype&::=&capture\_state \\ world\_state\_key&::=&string\\ required&::=&boolean array . Semantics not expressible above: ground_truth applies only to answerable types.