Paper deep dive
Large-Scale User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution
Rafael Ferreira, Diogo Tavares, Diogo Glória-Silva, David Semedo, João Magalhães
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/13/2026, 5:32:01 AM
Summary
This paper presents a large-scale analysis of user behavior in Conversational Task Assistants (CTAs) using real-world data from the TWIZ-v2 system deployed on Alexa. The study examines interaction flows, user intents, conversational traits, and satisfaction factors across thousands of users performing cooking and DIY tasks. Key findings include the dominance of task grounding over execution, the prevalence of non-cooperative behavior in real-world settings compared to crowdsourced data, and the strong correlation between task completion and user satisfaction.
Entities (10)
Relation Signals (8)
Wizard of Tasks → isa → Crowdsourced Dataset
confidence 95% · we compare our real user data with the crowdsourced Wizard of Tasks dataset
TWIZ-v2 → supports → Cooking
confidence 95% · The system supports manual, multi-step tasks in the cooking and DIY domains
TWIZ-v2 → supports → DIY
confidence 95% · The system supports manual, multi-step tasks in the cooking and DIY domains
TWIZ-v2 → uses → Multimodal Interaction
confidence 95% · provides structured task guidance via multimodal interaction (voice, text, images, and video)
TWIZ-v2 → wasdeployedin → Alexa TaskBot Challenge
confidence 95% · we analyze user interactions collected in the context of the Alexa TaskBot Challenge... TWIZ-v2
Task Progression → correlatespositivelywith → User satisfaction
confidence 94% · The strongest positive correlations with rating are task-related: Task Progression (0.26)
Task Completion → correlatespositivelywith → User satisfaction
confidence 93% · users who complete tasks (4.17) ... highest ratings
Non-Cooperative Behavior → ismoreprevalentin → Real-world Data
confidence 92% · uncooperative behavior ... appearing in ≈ 17% of interactions while being mostly absent from crowdsourced datasets
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Conversational Task Assistants (CTAs) are multimodal dialogue systems that support users in complex real-world tasks such as cooking and DIY through voice, text, image, and video interactions. Prior user studies have focused on controlled settings, leaving limited understanding of real-world CTA usage at scale. In this work, we present a large-scale study of CTA usage based on thousands of users in-the-wild. Our large-scale real-world data analysis unveils new understandings of (i) user-CTA interaction flows, (ii) user intents, (iii) user conversational traits, and (iv) behavioral factors associated with user satisfaction. Our findings reveal key opportunities for future research in CTAs, particularly in user interaction design and task engagement, concluding with concrete design guidelines.
Tags
Links
- Source: https://arxiv.org/abs/2609.07594v1
- Canonical: https://arxiv.org/abs/2609.07594v1
Trouble viewing inline? Open PDF directly →
Full Text
34,352 characters extracted from source content.
Expand or collapse full text
Large-Scale User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution Rafael Ferreira, Diogo Tavares, Diogo Glória-Silva, David Semedo, and João Magalhães NOVA University of Lisbon, Portugal, NOVA LINCS, Portugal rah.ferreira, dc.tavares, dmgc.silva@campus.fct.unl.pt df.semedo, jmag@fct.unl.pt Abstract. Conversational Task Assistants (CTAs) are multimodal di- alogue systems that support users in complex real-world tasks such as cooking and DIY through voice, text, image, and video interactions. Prior user studies have focused on controlled settings, leaving limited under- standing of real-world CTA usage at scale. In this work, we present a large-scale study of CTA usage based on thousands of users in-the-wild. Our large-scale real-world data analysis unveils new understandings of (i) user-CTA interaction flows, (i) user intents, (i) user conversational traits, and (iv) behavioral factors associated with user satisfaction. Our findings reveal key opportunities for future research in CTAs, particu- larly in user interaction design and task engagement, concluding with concrete design guidelines. Keywords: Conversational Assistants· User Analysis· Task Execution. 1 Introduction Supporting users through real-world tasks such as cooking and DIY remains a challenge for conversational AI. Unlike traditional task-oriented dialogue systems that focus on transactional interactions such as booking or recommendation [2, 24,32], Conversational Task Assistants (CTAs) are multimodal systems designed to support open-ended, mixed-initiative interactions across multi-step tasks [1, 8,13,27]. Despite growing interest in CTAs, most existing work evaluates these systems in controlled environments or crowdsourced settings [4,13,16]. While useful for controlled experimentation, these settings often fail to capture the variability and dynamics of real users in deployed systems [28, 30]. As a result, how users interact with tasks in real-world CTA deployments remains understudied. In this work, we address this gap through a large-scale user study of TWIZ- v2 [12], a CTA deployed in a production environment that provides structured task guidance via multimodal interaction (voice, text, images, and video). In particular, we analyze user interactions collected in the context of the Alexa TaskBot Challenge [1], where the system was available to the entire U.S. Alexa user base in a real-world setting. This setting enables observation of natural user arXiv:2609.07594v1 [cs.HC] 7 Sep 2026 2Ferreira, R. et al. behavior at scale, including task discovery, step-by-step interaction patterns, and task engagement. Additionally, a subset of these interactions includes 1–5 user ratings, enabling analysis of satisfaction signals. Therefore our goal is to understand how users interact with CTAs in the real- world, as shown in Figure 1. Specifically, our contributions are as follows: (1) we present a large-scale analysis of user–CTA interaction flows, examining how users initiate and progress through various stages; (2) we analyze user intents across conversations; (3) we study user conversational traits and compare real- world interactions with existing crowdsourced CTA datasets; and (4) we identify behavioral signals associated with user satisfaction in CTA systems. Our findings reveal opportunities to improve CTAs, particularly in handling uncooperative user behavior, improving task discovery, and supporting sustained engagement during task execution. Overall, this analysis provides concrete in- sights for designing more effective multimodal conversational task assistants. 2 Related Work Conversational Task Assistants. To support research in the CTA domain, crowdsourced datasets have been introduced. In particular, Wizard of Tasks [8] and Task2Dial [27] provide task-centric dialogues addressing challenges such as intent recognition and commonsense reasoning. However, these datasets do not fully capture the variability of real-world usage, highlighting a gap between crowdsourced and deployed systems [28]. Among efforts to collect real-world CTA data, the Alexa TaskBot Chal- lenge [1] focuses on CTA for cooking and DIY tasks. The challenge emphasizes robustness in real user conditions, requiring systems to handle multimodal data. Notable systems include GRILLBot [13], which demonstrated the effectiveness of hybrid architectures combining LLMs with specialized components for task guidance. Other systems, such as our CTA TWIZ [12] and ISABEL [26], further explore multimodal task assistance by integrating text, images, and video to pro- vide context-aware support. However, no fine-grain analysis of these interactions has been performed. User Behavior with CTAs. Prior work highlights the importance of natural, proactive conversational strategies [4], as well as the role of multimodal assis- tants in improving user autonomy and task understanding [31]. User studies have also identified limitations, especially in delivering context-aware and timely re- sponses, motivating the need for more adaptive dialogue strategies [14,16]. Visual aids also have a critical role in task comprehension [6], while also re- ducing cognitive load during interactions [16]. This is particularly important for voice-based systems, where users rely heavily on working memory, motivating the adaptation of content for spoken interfaces [15]. These challenges are fur- ther amplified in noisy, real-world environments where ASR errors are common, underscoring the need for resilient, error-tolerant design [21]. User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution3 How can I help you today? Show me popular recipes! This recipe is a classic! When ready say "Start". Let's start! Step 1: [Reads Step] How much butter?I think it is right here! Let's finish! How would you rate this interaction? Video InteractionTask Navigation [...] How to add the pasta? You will need 2 tbsps. Question I rate ... Rating Task Grounding Greeting, Search, Task Selection, Task Preview Task Execution Step-by-Step Guidance, Q&A, Fun Facts, Multimodal Interaction [...] Search Fig. 1. Illustrative conversation between a user and our CTA system showcasing the multi-stage nature of the interaction and its various conversation sub-flows. Building on these findings, our work goes beyond controlled studies to analyze large-scale, real-world user interactions, providing insights into CTA usage and behavior under everyday deployment conditions. 3 Designing CTAs for Effective Task Guidance In this section, we briefly introduce TWIZ-v2 1 , the multimodal CTA that en- abled the data collection for this user study. The system is designed around three principles: (1) modular architecture, (2) multimodal integration, and (3) context-aware responses. 3.1 Modular System Architecture To support complex CTA interactions, the system is organized into modular components for managing dialogue flow and task content, as illustrated in Fig- ure 2. The system is built using the Cobot [17] framework, where a serverless function orchestrates dialogue management and access to other modules. Con- versation history and task search are handled by a database and a search system, respectively. Dialogue Management. The dialogue manager follows an event-driven, stage- based state machine that tracks both dialogue and task progress. Figure 3 il- lustrates the interaction stages and their roles which are organized into two macro-stages, each composed of several sub-stages: 1 See [12] for comprehensive implementation details of all system components. 4Ferreira, R. et al. Heuristics Classifier Model Phonemes LLM Offensive Detection Model Intent Detection Task Search Data Layer AI Modules Dialogues Profiles User Historical Data Alexa Recipe Tasks DIY Tasks Task Knowledge-bases User Dialogue Manager Fig. 2. The system’s modular architec- ture. 1. Greeting The system greets the user and offers task suggestions. 2. Task Search3. Task Overview 4. Task Steps5. Task Completed User searches and browses potential tasks. The user selects a task and views its description. The user proceeds through the task’s steps. User completes the task and ends the interaction. Task Grounding Task Execution Fig. 3. Dialogue manager interaction stages and sub-stages. – Task Grounding, which establishes context and supports task selection, consists of the Greeting, Task Search, and Task Overview sub-stages. – Task Execution, which guides users through the selected task, consists of the Task Steps and Task Completed sub-stages. This structure provides clear interaction guidance and helps users remain aware of their current position within the task flow. To support flexible interactions, the system includes sub-flows for Q&A and multimodal interaction, allowing users to temporarily diverge from the main flow while preserving dialogue continuity as shown in the example of Figure 1. Intent Detection. For each user utterance, the system identifies the user’s intent and uses it to drive system behavior (full intent set in Figure 5). Here, accuracy is critical as ASR errors are common in voice-based systems [21]. To improve robustness, the system combines multiple complementary signals us- ing a voting-style decision mechanism, integrating a BERT-based neural intent classifier, phoneme-level similarity signals [22], and rule-based heuristics. Task Suggestions and Search. To facilitate discoverability [18], the system proactively provides voice and on-screen task suggestions during the interaction, while also supporting free-form search. Task retrieval is implemented using a Hy- brid Search strategy combining text and semantic search. Task content is sourced from WikiHow (DIY) and Whole Foods Market (recipes), covering thousands of diverse tasks across several domains. User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution5 3.2 Designing CTAs for Multimodal Dialogues Given the complexity of tasks, clear and structured guidance is essential for effectively supporting users throughout task execution. To facilitate this, the system incorporates several mechanisms. Adaptive Task Chunking. To reduce cognitive load, task instructions are broken into shorter steps based on length and formatting cues, omitting non- essential information [15]. This is particularly important in voice-only interac- tions, where users rely on their working memory [9]. UI and Navigation. Users interact via voice in audio-only settings or with combined voice and touch in multimodal environments. Contextual on-screen elements (e.g. images and videos) are synchronized with each dialogue stage, and the UI adopts a design that emphasizes clarity and ease of use (Figure 1). Interaction with Multimodal Content. Users can actively engage with mul- timodal content. Task-related questions are answered using an LLM-based ap- proach [5] prompted with the current dialogue context. Users can also request fun facts [29] or control video playback. Video control retrieves relevant video frames by computing the similarity of user queries against cross-modal textual and visual embeddings of keyframes [25]. 3.3 Contextual Dialogue and Response Design Response design balances clarity and engagement by controlling utterance length and variation. Responses are kept concise to reduce cognitive load [16], com- mands are placed at the end to aid recall [18], and phrasing diversity is introduced to reduce repetition. Responses are generated using a combination of templates and an LLM, with prompts designed to produce concise and contextual outputs. To ensure safe interactions, the system detects sensitive topics using heuris- tics and a RoBERTa-based [20] offensive language classifier. When such topics arise, predefined safety responses are issued [1]. For unrecognized requests, the system performs conversation recovery by providing stage-aware contextual ex- planations and suggesting alternative actions [18]. Generative responses further improve recovery by producing natural, context-aware replies. 4 Large-scale User Interaction Analysis In this section, we analyze dialogues with TWIZ-v2 from multiple perspectives, including: (1) user-CTA interaction flows, (2) user intent distribution, (3) user conversational traits, and (4) factors driving user satisfaction. 6Ferreira, R. et al. GreetingSearchTask OverviewTask StepsTask Completed Stage 0 20 40 60 80 100 120 Percentage (%) Task GroundingTask Execution 100.0% 90.8% 66.8% 33.4% 6.5% 24.0% 26.8% 17.7% 29.7% 1.7% Dialogues reaching stage Turns in each stage Fig. 4. Percentage of dialogues reaching a stage and Percentage of turns in each stage. Data Collection and Filtering Our analysis considers interactions with TWIZ- v2 [12] over a 12-month period. The system supports manual, multi-step tasks in the cooking and DIY domains via multimodal interactions (voice and visual) and is available to all U.S. Alexa users. This setup enables the collection of in-the- wild interactions reflecting genuine user behavior. At the end of the interaction, users are prompted to optionally rate the conversation on a 1–5 Likert scale, providing feedback for system evaluation. To focus on meaningful interactions, we only consider dialogues with at least three turns and removed sessions where over 10% of turns were unrelated to CTA functionality (e.g., music, or smart home control). The resulting data includes 30k conversations, of which over 5k (16.7%) received user ratings, a positive conversion rate enabled by explicit feedback request at the end of the interaction. To the best of our knowledge, this represents the largest real-world CTA dataset. Device and Domain. Conversations are from 92% unique users, with 91% on screen-equipped devices and 9% on voice-only devices, showing the predominance of multimodal interactions and capturing a broad spectrum of user behaviors. Regarding domain selection, among users who select a task (56% of the total), 54% engage with recipes and 46% with DIY tasks, highlighting diverse user interests and the importance of strong performance in both domains. 4.1 User Progression Through Dialogue Stages As the system adopts a stage-based interaction pattern that enables tracking user progress, Figure 4 shows both the percentage of dialogues reaching each stage and the distribution of turns in each stage. As expected, the number of dialogue sessions reaching each stage de- creases from the initial to the final stages. This is particularly evident in the transition from Search/Task Overview to Task Steps. This likely indicates that users are primarily exploring the system or evaluating available tasks rather than actively pursuing task completion. A smaller proportion of users progress through all stages, suggesting opportunities to further support sustained engage- ment throughout the interaction. User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution7 0 00 0 0 92'*64+#9637 #49(-38*38 $4.(*38*38 #49(-&3)$4.(*38*387.786.'98.43 92'*64+#96374,"(&1* "*1*(8"9,,*78.43 *<8"8*5 &(084"*&6(- 46*58.437 "8&68#&70 6*:.497"8*5 "*1*(8#&70 "8&68:*6 &(084:*6:.*; "-4;3,6 9117(6**3$.)*4 %442"8*5 !*792*#&70 #49(-38*387.786.'98.43 92'*64+#96374,"(&1* *<8"8*5 "*&6(- 43+.62 "8&68:*6 "8&68#&70 !*/*(8 "*1*(8#&70 6*:.497"8*5 4251*8*#&70 46*58.437 484"8*5 9*78.43 "*8#.2*6 "-4;3,6 9117(6**3$.)*4 46**8&.17 7096.47.8.*7 !*51&(*3, &11'&(0 "*37.8.:* 984+"(45* -.8-&8 !*5*&8 8-*6 !*792*#&70 *15 3(4251*8*#*<8 $4.(*38*387.786.'98.43 38*38#=5* &.314; 4251*< 43445 77468*) 0 00 0 0 92'*64+#9637 #49(-38*38 $4.(*38*38 #49(-&3)$4.(*38*387.786.'98.43 92'*64+#96374,"(&1* "*1*(8"9,,*78.43 *<8"8*5 &(084"*&6(- 46*58.437 "8&68#&70 6*:.497"8*5 "*1*(8#&70 "8&68:*6 &(084:*6:.*; "-4;3,6 9117(6**3$.)*4 %442"8*5 !*792*#&70 #49(-38*387.786.'98.43 92'*64+#96374,"(&1* *<8"8*5 "*&6(- 43+.62 "8&68:*6 "8&68#&70 !*/*(8 "*1*(8#&70 6*:.497"8*5 4251*8*#&70 46*58.437 484"8*5 9*78.43 "*8#.2*6 "-4;3,6 9117(6**3$.)*4 46**8&.17 7096.47.8.*7 !*51&(*3, &11'&(0 "*37.8.:* 984+"(45* -.8-&8 !*5*&8 8-*6 !*792*#&70 *15 3(4251*8*#*<8 $4.(*38*387.786.'98.43 38*38#=5* &.314; 4251*< 43445 77468*) Fig. 5. Intent distribution across all dialogues. (top-left) Share of Touch vs. Voice intents by total turns. (bottom-left) Touch intents by type, (right) Voice intents by type. Percentages represent the proportions within each modality. Analyzing the turns in each stage, the Task Steps stage accounts for 29.7% of all turns, making it the most interaction-intensive stage. The Search stage follows as users browse various tasks. In contrast, both Task Overview and Task Completed are brief, consisting of a single acknowledgment turn. Overall, users spend 68.5% of their interaction in Task Grounding, highlighting the importance of effective task discovery and guidance before entering Task Execution. 4.2 Analyzing User Intents Figure 5 shows the distribution of intents, grouped into Main Flow (core navi- gation), Complex (in-depth interactions), Non-Cooperative (outside the system domain), and Assorted (auxiliary support). For overall user behavior, we see that 64.7% of turns use touch controls, likely due to its reliability compared to voice commands [21], with most belonging to the main interaction flow. For voice-intents, we also see that task navigation dominates, where the most frequent intent is Next Step, showing a focus on moving forward in tasks. Other common voice intents include Search, Confirmation, and Start Task. Com- plex intents like Questions and Replace Ingredients are less frequent, indicating linear task navigation. We also observe several deviations from the main flow. Fallback intent appears in 8.6% of turns, indicating unclear input and high- lighting opportunities to improve NLU. Out-of-Scope (e.g., smart home control or music playback) and Sensitive requests (e.g., sex, politics) are also common, reflecting a presence of non-cooperative [19] behaviors in CTA interactions. 8Ferreira, R. et al. Fig. 6. Dialogue-level (top row) and utterance-level (bottom row) trait statistics. The metrics follow [11] and are shown on the x-axis. Emotion level and Fluency level are model-based metrics, as in [11]. 4.3 Analysis of Conversational User Traits To better understand user–CTA dynamics, we analyze conversational behavior through the lens of user traits. Specifically, we adopt the taxonomy of eight user traits defined in [11], which captures differences in how users interact with conversational systems. The distribution for each trait is shown in Figure 6. Considering Dialogue-level traits which impact the progression of a conver- sation, we see that Engagement has an average of 6.8 turns per dialogue, with shorter interactions often indicating user exploration, irrelevant tasks, or early abandonment. Cooperativeness is generally high (μ = 80%), though a small peak near zero reflects fully uncooperative users. For Task Exploration, users who start a task complete on average half of the steps, with task-length also influencing its completion. Tolerance is limited, as users generally accept only one or two system fallbacks before ending the session. For Utterance-level traits, which influence the style of the user utterances, we see that Verbosity is low (μ = 2.8 words), reflecting concise task-focused com- mands [23]. Emotion is mostly neutral (μ = 0.54), with extremes corresponding to frustration or satisfaction. Fluency is generally high (μ = 0.82) but drops for longer utterances, often due to ASR errors or hesitation. Repetition is low on average, indicating diverse vocabulary usage across turns. Real vs. Crowdsourced Behavior. Prior work on CTAs relies on crowd- sourced datasets, however, such controlled settings may not reflect real-world usage [28]. To quantify this gap, we compare our real user data with the crowd- sourced Wizard of Tasks dataset [8] across user traits, revealing substantial dif- ferences. Complex actions occur in only 8.8% of real interactions versus 47.5% in crowdsourced data, reflecting limited exploration in natural usage. Real ut- terances are shorter (2.9 vs. 12.8 words), noisier, and more informal, with un- cooperative behavior (e.g., fallbacks, out-of-scope, sensitive requests) appearing User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution9 RatedW/O Screen W/ Screen Not Started Task Started Task Finished Task Recipe Tasks DIY Tasks 0 20 40 60 80 100 % of Dialogues 22% 17% 24% 31% 13% 8% 11% 14% 10% 14% 8% 11% 8% 5% 8% 9% 12% 13% 12% 12% 12% 10% 9% 14% 14% 16% 14% 13% 16% 17% 14% 18% 42% 40% 42% 33% 51% 60% 58% 45% T: 100% μ: 3.44 ̃ x: 4.0 σ: 1.62 T: 21% μ: 3.49 ̃ x: 4.0 σ: 1.54 T: 79% μ: 3.42 ̃ x: 4.0 σ: 1.64 T: 51% μ: 3.05 ̃ x: 3.0 σ: 1.68 T: 49% μ: 3.84 ̃ x: 5.0 σ: 1.44 T: 21% μ: 4.17 ̃ x: 5.0 σ: 1.25 T: 22% μ: 4.01 ̃ x: 5.0 σ: 1.4 T: 26% μ: 3.72 ̃ x: 4.0 σ: 1.46 Ratings:12345 Fig. 7. Ratings distribution (%), mean (μ), median ( ̃x), and std (σ), across various settings. in≈ 17% of interactions while being mostly absent from crowdsourced datasets. This gap highlights the importance of real-world data for building robust CTAs. 4.4 User Rating Analysis in CTAs In Figure 7, we analyze the 5k rated conversations, providing insights into system performance and user satisfaction. The average rating is 3.44, clustering around 1 and 5, indicating users predominantly provide feedback after strong positive or negative experiences, with most rated interactions being positive (≥ 3). Ratings do not differ substantially between device types (3.49 for non-screen devices vs. 3.42 for screen-based devices). In contrast, engaging with tasks is strongly associated with satisfaction: users who start a task rate the system higher than those who do not (3.84 vs. 3.05), with the highest ratings among users who complete tasks (4.17). Task domain is also associated with differences in ratings, with recipes scoring higher than DIY tasks (4.01 vs. 3.72), likely due to more curated content. Behavioral Signals Associated with User Satisfaction. We extend the general features of Choi et al. [7] with CTA-specific signals (intents, task stages, and user traits), resulting in a total of 39 features. The Spearman correlation analysis in Figure 8 shows that no single feature is strongly correlated with user ratings, suggesting that user satisfaction is related to a combination of behavioral and contextual factors [3,7]. The strongest positive correlations with rating are task-related: Task Progres- sion (0.26), Start and Completion (0.24), and reaching Task Execution (0.23). These associations indicate that conversations involving task progression tend 10Ferreira, R. et al. to receive higher user ratings. Negative correlations are observed for more turns in earlier stages such as Greeting (-0.16) and Search (-0.08), User Verbosity (- 0.14), and users taking more time to answer (-0.06), which may reflect greater early-stage friction or user frustration. Trait: Task Progression StartedTask FinishedTask Stage: Task Steps Task Stage: Task Completed Screens Trait: Engagement Turns AvgWordsSystem Trait: Repetition AvgUserWordOverlap AvgUserSystemWordOverlap Intent: Search AvgUserLatency Intent: Fallback AvgTurnDuration Stage: Search Trait: Verbosity Stage: Greeting Spearman Correlation 0.260.240.240.230.230.230.210.20.20.17-0.03-0.04-0.04-0.05-0.05-0.07-0.07-0.08-0.14-0.15 Fig. 8. Top-20 absolute Spearman correlation coefficients between each feature and user ratings. All results are statistically significant at p < 0.01. Subjectivity of Ratings. To evaluate rating subjectivity, we sampled 100 conversations (20 per rating level, < 20% touch input). Following Cervone et al. [3], we asked three annotators to independently rate each transcript on the same 1–5 scale from the user’s perspective, assessing overall satisfaction with the system’s performance. Ratings were binarized (1–2 negative, 3–5 positive) and evaluated using Spearman correlation (ρ), agreement, and Fleiss’ κ. Inter-annotator agreement was high (ρ = 0.70, 80%, κ = 0.59), while agree- ment with user ratings was substantially lower (ρ = 0.24, 61%, κ = 0.22). We attribute this to annotators applying stricter evaluation criteria than end users, leading to lower average scores (2.7 vs. 3.4). These results highlight the vari- ability of user ratings and the difference between experiencing a dialogue and evaluating it post hoc, as also shown by Choi et al. [7]. 5 Challenges, Opportunities and Design Implications After the careful analysis of user-CTA interaction data, we identified multiple opportunities and research challenges associated with lower performing conver- sations. In addition, we put-forward key design implications for CTA systems. Ambiguous ASR and Contexts. ASR errors frequently disrupted interac- tions. Users often attempted standard voice assistant features (e.g., music, smart home) unavailable during the CTA interaction, suggesting the need for multi- tasking between core assistant features and 3 rd -party applications. Agent Knowledge Boundaries. Task search was the most common problem when content relevant to the user query was not found. Queries requiring external knowledge (e.g., ingredient substitutions or Q&A) were often associated with incorrect responses, pointing to potential improvements via RAG methods [10]. User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution11 G1 G2 G3 G4 G5 G6 G7 G8 G9 G10 Simple actions per utterance. Limit instructions to simple executable step(s) to match the user’s working pace. Show suggestions and tips. Offer contextual hints to help users explore features and navigate the interaction. Flexible task progression. Structure interactions into clear phases while supporting users to skip, repeat, or jump to steps. Adaptive task chunking. Break complex, lengthy procedures into smaller steps based on real-time progress and device characteristics. Synchronized visuals. Combine voice guidance with images or videos to enhance task understanding. Multimodal interaction. Allow users to query and control task and visual content (e.g. VQA, video control). Task-anchored resume. Briefly address out-of-scope requests, then automatically pivot back to the active task. Procedural workarounds. Offer ingredient/tool substitutes or recovery steps when errors or constraints occur. Natural conversation progression. Use LLMs to convert template responses into fluid, and contextual spoken dialogue. Procedural grounding. Anchor LLM responses on the task’s content to provide context-aware responses and reduce hallucinations. IDGuideline and DescriptionCategory Response Design Task Navigation and UI Multimodal Task Support Generative Reasoning Conversation Recovery Fig. 9. Design Guidelines for Conversational Task Assistants. User Behavior. Most users followed a linear task flow with limited use of com- plex features such as Q&A or multimodal interaction. Moreover, the prevalence of uncooperative behavior illustrates challenges, including feature discoverabil- ity [18] and proactive support during task execution. CTA Design Guidelines. Drawing on insights from our analysis of real-world interactions, we synthesize our findings into a set of design guidelines for more behavior-driven and robust CTAs, summarized in Figure 9. 6 Conclusions We presented a large-scale study of a CTA, TWIZ-v2, analyzing real-world inter- actions to characterize user behavior, conversational traits, and user satisfaction. Our findings highlight the importance of real-world data, revealing various inter- action patterns, including preferences for touch-based interaction, predominantly linear task progression, and frequent out-of-domain behavior. We further show that user satisfaction is associated with engagement in task execution. In summary, these results highlight key opportunities for improving real- world CTAs, including enhancing robustness, task discovery, system knowledge, and error recovery, and provide empirically grounded and actionable guidelines for designing more adaptive and context-aware CTAs. 12Ferreira, R. et al. Limitations The analyzed conversations originate from U.S. Alexa users interacting in En- glish. Although the U.S. user base is large and diverse, the observed interaction patterns may differ across other regions, languages, cultural backgrounds, and platforms. Furthermore, while the system supports a wide range of cooking and DIY tasks, the findings may not directly generalize to all task domains. Addi- tionally, the reported associations with ratings are correlational, and may reflect unmeasured factors such as user motivation or task relevance. Investigating these settings are avenues for future work. Ethical Considerations All data used in this study were collected from users who voluntarily interacted with the system and consented to the use of their interactions for research pur- poses. All conversations are fully anonymized, with no personally identifiable information retained, and all results are reported only in aggregate form. Acknowledgements This work has been partially funded by the Amazon Science - TaskBot Prize Challenge 2021 and 2022, by the NOVA LINCS project Ref. UIDP/04516/2020 and by FCT Ref. UI/BD/151261/2021. References 1. Agichtein, E., Johnston, M., et al.: Advancing conversational task assistance: the second alexa prize taskbot challenge. In: Alexa Prize TaskBot Challenge 2 (2023) 2. Budzianowski, P., Wen, T., et al.: Multiwoz - A large-scale multi-domain wizard-of- oz dataset for task-oriented dialogue modelling. In: EMNLP. p. 5016–5026 (2018) 3. Cervone, A., Gambi, E., et al.: Automatically predicting user ratings for conversa- tional systems. In: CLiC-it. CEUR Workshop, vol. 2253 (2018) 4. Chan, S., Li, J., et al.: "mango mango, how to let the lettuce dry without A spin- ner?": Exploring user perceptions of using an llm-based conversational assistant to- ward cooking partner. CoRR (2023). https://doi.org/10.48550/ARXIV.2310.05853 5. Chiang, W.L., Li, Z., et al.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality (2023), https://lmsys.org/blog/2023-03-30-vicuna/ 6. Chirumalla, K., Eriksson, Y., et al.: The influence of different media instructions on solving a procedural task. In: ICED. p. 173–182 (2015) 7. Choi, J.I., Ahmadvand, A., et al.: Offline and online satisfaction prediction in open-domain conversational systems. In: CIKM. p. 1281–1290. ACM (2019) 8. Choi, J.I., Kuzi, S., et al.: Wizard of tasks: A novel conversational dataset for solving real-world tasks in conversational settings. In: COLING. p. 3514–3529 (2022) 9. Cowan, N.: The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and brain sciences 24(1), 87–114 (2001) User Behavior Analysis in Multimodal AI-Assisted Manual Task Execution13 10. Fan, W., Ding, Y., et al.: A survey on RAG meeting llms: Towards retrieval- augmented large language models. In: SIGKDD. p. 6491–6501 (2024) 11. Ferreira, R., Semedo, D., et al.: Multi-trait user simulation with adaptive decoding for conversational task assistants. In: EMNLP. p. 16105–16130 (2024) 12. Ferreira, R., Tavares, D., et al.: Twiz-v2: The wizard of multimodal conversational- stimulus. In: Alexa Prize TaskBot Challenge 2 (2023) 13. Fischer, S., Gemmell, C., et al.: Grillbot in practice: Lessons and tradeoffs deploying large language models for adaptable conversational task assistants. In: SIGKDD. p. 4951–4961 (2024) 14. Frummet, A., Speggiorin, A., et al.: Cooking with conversation: Enhancing user engagement and learning with a knowledge-enhancing assistant. ACM Trans. Inf. Syst. 42(5), 122:1–122:29 (2024) 15. Hwang, A., Oza, N., et al.: Rewriting the script: Adapting text instructions for voice interaction. In: ACM DIS. p. 2233–2248. ACM (2023) 16. Jaber, R., Zhong, S., et al.: Cooking with agents: Designing context-aware voice interaction. In: CHI. p. 551:1–551:13. ACM (2024) 17. Khatri, C., Venkatesh, A., et al.: Alexa prize — state of the art in conversational ai. AI Magazine 39(3), 40–55 (Sep 2018). https://doi.org/10.1609/aimag.v39i3.2810 18. Kirschthaler, P., Porcheron, M., et al.: What can I say?: Effects of discoverability in vuis on task performance and user experience. In: CUI. p. 9:1–9:9. ACM (2020) 19. Lei, W., Zhang, Y., et al.: Interacting with non-cooperative user: A new paradigm for proactive dialogue policy. In: SIGIR. p. 212–222. ACM (2022) 20. Liu, Y., Ott, M., et al.: Roberta: A robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019) 21. Myers, C., Furqan, A., et al.: Patterns for how users overcome obstacles in voice user interfaces. In: CHI. p. 6 (2018) 22. Pine, A., Littell, P., et al.: G i 2P i rule-based, index-preserving grapheme-to- phoneme transformations. In: 5th Workshop on the Use of Comput. Methods in the Study of Endangered Languages. p. 52–60. ACL (2022) 23. Porcheron, M., Fischer, J.E., et al.: Voice interfaces in everyday life. In: CHI. p. 640 (2018). https://doi.org/10.1145/3173574.3174214 24. Qin, L., Pan, W., et al.: End-to-end task-oriented dialogue: A survey of tasks, methods, and future directions. In: EMNLP. p. 5925–5941 (2023) 25. Radford, A., Kim, J.W., Hallacy, C., et al.: Learning transferable visual models from natural language supervision. In: ICML. vol. 139, p. 8748–8763 (2021) 26. Sicilia, A., Asano, Y., et al.: Isabel: An inclusive and collaborative task-oriented dialogue system. In: Alexa Prize TaskBot Challenge 2 (2023) 27. Strathearn, C., Gkatzia, D.: Task2dial: A novel task and dataset for commonsense- enhanced task-based dialogue grounded in documents. In: DialDoc@ACL. p. 187– 196. ACL (2022) 28. Tavares, D.: Zero-shot generalization of multimodal dialogue agents. In: M. p. 6935–6939. ACM (2022) 29. Vedula, N., Castellucci, G., et al.: Leveraging interesting facts to enhance user engagement with conversational interfaces. In: NACL. p. 437–446 (2024) 30. Wang, Z., Geng, N., et al.: Human vs. agent in task-oriented conversations. In: SIGIR Asia Pacific Region. p. 133–142 (2025) 31. Weber, J., Esau-Held, M., et al.: Designing an interaction concept for assisted cooking in smart kitchens: Focus on human agency, proactivity, and multimodality. In: ACM DIS. p. 1128–1144 (2023) 32. Yang, P.H., Huang, H., et al.: Mmwoz: Building multimodal agent for task-oriented dialogue (2025)