Paper deep dive
Impact of Multimodal and Conversational AI on Learning Outcomes and Experience
Karan Taneja, Anjali Singh, Ashok K. Goel
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 4/3/2026, 12:19:12 AM
Summary
This study evaluates the impact of multimodal and conversational AI on learning outcomes in STEM education. Comparing three systemsâMuDoC (multimodal conversational AI), TexDoC (text-only conversational AI), and DocSearch (semantic search)âthe researchers found that MuDoC significantly improved post-test scores compared to TexDoC. While TexDoC was perceived as more engaging, it resulted in lower learning outcomes, highlighting a disconnect between student perception and actual performance. The findings are interpreted through Cognitive Load Theory, suggesting that multimodality enhances germane load, while conversationality reduces extraneous load.
Entities (5)
Relation Signals (4)
MuDoC â improves â Learning Outcomes
confidence 95% · Learners using MuDoC achieved the highest post-test scores
TexDoC â increases â Perceived Engagement
confidence 90% · TexDoC was rated as significantly more engaging and easier to use than DocSearch
Multimodality â increases â Germane Load
confidence 85% · visual-verbal integration induced by multimodality increases germane load
Conversationality â reduces â Extraneous Load
confidence 85% · conversationality reduces extraneous load
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal Large Language Models (MLLMs) offer an opportunity to support multimedia learning through conversational systems grounded in educational content. However, while conversational AI is known to boost engagement, its impact on learning in visually-rich STEM domains remains under-explored. Moreover, there is limited understanding of how multimodality and conversationality jointly influence learning in generative AI systems. This work reports findings from a randomized controlled online study (N = 124) comparing three approaches to learning biology from textbook content: (1) a document-grounded conversational AI with interleaved text-and-image responses (MuDoC), (2) a document-grounded conversational AI with text-only responses (TexDoC), and (3) a textbook interface with semantic search and highlighting (DocSearch). Learners using MuDoC achieved the highest post-test scores and reported the most positive learning experience. Notably, while TexDoC was rated as significantly more engaging and easier to use than DocSearch, it led to the lowest post-test scores, revealing a disconnect between student perceptions and learning outcomes. Interpreted through the lens of the Cognitive Load Theory, these findings suggest that conversationality reduces extraneous load, while visual-verbal integration induced by multimodality increases germane load, leading to better learning outcomes. When conversationality is not complemented by multimodality, reduced cognitive effort may instead inflate perceived understanding without improving learning outcomes.
Tags
Links
- Source: https://arxiv.org/abs/2604.02221v1
- Canonical: https://arxiv.org/abs/2604.02221v1
Trouble viewing inline? Open PDF directly â
Full Text
45,867 characters extracted from source content.
Expand or collapse full text
Impact of Multimodal and Conversational AI on Learning Outcomes and Experience Karan Taneja 1 , Anjali Singh 2 , and Ashok K. Goel 1 1 Georgia Institute of Technology, Atlanta GA 30332, USA 2 University of Texas at Austin, Austin TX 78712, USA karan.taneja,ashok.goel@c.gatech.edu, anjali.singh@ischool.utexas.edu Abstract. Multimodal Large Language Models (MLLMs) offer an op- portunity to support multimedia learning through conversational sys- tems grounded in educational content. However, while conversational AI is known to boost engagement, its impact on learning in visually- rich STEM domains remains under-explored. Moreover, there is limited understanding of how multimodality and conversationality jointly in- fluence learning in generative AI systems. This work reports findings from a randomized controlled online study (N = 124) comparing three approaches to learning biology from textbook content: (1) a document- grounded conversational AI with interleaved text-and-image responses (MuDoC), (2) a document-grounded conversational AI with text-only responses (TexDoC), and (3) a textbook interface with semantic search and highlighting (DocSearch). Learners using MuDoC achieved the highest post-test scores and reported the most positive learning experi- ence. Notably, while TexDoC was rated as significantly more engaging and easier to use than DocSearch, it led to the lowest post-test scores, revealing a disconnect between student perceptions and learning out- comes. Interpreted through the lens of the Cognitive Load Theory, these findings suggest that conversationality reduces extraneous load, while visual-verbal integration induced by multimodality increases germane load, leading to better learning outcomes. When conversationality is not complemented by multimodality, reduced cognitive effort may instead inflate perceived understanding without improving learning outcomes. Keywords: Multimodal AI· Conversational Multimedia Learning· RCT 1 Introduction Multimedia learning integrates verbal and visual representations and has been shown to improve knowledge retention and transfer compared to learning from text alone [15]. This is particularly important in visually rich STEM disciplines, where visual representations such as diagrams and graphs are central to learning. Recent advances in Multimodal Large Language Models (MLLMs) create new opportunities to support learning by enabling conversational systems grounded in educational content that can generate personalized multimedia explanations in response to learner inquiries [23]. This potential is especially salient given Accepted to International Conference on Artificial Intelligence in Education (AIED) 2026, Seoul, Korea (June 29 - July 3, 2026) arXiv:2604.02221v1 [cs.HC] 2 Apr 2026 2K. Taneja et al. the rapid adoption of AI tools for learning and tutoring, including large-scale deployments in higher education [10]. However, empirical evidence regarding the educational impact of AI chatbots remains mixed. While some studies report improvements in learning outcomes and student engagement [24], others raise concerns such as overreliance and reduced metacognitive effort [27]. Additionally, the impact of AI-generated multimedia content on learning outcomes is still not well understood. Finally, there is limited work examining how conversational interaction and multimodality jointly influence the learning process. To investigate how multimodality and conversational AIâboth independently and jointlyâinfluence studentsâ learning outcomes and experience, we conducted a randomized controlled trial (RCT) on Prolific with N = 124 participants learn- ing meiosis cell division, a fundamental topic in cell biology. We compared three systems: (i) MuDoC 2.0 (henceforth referred to as MuDoC), an AI system for conversational multimedia learning that uses text and visuals from learning material to construct interleaved text-and-image responses, extending our pre- vious system MuDoC 1.0 [21], (i) TexDoC, a system that is identical to MuDoC, except that it generates text-only responses without images, and (i) DocSearch (Document Search), a semantic search tool that supports mul- timedia learning by highlighting relevant content in the textbook in response to studentsâ search queries. Participants first completed a pre-test, then learned the assigned topic using one of the three systems, and finally completed a post-test and a survey probing their learning experience. Results show that MuDoC led to significantly higher post-test scores com- pared to TexDoC, whereas no significant differences were observed between the remaining pairs of systems. For learning experience, MuDoC and Tex- DoC were rated similarly, while DocSearch received significantly lower rat- ings. Interpreting these findings from the lens of the Cognitive Load Theory (CLT) [20] suggests that conversational multimedia learning supports learning by reducing extraneous load through conversationality while increasing peda- gogically-effective germane load through visual-verbal integration. On the other hand, conversational interaction without accompanying visuals enhances per- ceived learning experience despite resulting in poorer learning outcomes. These findings provide evidence in support of using MLLMs for conversational mul- timedia learning to enhance both studentsâ learning outcomes and experience. Further, by examining learner behaviors and the disconnect between subjective satisfaction and objective performance of the TexDoC group, this work offers a new perspective on the effects of conversational AI on learning and over-reliance. 2 Related Work Cognitive Theory of Multimedia Learning (CTML) [15] posits that learners pro- cess verbal and visual information through separate, capacity-limited channels. Learning is enhanced when these representations are meaningfully integrated with one another and with prior knowledge in long-term memory [15]. This in- 3. MUDOC, TEXDOC, AND DOCSEARCH3 tegration supports deeper cognitive processing and leads to improved learning outcomes compared to verbal information alone. Several text-based tutors have been used for learning and management in classrooms. Jill Watson [8] was an early question-answering system for answer- ing student queries on classroom forums based on structured information. More recently, it utilizes LLM-based methods with retrieval-augmented generation [22] and has been deployed at large scale at multiple institutions [10]. Since the pub- lication of this paper, MuDoC has been integrated into Jill Watson and has been deployed in classrooms at Georgia Institute of Technology. JeepyTA [2] is an- other such virtual teaching assistant that uses LLM-generated responses and a list of question-answer pairs which are ranked based on similarity matching. Ped- agogical Tutor [25] similarly relies on template-based slides to answer questions with LLMs, and also allows users to search relevant slides. Multimodal AI systems have also been explored for multimedia learning in several domains [3,4,5,23], but the impact of AI-generated multimedia content on learning outcomes remains unclear. Chen et al. [5] generated visualizations as learning aids for poetry but did not compare against a non-visual baseline. Bland et al. [3] used AI to generate cinematic clinical narratives for pharma- cology but focused only on student experience. In mathematics, the Interactive Sketchpad [4] tool uses an MLLM to generate and manipulate diagrams within a conversational tutoring setting. While learners reported a better learning expe- rience and problem-solving efficacy using this tool compared to using ChatGPT, this study did not examine the impacts on their learning outcomes. MuDoC 1.0 [23] enhanced learner engagement and trust but did not significantly improve learning outcomes compared to a baseline system with text-only responses. To the best of our knowledge, prior work does provide evidence for the effectiveness of MLLMs in generating multimedia content for improving learning outcomes. Regarding the effects of conversational AI, recent studies with educational chatbots, such as ChatGPT-style systems, report gains in academic performance, engagement, and perceived personalization, though effects vary across domains and contexts [6,24]. At the same time, researchers have also raised concerns about overreliance and shallow processing due to undermined cognitive and metacog- nitive processes that are essential for durable learning [7]. Further, it is unclear how conversational interfaces for interacting with textbooks compare to more classical document or semantic search baselines, which we address in this work. 3 MuDoC, TexDoC, and DocSearch In this section, we first describe MuDoC, an AI system that answers queries with interleaved text-and-image responses, extending MuDoC 1.0 [21], using the âReason-and-Actâ (ReAct) framework [26] (see Figure 1). We then discuss Tex- DoC and DocSearch made with minimal modifications to the MuDoC pipe- line to isolate the effects of visuals and conversational AI on learning. Additional details about these AI systems are provided in the supplementary material 3 . 3 Supplementary material (including demo videos): https://tinyurl.com/IMCAILOE 4K. Taneja et al. Retrieval Query GPT-4.1 Final Response Initial Search Multimodal Response Reason-and-Act Action Content Search Confirm Intent User Clarification 10 Texts and 5 Images 3 Text Search Results Retrieval + System Prompt + Context Fig. 1. MuDoC uses GPT-4.1 for Reason-and-Act (ReAct) loop with 4 possible actions viz. Initial Search, Content Search, Confirm Intent, and Final Response. 3.1 Preprocessing and Response Generation in MuDoC Document Preprocessing: The document processing pipeline involves parsing the document layout to extract content from each page, which is subsequently pro- cessed and stored in a vector database to facilitate the retrieval process. We uti- lize the HURIDOCS document layout analysis parser [1] to extract text, figures, and other structural elements from the document. These blocks extracted during the layout analysis are combined into partly overlapping chunks of at least 8,000 characters. For each text chunk, we generate and store a succinct summary us- ing the OpenAI gpt-4o-mini-2024-07-18 model 3 . Similarly, for every extracted image, we store a caption and a detailed image description generated using the same model. We employ the OpenAI text-embedding-3-large model for text embeddings and the google/siglip-so400m-patch14-384 model for image em- beddings [28]. The resulting chunks and their embeddings are stored in a vector database for retrieval (described later). Compared to MuDoC 1.0, among other differences, we now use a more recent and performant layout parser and text and image embedding models along with larger context windows. Response Generation: MuDoC uses OpenAI gpt-4.1-2025-04-14, which sup- ports multimodal inputs and function calling capabilities, for response gener- ation. It now uses reasoning, unlike MuDoC 1.0, by structuring response gen- eration as a ReAct [26] loop to improve search functions usage and generation quality. In each iteration of the ReAct loop, the model first Re asons its current state within the conversation, and then decides to take an Action to search for additional context or to generate a response (see Figure 1). Reasoning: Prior to executing an action, the LLM is instructed to perform the following three reflection steps 3 . (i) Query Reflection: The model evaluates the userâs intent, identifying any unknown terminology, acronyms, typos, or miscon- ceptions within the query, and determines whether a search is necessitated by the request. (i) Search Content Reflection: If content has been retrieved during the current or past conversation turns, the model assesses content relevance and identifies any missing information that requires further retrieval. The model also reasons about the most pertinent textual and visual content and determines how to structure a self-contained response, including definitions for terms that may 3. MUDOC, TEXDOC, AND DOCSEARCH5 be unfamiliar to the student. (i) Action Reasoning: In this final step, the model synthesizes the previous reflections to select the most appropriate course of action. The possible actions are detailed next. Actions: The model can execute four distinct actions, categorized into search and generation functions. If the model requires foundational information to interpret the query, such as definitions for unfamiliar terms or acronyms, it utilizes the Initial Search action. This retrieves the top three text results, ensuring a low- overhead clarification of the problem space. Conversely, if the model understands the query sufficiently but requires a search, it can trigger the Content Search action and subsequently generate specific search queries for text and image re- trieval function calls, which can return up to ten text chunks and five images respectively. The remaining two actions are generative. If the model determines that the userâs intent is ambiguous, it utilizes the Confirm Intent action to request a clarification, ensuring a precise final output. Finally, once the model possesses a clear understanding of the query and the necessary retrieved context, it executes the Final Response action to synthesize a comprehensive response for the user. In the event of a search action, the retrieved results are appended to the conversation history, serving as updated context for the subsequent ReAct loop. Conversely, when a generation action is performed, the model delivers the response and awaits the userâs input to proceed. Text and Image Retrieval: As opposed to purely embedding-based retrieval in MuDoC 1.0, MuDoC now employs a hybrid search strategy [11] combining dense vector embeddings with sparse keyword-based retrieval for both texts and images. For text retrieval, we utilize both the original content and the generated summaries (from preprocessing) for embeddings and keyword-based scoring. We found that prioritizing vector embedding similarity (75% weight) over keyword- based BM25 [18] score (25% weight) led to the highest retrieval accuracy in hybrid ranking. After text retrieval, we sort retrieved results in the natural order of the textbook and remove redundancy caused by overlapping chunks, leading to a leaner and organized context for the LLM. In image retrieval, the raw text index includes the image captions and descriptions while the image vector is the mean of the image and caption embedding. For ranking images, we found that prioritizing BM25 score (75% weight) led to better retrieval, potentially because image descriptions focus on key terminology, where keyword matching works well, rather than semantically-rich explanations. We plan to release a thorough evaluation of text and image retrieval methods used in MuDoC in future work. Interleaved Grounded Generation: MuDoC synthesizes an interleaved, grounded response based on retrieved texts and images while providing explicit references to the source material. To incorporate visuals, the model is instructed to gener- ate HTML figure, figcaption and img tags, based on source links and corre- sponding raw retrieved images. Textual claims are supported by in-text citations, similar to WebGPT [16], referring to the original document name and block IDs to ensure that the generated response is grounded and verifiable. We found that 6K. Taneja et al. block-based in-text citations (using LLMs directly) had higher reliability than embedding-based source attribution used in MuDoC 1.0. Pedagogically-aligned Prompt: The modelâs generation is guided by the principles of CTML [15]. Specifically, the model is prompted to prioritize functional visuals over decorative ones to minimize cognitive load. Information provided via images should be complementary rather than redundant to the text, and visual elements should be spatially integrated near the relevant text. The model is instructed to utilize concrete examples and analogies rather than abstract definitions in accor- dance with CTMLâs worked example effect. Further, explanations should follow a âsimple-to-formalâ progression, grounding new concepts in the context of prior conversation turns. At the conclusion of a response, MuDoC can pose reflective questions to encourage active learning, invite the user to explain concepts in their own words or make follow-up inquiries. The system prompt 3 also includes instructions to ensure that all responses remain safe, polite, and constructive. Response Post-processing and Rendering: Following response generation, we em- ploy regular expressions (regex) to identify in-text citations and image placehold- ers within the modelâs output. These identifiers are dynamically replaced during token streaming with interactive link icons that connect directly to the source document. When a user clicks a citation link, the system retrieves the precise coordinates of the target block from the backend, triggering the frontend to switch to the âDocumentâ tab, navigate to the corresponding page, and highlight the relevant passage for several seconds to facilitate verification. For HTML fig- ure references, the specific imageâextracted from the source document during document preprocessingâis embedded directly into the chat message. Similar to text citations, each image is also accompanied by a link icon that allows the user to locate the figure within the original document. To ensure transparency, the modelâs reasoning process is accessible via a dedicated icon adjacent to the agentâs avatar. Furthermore, the token stream is monitored in real-time to detect ReAct loops, allowing the interface to display agent status indicators when the system is performing reasoning and retrieval. 3.2 TexDoC and DocSearch TexDoC, a Text-only version of MuDoC, lacks the image retrieval pipeline of MuDoC, image-related prompt instructions, the image search function call, and raw input images. As a result, it provides text-only responses similar to Jill Wat- son [22], but with interactive citations, indirectly allowing users to find visuals in the textbook. Aside from the omissions, TexDoC is identical to MuDoC and helps us isolate the impact of multimodality in our experiments. DocSearch is a stateless semantic search tool integrated within the docu- ment viewer. It employs the same retrieval pipeline as TexDoC and uses the same LLM to select the most relevant text and image blocks from the retrieved content to provide up to ten search results. Users can navigate these results us- ing arrow keys or by clicking corresponding buttons. As a user navigates to a 4. EXPERIMENTAL METHODS7 Objectives Textbook Chat Fig. 2. Snapshots from self-learning session: âChatâ tab shows the conversational AI interface (MuDoC and TexDoC). âObjectivesâ tab contains the learning objectives with checkboxes to track progress. âDocumentâ tab shows the source document for all AI systems. A âNotepadâ box is present on the right for note-taking. search result, a block of text or image is highlighted for a few seconds, similar to MuDoC and TexDoC citations. Overall, DocSearch is a convenient way to navigate a document with very specific search queries, but it does not synthe- size responses or use past context for follow-up questions, helping us study the impact of conversational AI compared to non-conversational multimedia. 4 Experimental Methods Participant Recruitment: We conducted the study using the online platform Prolific, recruiting participants aged 18+, residing in the US, who had a major in biology and had completed at least a high school diploma. The compensation was US$20 for completing the study with an estimated duration of 75 minutes. A total of 162 participants completed the study, of which 17 were excluded for not passing attention checks, 18 were excluded for insufficient activity during the learning session, and 3 were excluded due to technical issues, resulting in a 8K. Taneja et al. final sample size of N = 124. Out of 124, 117 (94%) of participants were 18-35 years old, and 112 (90%) had basic familiarity with AI as end users. Study Protocol: The participants first signed the consent form for our study, approved by the Institutional Review Board (IRB) at Georgia Tech. Fol- lowing consent, they were randomly assigned to one of the three experimental conditions, viz. MuDoC, TexDoC and DocSearch. In each condition, par- ticipants first completed a pre-test with 10 multiple-choice questions (MCQs) covering basics of cell division. Participants who scored fewer than 4 correct answers were screened out. Those who scored 4 or higher proceeded to watch a 15-minute instructional video on the meiosis cell division process and a two- minute video on the AI system they would subsequently utilize for learning. In the next step, called âself-learning with AIâ, participants were instructed to learn about meiosis cell division using only the assigned system, guided by three predefined learning objectives, and informed that they could use the pro- vided notepad (described below) for note-taking to support their learning. The system interface displayed two tabs for the DocSearch condition and three tabs for TexDoC and MuDoC. The first âObjectivesâ tab, in all conditions, presented the learning objectives. For the TexDoC and MuDoC conditions, the middle âChatâ tab contained the chat interface. The final âDocumentâ tab, in all conditions, displayed the source document, the OpenStax Biology textbook [9] (Chapter 4-14, 447 pages 3 ), and a search bar in the case of DocSearch. In all conditions, the notepad was displayed on the right side, occupying one-fourth of the interface width. The interface required participants to spend at least 15 minutes (by disabling the âNextâ button) and at most 25 minutes (by automat- ically advancing the session) on self-learning. Snapshots from the self-learning session are shown in Figure 2. Next, participants completed a post-test with 10 MCQs related to the three learning objectives. Unlike the pre-test which focused on broader knowledge of cell division with 3 visual-based questions, the post-test focused on three specified learning objectives related to meiosis cell division with 7 visual-based questions. Finally, they answered survey questions with Likert-scale responses (see Figure 3), provided subjective feedback and answered demographic ques- tions. The survey, learning objectives, pre- and post-tests, and other materials are provided in the supplementary material 3 . Data Collection: In addition to test scores and surveys, we measured the time spent using the assigned system, the number of queries, and the frequency of adding or editing notes in the provided notepad across all three conditions (Table 1). For MuDoC and TexDoC, we also measured the average response length from participantsâ conversations, usage of interactive citations, and the proportion of time spent on the textbook tab (Section 5.2). 5. RESULTS9 Table 1. M, SD values for Pre-, Post-test scores (out of 10), Time (mins) spent interacting with AI, number of Queries and edits to Notes (highest in bold). System N Pre-test Post-test TimeQueriesNotes MuDoC 42 6.92 (1.25) 7.24 (1.15) 17.6 (2.9) 7.26 (3.32) 7.1 (7.3) TexDoC 41 6.91 (1.32) 6.55 (1.11) 18.0 (3.7) 8.29 (5.02) 7.5 (8.2) DocSearch 41 6.98 (1.29) 6.77 (0.93) 17.6 (3.5) 5.27 (2.74) 11.4 (8.4) 5 Results 5.1 Pre-test and Post-test Scores Descriptive statistics for performance across the three experimental conditions are presented in Table 1. To test analysis of variance (ANOVA) assumptions, normality was assessed using Shapiro-Wilk tests for each group (all p > 0.05) and homogeneity of variance was confirmed using Leveneâs test, F(2, 121) = 0.117,p = .89 > .05 for pre-test scores and F(2, 121) = 1.711,p = .18 > .05 for post-test scores. A one-way ANOVA on pre-test scores indicated no difference across the three groups, F(2, 121) = 0.27,p = .97,η 2 < .001, confirming that participants had comparable prior knowledge across the three groups. A one-way ANOVA on post-test scores revealed a statistically significant ef- fect of the AI system type with a moderate effect size, F(2, 121) = 4.60,p = .012,η 2 = 0.071. Post-hoc comparisons using Tukeyâs honestly significant differ- ence (HSD) test showed that MuDoC participants achieved significantly higher scores than TexDoC (p = .010, diff = 0.69). While MuDoC also outper- formed DocSearch (diff = 0.47), this difference did not reach the threshold for adjusted significance in the Tukey HSD test (p = .11). We also examined the influence of prior knowledge (pre-test scores) on learn- ing outcomes (post-test scores) using an Ordinary Least Squares (OLS) regres- sion model while controlling for the experimental condition. As expected, pre- test scores were significantly positively associated with post-test scores (ÎČ = 0.21,p = .005), providing evidence that the tests are sensitive to differences in prior knowledge and learning gains. 5.2 Why MuDoC is better than TexDoC? To confirm that MuDoC leads to better performance than TexDoC due to its high-quality multimedia content, we performed in-depth analysis of user inter- actions and system behavior. While none of the behaviors we captured showed a statistically significant difference, MuDoC group consistently had lower mean values. They jumped to the textbook using interactive citations less frequently (p = .08,d =â0.38 using t-test), asked fewer questions (p = .27,d =â0.36), spent less time with the AI system (p = .31,d =â0.11), spent a lower propor- tion of time on the textbook tab (p = .60,d =â0.12), and made fewer updates to their notes (p = .80,d =â0.06). In terms of system behavior, the average 10K. Taneja et al. response length of MuDoC (excluding figures) was lower than that of Tex- DoC , but not significantly (p = 0.10,d =â0.36). Besides text, an average conversation with MuDoC had M = 3.1,SD = 1.2 unique images and a total of M = 6.2,SD = 2.3 images. These results show that there was no significant differences in learning behav- iors and system behavior, except for the visuals included in MuDoC responses, which likely drove the improvements in learning outcomes. Further, if we examine the weak trends discussed above, their consistency suggests that MuDoC group may have allocated more time to examining the visuals rather than relying on the textbook or asking follow-up questions. TexDoC users likely had to read complex text, ask follow-up questions, and seek visuals in the source textbook, all of which are time-consuming and cognitively demanding. How does DocSearch compare? Compared to the systems with conversa- tional AI, the DocSearch group made significantly fewer queries (p < .001,d = â0.63), but made significantly more updates to their notes (p = .008,d = 0.50). This group made M = 5.27, SD = 2.74 queries and examined M = 18.8, SD = 10.1 search results. In summary, DocSearch interactions show a more traditional approach to reading with fewer queries and more note-taking. 5.3 Quantitative Data on Learning Experience The results from seven Likert-scale survey questions are summarized in Figure 3. Since normality and homogeneity of variance are not satisfied by Likert-scale data, we used the non-parametric Kruskal-Wallis test. The results revealed sig- nificant between-group differences across all seven questions (p < .05), with effect sizes (η 2 ) ranging from moderate (.051) to large (.123). Post-hoc analyses were performed using Dunnâs test with Holm correction for multiple comparisons. Usability (1, 2 in Figure 3): The analysis indicates that conversational inter- faces (MuDoC and TexDoC) provided a superior user experience compared to DocSearch. Participants found the two systems significantly easier to navigate (p = .002,η 2 = .09) and more engaging to use (p < .001,η 2 = .12). No significant differences between MuDoC and TexDoC suggest that conversationality was the primary driver of better usability ratings. Response Quality (3, 4, 5 in Figure 3): In terms of perceived quality, both conversational AI systems outperformed DocSearch in providing detailed (p = .001,η 2 = .10), helpful (p = .017,η 2 = .05), and clear responses (p = .001,η 2 = 0.10). Again, we did not see a difference between MuDoC and TexDoC, sug- gesting that conversationality in AI systems leads to better perceived quality for information needs as compared to stateless semantic search in DocSearch. Cognitive Engagement (6, 7 in Figure 3): Both conversational systems signif- icantly outperformed the baseline in helping users formulate follow-up questions (p = .001,η 2 = .10), indicating a shift from passive reading to active cognitive engagement. However, for the question âmakes me thinkâ (p = .012,η 2 = .06), MuDoC was the only system to maintain a statistically significant advantage over the DocSearch baseline (p = .010 in Dunnâs test). TexDoC showed a positive trend but did not reach statistical significance (p = .163 in Dunnâs test). 5. RESULTS11 MuDoC MuDoC MuDoC MuDoC MuDoC MuDoC MuDoC TexDoC TexDoC TexDoC TexDoC TexDoC TexDoC TexDoC DocSearch DocSearch DocSearch DocSearch DocSearch DocSearch DocSearch 2 1 1 1 1 2 2 3 8 5 5 9 14 9 19 23 20 22 20 15 17 20 8 16 15 12 10 14 1 1 3 2 2 10 6 5 6 10 13 16 15 22 20 22 18 17 22 16 12 16 13 10 9 3 2 4 1 8 5 5 6 11 3 8 12 12 6 14 15 13 24 14 20 23 16 9 17 8 4 4 7 5 4 4 (1) Is easy to navigate (2) Is engaging to use (3) Gives results/responses that have adequate level of detail (4) Gives results/responses that help me learn (5) Gives easy to understand results/responses (6) Helps me think of new questions (7) Makes me think Completely DisagreeDisagreeNeutralAgreeCompletely Agree Fig. 3. Perceived Learning Experience with MuDoC, TexDoC and DocSearch. This suggests that the integration of visuals, unique to MuDoC, helps learners integrate information from two modalities, therefore making them think more. 5.4 Qualitative Data on Learning Experience Participants answered two survey questions about the strengths and limitations of the AI systems they were assigned. We split their responses into distinct asser- tions and thematically organized similar assertions, of which the most common ones are elaborated below. What was the best part of using âSystem Xâ during the self-learning session? DocSearch was primarily lauded by 18 participants for its efficiency, specif- ically its ability to act as a âsmartâ search tool; one participant noted that it âlikely saved 20-30 minutes of readingâ by streamlining the identification of rele- vant material in the textbook. In contrast, TexDoCâs main strength was in its transparency (14 participants), with users appreciating the âlinks back to where in the textbook [it] was referencing,â which served as a vital tool for information verification, corroborating findings from MuDoC 1.0 [23]. While MuDoC had identical source attribution, these strengths were overshadowed by participantsâ appreciation for its multimodal grounding and simplified responses (9+9 partici- pants). Users highlighted that the system âintegrated visual aids to help explain the conceptsâ while âsummariz[ing] text that is typically long and tiring.â 12K. Taneja et al. What are some frustrations you experienced when using âSystem Xâ? Doc- Search users reported significant frustration with its perceived lack of âintel- ligenceâ, with 10 participants noting it felt like a âglorified search toolâ rather than a helpful AI. In the TexDoC condition, the most prominent frustration was the lack of visual aids (9 participants), which forced users to âswitch back and forth between tabsâ to view necessary diagrams. While MuDoC success- fully addressed the visual needs, a few participants mentioned limitations such as cognitive overload (7 participants) and system latency (6 participants) that were also brought up by 5 and 3 TexDoC participants respectively, while only 4 DocSearch participants indicated the former as their biggest frustration. Very few participants in each group (1-2) reported difficulties with the document or chat interface, which alleviates concern about the interface design negatively impacting the learning experience. Summary: For DocSearch, across the two responses, 13 participants expressed a desire for content summarization, reflecting the prevalent expectation of syn- thesized summaries from AI systems. For TexDoC, the feedback centered on the limitations of a text-only conversational AI, particularly for the complex STEM topic they were learning. MuDoC received the most enthusiastic feed- back overall, particularly for learning âwithout needing to look at paragraphs of textâ and âthe fact [that] it pulled [visuals] from the textbookâ. 6 Discussion and Future Work The results of our study suggest a complex interplay between multimodality, conversationality, learning experience, and learning outcomes. MuDoC enabled conversational multimedia learning that led to significantly better learning outcomes with a moderate effect size compared to TexDoC. It also led to better learning outcomes compared to DocSearch , although this result was not statistically significant. In terms of learning experience, including ease of use, engagement, and perceived helpfulness, MuDoC is at par with TexDoC and outperforms DocSearch. Loosely speaking, trends show that MuDoC > DocSearchâ„ TexDoC for learning outcomes (Table 1), while MuDoC = TexDoC > DocSearch for learning experience (Figure 3). We interpret these results through the lens of Cognitive Load Theory (CLT) [20]. According to CLT, effective learning occurs when instructional design re- duces extraneous cognitive load and promotes germane load within the limits of working memory [20]. Extraneous load is the mental effort caused by poorly designed learning materials or distracting environmental factors, while germane load is the beneficial mental effort used to process new information, build mental models, and integrate knowledge into long-term memory. Multimedia learning leads to higher germane load through careful content de- sign [15]. In the case of DocSearch, while the presence of multimedia content would have led to high germane load, lack of conversationality likely increased extraneous loadâby requiring learners to repeatedly search and integrate frag- mented contentâpotentially exceeding cognitive capacity. This imbalance may 6. DISCUSSION AND FUTURE WORK13 explain the poorer learning experience observed with DocSearch. In the case of TexDoC, low extraneous load due to conversationality likely led to better learn- ing experience, but lower germane load due to lack of multimodality could have resulted in poorer learning outcomes. Compared to TexDoC, in DocSearch, in addition to the multimodal content, the more frequent note-taking (Table 1) would have also contributed to higher germane load. This may explain why, de- spite higher extraneous load and a poorer learning experience, the DocSearch groupâs learning outcomes were comparable to TexDoC. Finally, in the case of MuDoC, conversationality reduced extraneous load, thereby enhancing the learning experience. This preserved cognitive capacity could have enabled learn- ers to capitalize on the heightened germane load due to multimodality, leading to better learning outcomes. In summary, these findings suggest that conversa- tionality reduces extraneous load by maintaining contextual continuity, eliminat- ing the need for repeated search and simplifying the integration of information. Further, conversational multimedia learning exhibits an improved cognitive load balance by lowering extraneous load and increasing germane load, leading to both improved learning outcomes and experience. It is also worth noting that there is a striking disconnect between partici- pantsâ perceptions of TexDoC and its actual impact on learning outcomesâ participants rated TexDoC as highly engaging and easy to use, yet it yielded the lowest post-test scores. While many in the TexDoC group realized that visuals would support learning, they did not spend significantly more time exploring visuals in the textbook compared to the MuDoC group, likely as that would have led to increased extraneous load [13]. In contrast, by reducing the effort required to access and interpret visuals, MuDoC facilitated integration between text and graphical visuals, therefore replacing extraneous load with germane load. This suggests that when reduced extraneous load from conversationality is not complemented by higher germane load, it can lead to a fluency effect where ease of digesting information is mistaken as a sign of learning [17]. Finally, MuDoC establishes the efficacy of recent MLLMs to adaptively gen- erate high-quality multimedia content for educational contexts. Further improve- ments in image retrieval, image understanding in MLLMs, and generative AI for diagram creation and editing will also enhance context-specificity of visuals. Our results also highlight opportunities for future research on how conversational AI can be effectively designed to enhance learning. While conversational interaction appears beneficial for improving subjective learning experience, text-only conver- sational systems may be insufficient for improving learning outcomes. In visually rich STEM domains, the inclusion of relevant textbook-based visuals alongside textual explanations is both necessary and beneficial for learning, as evidenced by our findings. In domains where visuals are less critical, future work should ex- plore alternative design strategies for introducing desirable germane loadâsuch as metacognitive prompting [19] or introducing âfrictionâ in human-AI interac- tions [12]âwithin conversational AI systems to support learning outcomes. Limitations: First, the study was conducted within a single domain and for a single biology topic. Second, the time constraints (15-25 minutes) may have 14K. Taneja et al. affected learning outcomes, but this effect should be minimal as 88% of partic- ipants completed the learning session before the time limit, suggesting that the time allotted for the task was sufficient. Third, we assessed learning immedi- ately after the intervention; future work should also consider measuring impact on long-term learning outcomes such as retention. Additionally, we grounded our interpretation in CLT, but did not directly measure cognitive load during learning. Future work should include explicit cognitive load measures [14] to bet- ter understand the influence of conversational multimedia learning. Fourth, the study recruited participants via Prolific. Future research should examine these findings in more authentic educational settings. Finally, MuDoCâs performance may be dependent on the quality of the source textbook and visuals which im- pacts the accuracy of the document preprocessing pipeline, a subject of system evaluation that is outside the scope of this work. 7 Conclusion In this paper, we described MuDoC, a conversational multimedia learning sys- tem that generates grounded, interleaved text-and-image responses based on textbook content, and compared it to two baselinesâTexDoC (Text-only Mu- DoC) and DocSearch (Document with semantic Search)âthrough an RCT with N = 124 participants. MuDoC led to the highest post-test scores and a bet- ter learning experience based on qualitative and quantitative feedback, showing that multimodal LLMs can be effectively leveraged for learning by introducing in- terleaved responses containing grounded visuals. Examining the results through the lens of Cognitive Load Theory (CLT), we argued that conversational multi- media learning improves cognitive load balance by reducing extraneous load as a result of conversationality while increasing germane load as a result of multi- modality. Our findings also suggest that conversational AI systems can lead to an illusion of learning, unless complemented by strategies to cultivate higher ger- mane load, such as visually-enriched explanations. Future work should explore the application of conversational multimedia learning across diverse domains and develop improved methods for image retrieval, understanding, and context- specific visual generation to further enhance instruction. Acknowledgments. We are grateful for the support provided by National Science Foundation under Grant No. 2247790 and Grant No. 2112532. We also wish to thank Dr. Emily G. Weigel and Joon Kum for support in creation of assessments. References 1. huridocs/pdf-document-layout-analysis (Dec 2025), https://github.com/huridocs/ pdf-document-layout-analysis, original-date: 2024-05-06T09:36:50Z 2. Baker, R.S., Pankiewicz, M., Liu, X.: A Step Towards Adaptive Online Learning: Exploring the Role of GPT as Virtual Teaching Assistants in Online Education (Jul 2024). https://doi.org/10.35542/osf.io/rw45b, https://osf.io/rw45b 7. CONCLUSION15 3. Bland, T.: Enhancing Medical Student Engagement Through Cinematic Clinical Narratives: Multimodal Generative AI-Based Mixed Methods Study. JMIR Medi- cal Education 11, e63865 (Jan 2025) 4. Chen, S.S., Lee, J., Liang, P.P.: Interactive Sketchpad: A Multimodal Tutoring System for Collaborative, Visual Problem-Solving. CHI Extended Abstracts 2025 (Apr 2025) 5. Chen, X., Wu, D.: Automatic Generation of Multimedia Teaching Materials Based on Generative AI: Taking Tang Poetry as an Example. IEEE Transactions on Learning Technologies 17, 1327â1340 (2024) 6. Deng, R., Jiang, M., Yu, X., Lu, Y., Liu, S.: Does chatgpt enhance student learn- ing? a systematic review and meta-analysis of experimental studies. Computers & Education 227, 105224 (2025) 7. Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., GaĆĄeviÄ, D.: Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology (2024) 8. Goel, A.K., Polepeddi, L.: Jill Watson: A Virtual Teaching Assistant for Online Education. In: Education at scale: Engineering online teaching and learning. NY: Routledge. (2018) 9. Julianne Zedalis, John Eggebrecht: Biology for APÂź Courses. OpenStax, Houston, Texas (Mar 2018), https://openstax.org/details/books/biology-ap-courses 10. Kakar, S., Maiti, P., Taneja, K., Nandula, A., Nguyen, G., Zhao, A., Nandan, V., Goel, A.: Jill Watson: Scaling and Deploying an AI Conversational Agent in Online Classrooms. In: Intelligent Tutoring Systems. p. 78â90 (2024) 11. Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., Yih, W.t.: Dense Passage Retrieval for Open-Domain Question Answering. In: EMNLP 2020. p. 6769â6781 (2020) 12. Kazemitabaar, M., Huang, O., Suh, S., Henley, A.Z., Grossman, T.: Exploring the design space of cognitive engagement techniques with AI-generated code for enhanced learning. In: Intelligent User Interfaces. p. 695â714 (2025) 13. Kool, W., McGuire, J.T., Rosen, Z.B., Botvinick, M.M.: Decision making and the avoidance of cognitive demand. Journal of Experimental Psychology. General 139(4), 665â682 (Nov 2010) 14. Korbach, A., BrĂŒnken, R., Park, B.: Measurement of cognitive load in multimedia learning: a comparison of different objective measures. Instructional Science 45(4), 515â536 (2017) 15. Mayer, R.E.: Multimedia learning. In: Psychology of Learning and Motivation, vol. 41, p. 85â139. Academic Press (2002) 16. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., Schulman, J.: WebGPT: Browser-assisted question-answering with human feedback (Jun 2022), arXiv:2112.09332 17. Oppenheimer, D.M.: The secret life of fluency. Trends in Cognitive Sciences 12(6), 237â241 (Jun 2008) 18. Robertson, S., Walker, S., Jones, S., Hancock-Beaulieu, M., Gatford, M.: Okapi at TREC-3. In: Text Retrieval Conference (1994) 19. Singh, A., Guan, Z., Rieh, S.Y.: Enhancing critical thinking in generative AI search with metacognitive prompts. Assocication for Information Science and Technology 62(1), 672â684 (2025) 20. Sweller, J.: Cognitive Load Theory. In: Psychology of Learning and Motivation, vol. 55, p. 37â76. Academic Press (2011) 16K. Taneja et al. 21. Taneja, K., Goel, A.K.: MuDoC: An Interactive Multimodal Document-grounded Conversational AI System. In: AAAI Machine Learning and Knowledge Engineer- ing (MAKE) for Trustworthy Multimodal and Generative AI (2025) 22. Taneja, K., Maiti, P., Kakar, S., Guruprasad, P., Rao, S., Goel, A.K.: Jill Watson: A Virtual Teaching Assistant Powered by ChatGPT. In: AIED 2024. p. 324â337 (2025) 23. Taneja, K., Singh, A., Goel, A.K.: Towards a Multimodal Document-Grounded Conversational AI System for Education. In: AIED 2025. p. 92â99 (2025) 24. Wu, R., Yu, Z.: Do AI chatbots improve students learning outcomes? Evidence from a meta-analysis. British Journal of Educational Technology 55(1), 10â33 (2024) 25. Wölfel, M., Shirzad, M.B., Reich, A., Anderer, K.: Knowledge-Based and Generative-AI-Driven Pedagogical Conversational Agents: A Comparative Study of Griceâs Cooperative Principles and Trust. Big Data and Cognitive Computing 8(1), 2 (2024) 26. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K.R., Cao, Y.: ReAct: Synergizing Reasoning and Acting in Language Models. In: ICLR 2023 (Sep 2022) 27. Zhai, C., Wibowo, S., Li, L.D.: The effects of over-reliance on AI dialogue systems on studentsâ cognitive abilities: a systematic review. Smart Learning Environments 11(1), 28 (Jun 2024) 28. Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid Loss for Language Image Pre-Training. In: ICCV 2023. p. 11941â11952. IEEE (Oct 2023)