Paper deep dive
Rethinking Health Agents: From Siloed AI to Collaborative Decision Mediators
Ray-Yuan Chung, Xuhai Xu, Ari Pollack
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 92%
Last extracted: 3/27/2026, 1:19:21 AM
Summary
This paper proposes a shift in healthcare AI design from siloed, single-user conversational agents to collaborative 'AI mediators' embedded within multi-stakeholder care interactions. Using a pediatric chronic kidney disease case study, the authors demonstrate that current AI tools often exacerbate fragmented situational awareness and misaligned goals. They introduce a framework for AI collaborators that surface contextual information, reconcile mental models, and scaffold shared understanding while maintaining human decision authority.
Entities (4)
Relation Signals (3)
AI Collaborator â mediates â Multi-stakeholder Care
confidence 95% ¡ AI collaborator is embedded within triadic (or multi-party) care interactions
AI Collaborator â improves â Team Situational Awareness
confidence 92% ¡ AI collaborator can be designed to strengthen shared awareness across stakeholders.
Siloed AI â exacerbates â Misalignment
confidence 90% ¡ siloed use of general-purpose AI tools does little to address these collaboration gaps.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language model based health agents are increasingly used by health consumers and clinicians to interpret health information and guide health decisions. However, most AI systems in healthcare operate in siloed configurations, supporting individual users rather than the multi-stakeholder relationships central to healthcare. Such use can fragment understanding and exacerbate misalignment among patients, caregivers, and clinicians. We reframe AI not as a standalone assistant, but as a collaborator embedded within multi-party care interactions. Through a clinically validated fictional pediatric chronic kidney disease case study, we show that breakdowns in adherence stem from fragmented situational awareness and misaligned goals, and that siloed use of general-purpose AI tools does little to address these collaboration gaps. We propose a conceptual framework for designing AI collaborators that surface contextual information, reconcile mental models, and scaffold shared understanding while preserving human decision authority.
Tags
Links
- Source: https://arxiv.org/abs/2603.24986v1
- Canonical: https://arxiv.org/abs/2603.24986v1
Trouble viewing inline? Open PDF directly â
Full Text
22,150 characters extracted from source content.
Expand or collapse full text
Rethinking Health Agents: From Siloed AI to Collaborative Decision Mediators Ray-Yuan Chung raychung@uw.edu University of Washington Seattle, WA, USA Xuhai âOrsonâ Xu x2489@cumc.columbia.edu Columbia University New York, NY, USA Ari Pollack ari.pollack@seattlechildrens.org Seattle Childrenâs Hospital Seattle, WA, USA Abstract Large language model (LLM)âbased health agents are increasingly used by health consumers and clinicians to interpret health in- formation and guide health decisions. However, most AI systems in healthcare operate in siloed configurations, supporting individ- ual users rather than the multi-stakeholder relationships central to healthcare. Such use can fragment understanding and exacer- bate misalignment among patients, caregivers, and clinicians. We reframe AI not as a standalone assistant, but as a collaborator em- bedded within multi-party care interactions. Through a clinically validated fictional pediatric chronic kidney disease case study, we show that breakdowns in adherence stem from fragmented sit- uational awareness and misaligned goals, and that siloed use of general-purpose AI tools does little to address these collaboration gaps. We propose a conceptual framework for designing AI col- laborators that surface contextual information, reconcile mental models, and scaffold shared understanding while preserving human decision authority. CCS Concepts ⢠Computing methodologiesâArtificial intelligence;⢠Human- centered computingâ Human computer interaction (HCI). Keywords HumanâAI Collaboration, Collaborative Decision-Making, Pedi- atric Chronic Care, Health AI ACM Reference Format: Ray-Yuan Chung, Xuhai âOrsonâ Xu, and Ari Pollack. 2026. Rethinking Health Agents: From Siloed AI to Collaborative Decision Mediators. In CHI â26 Workshop on Human-Agent Collaboration, April, 2026, Barcelona, Spain. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/n. n 1 Introduction The rapid development of artificial intelligence (AI) has transformed how health consumers and clinicians access and interpret health information. Large language model (LLM)âbased systems are in- creasingly used by consumers to seek medical advice and interpret Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. CHI â26 Workshop on Human-Agent Collaboration, Barcelona, Spain Š 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-x-x-x/Y/M https://doi.org/10.1145/n.n symptoms [13,20], while generative AI tools such as OpenEvidence support clinicians in summarizing literature and assisting with diag- nostic reasoning [9]. As we move toward the era of personal health agents [7], AI is becoming embedded in everyday health decision- making. Despite advances in AI capabilities, current systems are largely designed for individual use and are not structured to support the multi-stakeholder interactions that characterize most clinical care. This parallel, siloed adoption raises important concerns: over- reliance on conversational agents may reduce critical evaluation of health information, particularly when responses are presented with high confidence, and clinicians have expressed concerns about accuracy, provenance, and accountability when such tools are used to search for or validate health-related information [1]. Although prior work has explored AI systems that support multi- ple users [16,19] in healthcare, these systems primarily facilitate in- formation exchange rather than collaborative decision-making. For example, Talk2Care collects health information from older adults at home and transmits it to healthcare providers for clinical decision- making [19]. Similarly, the ARCH chatbot gathers information from children and shares it with parents and providers [16]. In these sys- tems, AI primarily functions as an information conduitâcollecting data from one party and passing it to anotherâwithout mediating perspectives, clarifying misunderstandings, or scaffolding shared reasoning across decision-makers. As a result, decision-making may still suffer from fragmented understanding, misaligned expec- tations, or invisible tensions among stakeholders. To address these limitations, we propose an AI collaborator framework designed to participate in and support multi-party health- care decision-making, rather than serving a single end user (Fig- ure 1). Building on prior multi-user systems, we move beyond information transmission toward supporting collaborative decision- making. In our conceptualization, the AI collaborator is embedded within triadic (or multi-party) care interactions and is designed to: â˘Surface and contextualize relevant information among stake- holders, highlighting critical discrepancies, and information gaps. â˘Identify and reconcile differing mental models, assumptions, or priorities among decision-makers. â˘Scaffold shared understanding and coordinated reasoning processes to anticipate potential outcomes and implications of different decisions, while preserving human decision au- thority. This shift in the role of AI raises critical design questions: What level of autonomy should such an AI possess? How should respon- sibility and accountability be distributed across stakeholders? And how can AI systems strengthenârather than erodeâprofessional judgment and patient autonomy? arXiv:2603.24986v1 [cs.HC] 26 Mar 2026 CHI â26 Workshop on Human-Agent Collaboration, April 2026, Barcelona, SpainRay-Yuan Chung, Xuhai âOrsonâ Xu, and Ari Pollack To illustrate both the limitations of current healthcare inter- actions and the opportunities for AI-mediated collaboration, we present a fictional pediatric case study. This case highlights common barriers in todayâs care models, including fragmented communi- cation, mismatched expectations, incomplete contextual knowl- edge, and asymmetries in expertise. We then demonstrate how a deliberately designed AI collaborator could interveneânot as a decision-maker, but as a mediator that aligns stakeholders, clarifies trade-offs, and supports shared understanding. Through this con- ceptual and design exploration, we articulate principles for building AI collaborators as a carefully bounded participant in collaborative care. 2 Design Scenario: A Pediatric Chronic Kidney Disease Case Study 2.1 Case Description Here, we present a fictional chronic kidney disease case study, validated by a pediatric nephrologist and a registered dietitian to ensure clinical relevance and ecological validity: Alex, a 16-year-old high school student, was diagnosed with chronic kidney disease. During a clinic visit, the healthcare team explains that although Alex feels well, his blood pressure is ele- vated and dietary sodium intake must be reduced to protect kidney function. A strict low-sodium diet, daily medication, and regular monitoring are recommended. The teenager and his parent agree without raising concerns, trusting the clinicianâs expertise and assuming that strict adherence is the only responsible course of action. In the following weeks, dietary changes become the primary source of strain. At home, meals are carefully prepared to reduce sodium, but at school Alex struggles to avoid cafeteria food and snacks shared with friends. Reading nutrition labels feels tedious, and declining food in social settings feels isolating. In an effort to help, the parent turns to a general-purpose AI chatbot for low- sodium meal plans. While the system generates structured recom- mendations, the suggestions are generic and insufficiently tailored to the familyâs cultural food preferences, budget constraints, and school schedule. As a result, many proposed meals feel impractical or unfamiliar. Small exceptions begin to accumulateâchips after practice, fast food with classmatesâwhile these lapses are mini- mized at home. Tension grows as the parent reinforces the diet, and Alex becomes increasingly frustrated. By the next clinic visit, adherence has been inconsistent. 2.2 Collaborative Decision-making Needs Healthcare decision-making has shifted from a clinician-driven model toward a collaborative approach that encourages greater patient participation [4]. In pediatric chronic care, collaboration is critical because youth patients, caregivers, and clinicians must jointly navigate ongoing treatment decisions, yet such collaboration is often difficult due to their divergent goals, roles, and constraints [11]. In the chronic kidney disease case, the clinician prioritizes long- term risk reduction, the parent emphasizes compliance and safety, and the teenager values normalcy and autonomy. These differing priorities remain implicit during the visit and later surface as non- adherence to clinician recommendations and tension. Although the treatment plan is clinically appropriate, the breakdown still occurs due to misaligned goals and the absence of shared understanding of trade-offs and contextual constraints. To conceptualize this challenge, we draw on Endsleyâs team sit- uational awareness (SA) framework [5]. Originally developed in high-stakes domains such as aviation, team situational awareness (SA) is defined as the degree to which each team member possesses the situational awareness required for their responsibilities, aligned with a shared team goal and coordinated efforts to achieve that goal [5]. This includes: (1) perception of relevant elements, (2) compre- hension of their meaning, and (3) projection of their future status. Applied to pediatric chronic care, Level 1 includes access to rele- vant clinical indicators (e.g., blood pressure trends, sodium intake patterns), contextual factors (e.g., school routines, social pressures) and values of all decision-makers involved. Level 2 requires shared interpretationâunderstanding what those data mean medically and personally. Level 3 involves anticipating future outcomes, such as the long-term impact of choices or the sustainability of strict dietary rules. Prior work has shown that enhancing team situa- tional awareness supports collaborative decision-making aimed at achieving shared goals in healthcare [14]. In our case, breakdowns occur because these three levels of awareness are fragmented across stakeholders rather than shared. 2.3 AI Collaborator Design Consideration The team situational awareness framework provides a principled lens for designing an AI collaborator in this context. Rather than functioning as an autonomous decision-maker, an AI collaborator can be designed to strengthen shared awareness across stakehold- ers. Prior to appointments, the system could support low-burden meal logging, identify sodium intake patterns, and prompt reflec- tion on situations where adherence feels most difficult (e.g., school lunches, team events). The AI could then synthesize these data into structured summaries that highlight both medical indicators and patient-reported challenges. During clinic visits, the system might surface trade-offs and simulate possible trajectories (e.g., pro- jected blood pressure trends under different adherence scenarios), enabling projection and future-oriented discussion. This design contrasts fundamentally from the siloed use of gen- eral-purpose AI tools. When patients or caregivers independently consult AI chatbots, the resulting advice is often decontextualized, disconnected from clinical records, and invisible to clinicians. Such siloed interactions can introduce inconsistencies, reinforce misun- derstandings, or create parallel decision pathways that undermine collaborative decision-making. In contrast, an AI collaborator em- bedded within the care relationship generates shared artifacts visi- ble to all stakeholders, mediates perspectives, and aligns contextual realities with clinical goals. The clinician retains decision author- ity, and the teenager retains agency; the AI supports sustainable collaboration without replacing human judgment. 3 Future Directions We invite the HCI community to further explore the following research directions for future AI collaborators in healthcare. (1) In different health-related contexts and tasks, what levels of agent autonomy are preferred by decision-makers? Rethinking Health Agents: From Siloed AI to Collaborative Decision MediatorsCHI â26 Workshop on Human-Agent Collaboration, April 2026, Barcelona, Spain Figure 1: Proposed diagram comparing siloed AI use with an embedded AI collaborator in multi-stakeholder care. Across domains like software engineering, and autonomous ve- hicles, we are observing a broader shift from AI as passive tools toward AI as active collaboratorsâand in some contexts, even to- ward configurations where humans become observers of automated systems [6]. Healthcare, however, presents distinct ethical and prac- tical constraints. Licensed clinicians and patients are expected to retain final decision-making authority, especially patients must retain agency over their care. In addition, patients of different ages vary in their desire and readiness to take responsibility for their care [3]. Therefore, designing AI collaborators therefore requires careful calibration of autonomyâbalancing guidance, delegation, and oversightâbased on developmental stage, task complexity, and stakeholder roles. (2) When AI collaborators are introduced into multi-stake- holder care teams, do they mitigate or exacerbate tensions in humanâhuman relationships? For example, how should systems respond when AI-generated suggestions conflict with clinician rec- ommendations or caregiver preferences? These questions apply to any context involving multiple parties, particularly caregivers who play active roles in decision making, including pediatrics, mental health [15,18], and long term care [10,12]. Prior work shows that mismatched expectations regarding caregiver participation versus patient autonomy often create friction in collaborative decision- making [2,8]. AI collaborators must therefore be designed with explicit mechanisms for transparency, role clarity, and conflict me- diation, while accounting for evolving patientâcaregiver dynamics across developmental stages [17]. 4 Conclusion In this work, we reframed AI in healthcare not as a standalone assistant, but as a collaborator embedded within multi-stakeholder care relationships. Through a fictional pediatric chronic kidney disease case study grounded in clinical validation, we illustrated how breakdowns in adherence stem from fragmented situational awareness and misaligned goals across patients, caregivers, and clin- icians. Drawing on team situational awareness theory, we outlined how AI collaborators can strengthen shared perception, compre- hension, and projection without displacing human authority or autonomy. By positioning AI as a mediator that surfaces contextual information, reconciles mental models, and scaffolds longitudinal decision-making, this work offers a conceptual foundation for de- signing ethically grounded collaborative AI systems in healthcare. References [1]Andrew M. Bean, Rebecca E. Payne, Guy Parsons, Hannah R. Kirk, Juan Ciro, Rafael Mosquera-GĂłmez, Sara HincapiĂŠ, Aruna S. Ekanayaka, Lionel Tarassenko, Luc Rocher, and Adam Mahdi. 2026. Reliability of LLMs as medical assistants for the general public: a randomized preregistered study. Nature Medicine 32, 3 (2026), 654â662. doi:10.1038/s41591-025-04074-y [2] Laura Boland, Ian D. Graham, France LĂŠgarĂŠ, Krystina Lewis, Janet Jull, Allyson Shephard, Margaret L. Lawson, Alexandra Davis, Audrey Yameogo, and Dawn Stacey. 2019. Barriers and facilitators of pediatric shared decision-making: a systematic review. Implementation Science 14, 1 (Dec. 2019), 7. doi:10.1186/s13012- 018-0851-5 [3]Julia C. Dunbar, Emily Bascom, Wanda Pratt, Jaime Snyder, Jodi M. Smith, and Ari H. Pollack. 2022. My Kidney Identity: Contextualizing pediatric patients and their families kidney transplant journeys. Pediatric Transplantation 26, 7 (Nov. 2022), e14343. doi:10.1111/petr.14343 [4] Glyn Elwyn, Dominick Frosch, Richard Thomson, Natalie Joseph-Williams, Amy Lloyd, Paul Kinnersley, Emma Cording, Dave Tomson, Carole Dodd, Stephen Rollnick, Adrian Edwards, and Michael Barry. 2012. Shared Decision Making: A Model for Clinical Practice. Journal of General Internal Medicine 27, 10 (oct 2012), 1361â1367. doi:10.1007/s11606-012-2077-6 [5] Mica R. Endsley. 2016. Designing for Situation Awareness: An Approach to User- Centered Design, Second Edition (2 ed.). CRC Press, Boca Raton. doi:10.1201/b11371 [6]K. J. Kevin Feng, David W. McDonald, and Amy X. Zhang. 2025. Levels of Autonomy for AI Agents. arXiv:2506.12469 [cs.HC] https://arxiv.org/abs/2506. 12469 [7] A. Ali Heydari, Ken Gu, Vidya Srinivas, Hong Yu, Zhihan Zhang, Yuwei Zhang, Akshay Paruchuri, Qian He, Hamid Palangi, Nova Hammerquist, Ahmed A. Met- wally, Brent Winslow, Yubin Kim, Kumar Ayush, Yuzhe Yang, Girish Narayan- swamy, Maxwell A. Xu, Jake Garrison, Amy Armento Lee, Jenny Vafeiadou, Ben Graef, Isaac R. Galatzer-Levy, Erik Schenck, Andrew Barakat, Javier Perez, Jacqueline Shreibati, John Hernandez, Anthony Z. Faranesh, Javier L. Prieto, Connor Heneghan, Yun Liu, Jiening Zhan, Mark Malhotra, Shwetak Patel, Tim Althoff, Xin Liu, Daniel McDuff, and Xuhai "Orson" Xu. 2025. The Anatomy of a Personal Health Agent. arXiv:2508.20148 [cs.AI] https://arxiv.org/abs/2508.20148 [8]Matthew K. Hong, Lauren Wilcox, Daniel Machado, Thomas A. Olson, and Stephen F. Simoneaux. 2016. Care Partnerships: Toward Technology to Support Teensâ Participation in Their Health Care. Proceedings of the SIGCHI conference on human factors in computing systems . CHI Conference 2016 (May 2016), 5337â5349. doi:10.1145/2858036.2858508 [9] R. T. Hurt, C. R. Stephenson, E. A. Gilman, C. A. Aakre, I. T. Croghan, M. S. Mundi, K. Ghosh, and J. Edakkanambeth Varayil. 2025. The Use of an Artificial Intelligence Platform OpenEvidence to Augment Clinical Decision-Making for Primary Care Physicians. Journal of Primary Care & Community Health 16 (2025), 21501319251332215. doi:10.1177/21501319251332215 Epub 2025 Apr 16. [10]Luzan Koster, Hilde Verbeek, Bernadette de Boer, and Jan P. H. Hamers. 2021. It takes three to tango: An ethnography of triadic involvement of residents, families and nurses in long-term dementia care. Health Expectations 24, 4 (2021), 1305â1315. doi:10.1111/hex.13224 CHI â26 Workshop on Human-Agent Collaboration, April 2026, Barcelona, SpainRay-Yuan Chung, Xuhai âOrsonâ Xu, and Ari Pollack [11]Victoria A. Miller. 2018. Involving Youth With a Chronic Illness in Decision- making: Highlighting the Role of Providers. Pediatrics 142, Suppl 3 (Nov. 2018), S142âS148. doi:10.1542/peds.2018-0516D [12] Katharina Niedling, Stefanie Richter, and Kerstin Hämel. 2025. Triadic relation- ships in home care nursing: an integrative review of the views and experiences of older couples and nurses. BMC Nursing 24, 1 (2025), 671. doi:10.1186/s12912- 025-03378-1 Published 2025 Jun 27. [13]Akshay Paruchuri, Maryam Aziz, Rohit Vartak, Ayman Ali, Best Uchehara, Xin Liu, Ishan Chatterjee, and Monica Agrawal. 2025. âWhatâs Up, Doc?â: Analyzing How Users Seek Health Information in Large-Scale Conversational AI Datasets. In Findings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Linguistics, Suzhou, China, 2312â2336. doi:10. 18653/v1/2025.findings-emnlp.125 [14]Ari H Pollack, Sonali R Mishra, Calvin Apodaca, Maher Khelifi, Shefali Haldar, and Wanda Pratt. 2020. Different roles with different goals: Designing to support shared situational awareness between patients and clinicians in the hospital. Journal of the American Medical Informatics Association : JAMIA 28, 2 (Nov. 2020), 222â231. doi:10.1093/jamia/ocaa198 [15]F. Schuster, F. HolzhĂźter, S. Heres, and J. Hamann. 2021. âTriadicâ shared deci- sion making in mental health: Experiences and expectations of service users, caregivers and clinicians in Germany. Health Expectations 24, 2 (2021), 507â515. doi:10.1111/hex.13192 [16]Woosuk Seo, Young-Ho Kim, Ji Eun Kim, Megan Tao Fan, Mark S. Ackerman, Sung Won Choi, and Sun Young Park. 2025. Enhancing Pediatric Communication: The Role of an AI-Driven Chatbot in Facilitating Child-Parent-Provider Interac- tion. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1â16. doi:10.1145/3706598.3713134 [17] Tammy Toscos, Kay Connelly, and Yvonne Rogers. 2012. Best intentions: health monitoring technology and children. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Austin Texas USA, 1431â1440. doi:10.1145/2207676.2208603 [18]R. Tuijt, J. Rees, R. Frost, J. Wilcock, J. Manthorpe, and G. Rait. 2021. Exploring how triads of people living with dementia, carers and health care professionals func- tion in dementia health care: A systematic qualitative review and thematic syn- thesis. Dementia (London) 20, 3 (2021), 1080â1104. doi:10.1177/1471301220915068 [19]Ziqi Yang, Xuhai Xu, Bingsheng Yao, Ethan Rogers, Shao Zhang, Stephen Intille, Nawar Shara, Guodong Gordon Gao, and Dakuo Wang. 2024. Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 8, 2, Article 73 (May 2024), 35 pages. doi:10.1145/3659625 [20]H. Yun and T. Bickmore. 2025. Online Health InformationâSeeking in the Era of Large Language Models: Cross-Sectional Web-Based Survey Study. Journal of Medical Internet Research 27 (2025), e68560. doi:10.2196/68560