Paper deep dive

PSI: Shared State as the Missing Layer for Coherent AI-Generated Instruments in Personal AI Agents

Zhiyuan Wang, Erzhen Hu, Mark Rucker, Laura E. Barnes

Year: 2026Venue: arXiv preprintArea: cs.HCType: PreprintEmbeddings: 43

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 4/10/2026, 4:33:02 AM

Summary

PSI is a shared-state architecture designed to transform isolated AI-generated personal software modules into coherent, persistent, and connected instruments. By utilizing a shared personal-context bus, PSI enables cross-module reasoning and synchronized interactions across both GUI and chat interfaces, addressing the fragmentation common in personal informatics.

Entities (5)

PSI · architecture · 100%Shared Personal-Context Bus · software-component · 99%Facai · chat-agent · 98%BoBo · instrument · 95%RyanHub · personal-ai-environment · 95%

Relation Signals (4)

PSI → utilizes → Shared Personal-Context Bus

confidence 100% · PSI consists of three layers: a generation layer... a runtime layer built on the shared personal-context bus

Facai → interactswith → Shared Personal-Context Bus

confidence 98% · Facai... can interpret user questions, coordinate across modules, and reason over the shared personal-context bus

RyanHub → hosts → PSI

confidence 95% · the PSI system is instantiated in RyanHub, a self-developed personal AI environment

BoBo → publishesstateto → Shared Personal-Context Bus

confidence 95% · BoBo publishes its state to PSI’s shared personal-context bus

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Personal AI tools can now be generated from natural-language requests, but they often remain isolated after creation. We present PSI, a shared-state architecture that turns independently generated modules into coherent instruments: persistent, connected, and chat-complementary artifacts accessible through both GUIs and a generic chat agent. By publishing current state and write-back affordances to a shared personal-context bus, modules enable cross-module reasoning and synchronized actions across interfaces. We study PSI through a three-week autobiographical deployment in a self-developed personal AI environment and show that later-generated instruments can be integrated automatically through the same contract. PSI identifies shared state as the missing systems layer that transforms AI-generated personal software from isolated apps into coherent personal computing environments.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

42,204 characters extracted from source content.

Expand or collapse full text

PSI: Shared State as the Missing Layer for Coherent AI-Generated Instruments in Personal AI Agents Zhiyuan Wang University of Virginia Charlottesville, Virginia, USA vmf9pr@virginia.edu Erzhen Hu University of Virginia Charlottesville, Virginia, USA eh2qs@virginia.edu Mark Rucker University of Virginia Charlottesville, Virginia, USA mr2an@virginia.edu Laura Barnes University of Virginia Charlottesville, Virginia, USA lb3dp@virginia.edu /new-app Build an instrument to track my daily timeline from phone + watch sensor. ‘BoBo’ generated and pushed to our phone! Hey, your heart rate is normal for right now. You just finished your workout and you're still on the move. But you need to rest, especially after that bad sleep last night. Let’s get you in bed early tonight! Wait—are you still walking? I see your parking just expired! You should hurry. : Personal AI agent, back-ended by : GUIinstrumentshosting personalcontext HealthLoggerParking OtherPersonalInstruments SharedPersonalContextBus BoBoHealth Parking ChestPress 4 sets 100b Sensing timeline Auto-purchased! Zone: X Vehicle: X*** Duration: 1h 30m Total: $1.75 Isn’t my current heart rate too high? GENERATE FROM USER INTENT SHARED, PERSISTENT INSTRUMENTS SITUATED QUESTIONCROSS-MODULE ANSWER Isn’t my current heart rate too high? Figure 1: PSI at a glance. (1) A user describes a personal need in natural language; an AI generation engine produces a shared-state instrument. (2) Each module publishes structured state to a shared personal-context bus. (3) Both a chat agent and persistent GUIs read from the same bus. (4) This shared state enables grounded, cross-module reasoning and bidirectional actions across the personal computing environment. Abstract Personal AI tools can now be generated from natural-language requests, but they often remain isolated after creation. We present PSI, a shared-state architecture that turns independently generated modules into coherent instruments: persistent, connected, and chat- complementary artifacts accessible through both GUIs and a generic chat agent. By publishing current state and write-back affordances to a shared personal-context bus, modules enable cross-module reasoning and synchronized actions across interfaces. We study PSI through a three-week autobiographical deployment in a self- developed personal AI environment and show that later-generated instruments can be integrated automatically through the same contract. PSI identifies shared state as the missing systems layer that transforms AI-generated personal software from isolated apps into coherent personal computing environments. CCS Concepts • Human-centered computing→Interactive systems and tools. Keywords personal AI, generated interfaces, context-aware systems, personal informatics, AI-native computing 1 Introduction People increasingly rely on a growing ecosystem of personal digital tools: health apps that log workouts and sleep, parking services that track time and payments, calendars, location traces, wearable sensors, and lightweight dashboards for everyday routines. Each tool is useful in isolation, yet everyday personal work rarely stays within a single application, and many data is interconnected. A simple situated question such as “Is my heart rate too high right now? ” may require combining recent workout activity, current motion, sleep quality, and even contextual signals such as whether the user is still walking back to an expiring parking spot. The difficulty is not that any single signal is unavailable; it is that these signals remain fragmented across apps, services, and interfaces. This fragmentation is a long-standing problem in personal in- formatics [5,9,21]. Recent AI coding agents make it increasingly arXiv:2604.08529v1 [cs.HC] 9 Apr 2026 Conference’17, July 2017, Washington, DC, USAZhiyuan Wang, Erzhen Hu, Mark Rucker, and Laura Barnes plausible for one person to generate lightweight, highly person- alized tools from natural-language intent, but generation alone does not solve the integration problem. If each generated artifact becomes another siloed app, personal software scales in quantity rather than coherence [19, 23, 24, 31]. We present PSI, a shared-state architecture for coherent AI- generated instruments for AI-generated personal software. Bor- rowing the term from Beaudouin-Lafon’s instrumental interaction model [3] while extending it to AI-generated personal software, we define an instrument as a generated artifact that is (a) persistent: it remains available without regeneration; (b) connected: it publishes state to a shared personal-context layer and may expose write-back affordances; and (c) complementary to chat: it supports glanceable monitoring while chat handles synthesis, ambiguity resolution, and stateful actions. A module is the full software bundle behind an instrument, its GUI, provider, and optional services. Both persis- tent instruments and a generic chat agent (Facai) operate over the same shared-context bus. The result is not a new coding agent, but a minimal integration contract that lets independently generated modules become legible to one another and to multiple interfaces. We study PSI through a three-week autobiographical deploy- ment [7,32] in RyanHub, a self-developed personal AI environment, together with a broader artifact in which new modules can be gen- erated and then automatically integrated into PSI through the same contract and registration path. The point is not the exact module count, but that independently authored modules can join the shared context after generation. Our aim is a versatile application rather than general utility: the deployment involves a single technically skilled user, and the evaluation is a bounded proof-of-concept rather than a claim about population-wide adoption. This paper makes two contributions: (1)A shared-state architecture for coherent AI-generated instruments that turns independently generated modules into persistent, connected artifacts accessible through both GUI surfaces and chat. (2)Evidence that shared state improves both reasoning and action in personal AI, showing stronger cross-module reasoning than search-only or single-module baselines while preserving reliable write-back across persistent instruments in an autobiographical deployment. (3)An open-sourced artifact including the PSI, RyanHub IOS app, and representative modules, will be open-sourced upon acceptance. 2 Related Work AI-generated software and end-user programming. PSI builds on a long arc of end-user software creation, from task-specific end-user programming environments [31] and end-user software engineering [19] to specification-by-demonstration systems such as SUGILITE and PUMICE [23,24]. Commercial copilots and recent arguments for malleable software similarly shift attention toward software generation as an end-user-facing capability [12,26]. More recent research systems move closer to dynamic UI synthesis and agentic software production [2,4,37,38]. This literature shows that users can increasingly author or request new software artifacts; PSI focuses on a different question: what runtime contract makes many generated personal artifacts cohere after they have been created? Interactive AI interfaces beyond chat. Direct-manipulation and post-WIMP traditions emphasize persistent, inspectable interac- tion objects rather than transient dialogue alone [3,33,36]. Classic visions of ubiquitous and personally meaningful computing simi- larly foreground interfaces that remain embedded in everyday life rather than appearing only on demand [39]. Recent LLM interac- tion work shows the value of visual structure, diagrams, and hybrid interfaces [16,18,30]. PSI adopts the intuition that chat should not be the only interface to personal AI, but pushes it toward an architectural claim: persistent GUI instruments and chat become complementary only when they operate over the same underlying state. Personal informatics, context, and proactive assistance. Personal informatics research has long identified fragmentation and integration as core challenges [5,9,21]. Context-aware comput- ing argues that richer state enables more appropriate assistance [8], while proactive-assistant and mixed-initiative work highlights the long-standing appeal of systems that reduce information and coor- dination burden without removing the user from the loop [14,29]. Recent AI systems such as OmniActions and GLOSS derive assis- tance from multimodal sensing and language models [6,22]. PSI differs in publishing person-scoped, module-produced state that persists across sessions and is shared by both conversational and graphical interfaces, rather than only supporting immediate pre- diction or one-shot interpretation. Method, architectural substrates, and agent memory. Our evidence comes from an autobiographical deployment, a method for systems whose value depends on authentic everyday use [7, 32]. This also aligns with cultural-probe, technology-probe, and research-through-design traditions that use artifacts to surface design knowledge and system tensions [10,11,15,41]. At the archi- tectural level, PSI is closest to interaction substrates [28]: it offers a reusable integration layer rather than a single application. At the agent level, prior work on generative agents, personal LLM agents, user modeling, and personal knowledge ecosystems similarly seeks person-relevant state [25,34,35,40], but typically keeps that state inside an agent or knowledge model rather than exposing it as a shared runtime contract for multiple interfaces. 3 PSI System Overview Prior systems support pieces of the personal AI workflow—generation, sensing, automation, or conversational access—but the experience breaks down after creation because these capabilities remain iso- lated. PSI introduces instruments: persistent, connected, chat- com- plementary artifacts that address this missing layer through a shared personal-context substrate and provider contract, letting independently created modules interoperate and remain accessible through both GUI surfaces and chat. 3.1 Motivating Scenarios Ryan has recently been having trouble making sense of his passive health data. Signals such as motion, step count, heart rate, sleep, and other sensor streams are continuously collected, but the information remains fragmented across separate apps and logs. PSI: Shared State as the Missing Layer for Coherent AI-Generated Instruments in Personal AI AgentsConference’17, July 2017, Washington, DC, USA I want an app that tracks all my health & motion sensors GenerationLayer GUI (sensors + timeline) Service / Backend ShareContext Provider Action endpoints InstrumentGeneration PSI AI generation engine Shared Personal-Context Layer providers publish state here Health ⚡Fitness Activity 喝Meals & Weight Calendar Meetings Gym Persistent GUI Customizeable, galanceablemonitoring Sensor Data Timelineformonitoring BoBo ❤Heart Rate HRV Sleep Score Generic Chat Agent (Facai) Reads bus, reason across apps, responds a b c1 InteractionLayer GUI and Chat Agent Dual-Modailityuse c c2  AI generation engine c3 Figure 2: PSI pipeline and interface walkthrough: PSI turns generated personal apps into persistent, connected, and chat- complementary instruments. (a) Ryan uses PSI to generate BoBo, a personalized health instrument that connects passive sensor streams such as motion, steps, heart rate, and sleep. (b) BoBo publishes its state to PSI’s shared personal-context bus, enabling interoperability with other instruments and apps (e.g., calendar and health logs). (c1) PSI provides a persistent, customizable GUI with a glanceable dashboard and interactive timeline for longitudinal monitoring. (c2) When Ryan asks Facai, “Why do I feel so drained lately?”, the agent retrieves relevant state across connected instruments and jointly reasons over sleep, activity, and calendar load to provide a grounded explanation without requiring Ryan to manually inspect fragmented apps. (c3) BoBo will actively track the user status and nudge the user proactively. Table 1: Capability comparison for personal AI systems.✓= full,∼= partial, – = not supported. Conv.: conversational AI; Gen.: generates persistent GUIs; Ctx: structured personal context; X-Mod: cross-module synthesis; App: app-level write- back. Conv.Gen.CtxX-ModApp ChatGPT / Claude✓ ∼ ∼– Siri / Google Asst.✓– ∼– Shortcuts / IFTTT– ∼ ∼ Home Assistant ∼–✓ v0 / Lovable✓– ∼ M365 Copilot✓– ∼✓ ∼ Copilot / Cursor✓– DynaVis [38]– ∼– SUGILITE [23] ∼– ∼ ∼ OmniActions [22]✓–✓– OpenClaw [1]✓ ∼ ∼ ∼ ∼ PSI✓ With PSI, he quickly generates a personalized modules (Figure 2a) called BoBo (Behavioral Observer Bot), which connects to all of his health- and motion-related sensors. First, Ryan wanted a customized timeline to track these sensor data. Hence, PSI helps Ryan to create a persistent, customizable GUI (Figure 2c1) specific for BoBo that visualizes these signals along an interactive timeline, allowing Ryan to monitor trends over time at a glance. Second, it publishes its state to PSI’s shared personal-context bus (Figure 2b), allowing the app to interoperate with other existing instruments, such as health logging and parking history. At the center of this ecosystem is Facai, a generic chat agent (Figure 2c2) that can interpret user questions, coordinate across modules, and reason over the shared personal-context bus, and nudge user proactively (Figure 2c3). One day, while walking between meetings, Ryan suddenly no- tices that his heart is racing, and he felt so tired lately. Unsure whether this is normal or a sign of something concerning, he asks Facai: “Why do I feel so drained lately?” (Figure 2c2). Instead of forcing Ryan to manually open and compare multiple disconnected apps, Facai retrieves relevant state from BoBo, his other modules, such as Health, and even Calendar history through the shared context bus. By jointly reasoning over these connected signals, Facai helps Ryan discover the likely explanation: he had been rushing from meetings and interviews, exercising too much, with poor sleep the night before. Through PSI, Ryan no longer needs to inspect fragmented apps one by one. Instead, PSI turns generated apps into instruments: persistent, connected, and chat-complementary artifacts that sup- port both glanceable monitoring through GUIs and cross-context reasoning through conversation. Conference’17, July 2017, Washington, DC, USAZhiyuan Wang, Erzhen Hu, Mark Rucker, and Laura Barnes 3.2 PSI Architecture PSI consists of three layers: a generation layer for authoring mod- ules, a runtime layer built on the shared personal-context bus, and an interaction layer that exposes module state through both persis- tent instruments and a chat agent. 3.2.1 Generation Layer. PSI enables an agentic coding workflow rather than a fixed application catalog. Modules are generated through a multi-phase pipeline, consisting of specification, code generation, auto-fix, and compile verification. The architectural contribution is not the pipeline itself but the provider contract each generated module must satisfy. The formal contract is a single Swift protocol (ToolkitDataProvider) requiring a toolkit identifier, rel- evance keywords, and one method—buildContextSummary() -> String?—that returns a tagged, human-readable snapshot of the module’s current state (e.g., today’s sensed events, recent meals, upcoming calendar entries) together with any write-back endpoints it exposes, so the same method serves both read context and action discovery for the chat agent. A co-evolving memory file captures informal conventions (naming, data formats) discovered during development but not enforced at compile time; this is how modules generated on different days fit one architecture. 3.2.2 Shared Personal Context Layer. The shared personal-context layer is a central registry that collects current-state snapshots from all registered modules—both built-in and dynamically generated— and prepends them as a single tagged block before every chat mes- sage. Each snapshot captures today’s data rather than cumulative history; if a module has nothing to report it is silently omitted, so the system degrades gracefully. This turns integration into a local obligation: a new module needs only to implement the provider interface and register, rather than wire into every existing mod- ule. Unlike agent orchestration frameworks where state is scoped to a single task, PSI’s shared context is person-scoped—it persists on-device across tasks and sessions, requires no coordinating task graph, and is consumed by all interfaces. Modules never read each other’s state directly; all cross-module communication is mediated by the LLM through the assembled context. 3.2.3 Interaction Layer: Dual-Modality Use. The interaction layer provides two coordinated interfaces: a persistent GUI (Figure 2c1) and a generic chat agent (Figure 2c2). PSI’s contribution is not sim- ply the coexistence of chat and GUI, but their role as synchronized entry points to the same person-scoped mutable state. Users can inspect state in a persistent instrument, revise it through chat, and immediately verify the effect in the GUI without duplicated state paths or re-specifying intent. For example, in BoBo, the behavioral timeline remains persistently glanceable in the GUI while Facai reasons over the same visible state in follow-up queries. 4 Evidence We evaluate PSI through two complementary evidence lenses [13, 17, 20]: Applications and Proof-of-concept. For evaluation, the PSI system is instantiated in RyanHub, a self- developed personal AI environment comprising four cooperating services: (1) a SwiftUI iOS app (94 Swift files, 36,517 LOC) hosting persistent GUI instruments and the chat interface; (2) a Python bridge server providing a unified REST gateway for on-device data Figure 3: Automated Parking Example (behavioral timeline, health entries, parking state) persisted as local JSON files; (3) a Python dispatcher maintaining WebSocket sessions with the iOS client, injecting shared context server-side, and routing tool calls through the LLM; All services run on localhost; personal data stays on-device by default. 4.1 Versatile Applications PSI supports one shared personal-context contract supports a het- erogeneous tool ecosystem without pairwise integration. We cre- ated 14 modules across behavioral sensing, health, scheduling, parking, reading, vocabulary learning, and several post-pilot self- tracking domains. These include six core modules used during the three-week autobiographical deployment, along with eight addi- tional modules newly generated to validate the extensibility of the generation layer (see the full listing in Appendix C). Here, we present two deployment cases illustrate the payoff of that contract in everyday use, Bobo and Automated Parking. BoBo: A Generalizable Behavioral Sensing Instrument. Beyond the motivating walkthrough, BoBo demonstrates how PSI supports a broader class of persistent behavioral sensing instruments. Rather than serving a single question-answer interaction, BoBo maintains a continuously updated behavioral state that can be accessed through both glanceable GUIs and conversational reasoning. This enables diverse query patterns, including cross-signal synthesis (e.g., re- lating heart rate spikes to location, activity, and calendar load), temporal grounding (e.g., comparing sleep or recovery trends over multiple days), and action-oriented follow-ups (e.g., recommending rest, suppressing evening workouts, or adjusting the next day’s schedule). Because these interactions operate over the same shared personal-context substrate, users can fluidly move between pas- sive monitoring, situated questioning, and proactive intervention without manually reconstructing state across fragmented apps. Automated Parking. Ryan faces a recurring parking challenge: if he does not reserve a spot before 7 a.m., the lot is typically fully booked. To avoid waking up early for this routine, he created a module that automatically books parking on his behalf. The Park- ing module demonstrates the same PSI pattern in a hyper-personal, PSI: Shared State as the Missing Layer for Coherent AI-Generated Instruments in Personal AI AgentsConference’17, July 2017, Washington, DC, USA market-of-one workflow. Tailored to a single user’s weekday park- ing routine, it supports configurable zones, vehicles, and schedules. Through PSI, the generic chat agent Facai can trigger ParkMobile purchases via web automation, while the user may also interact through a persistent GUI. For example, the user can issue a com- mand such as “No parking this Thursday” (Figure 3d) or toggle the same skip state directly in the GUI (Figure 3c). The module can also integrate state from other instruments via the shared personal- context bus (Figure 3b), such as using the calendar’s end-of-day event to infer parking duration. In both modes, chat and GUI op- erate over the same persistent parking state, including schedule, purchase history, and active sessions, as a single source of truth. More broadly, this pattern generalizes to other recurring market-of- one routines, such as gym bookings, commute ticketing, medication reminders, and home-device schedules. 4.2 Proof-of-concept Evaluation To understand the benefit and performance of generic chat agent powered by shared-personal context bus across modules, we evalu- ated three conditions: (1) Shared Personal-Context; (2) Search-Only; (3) Single-Module (Figure 4a-c). In Search-Only, the agent received no preassembled personal snapshot and had to recover relevant state opportunistically from user database from in the file system during the turn (comparable to how OpenClaw works). In Single- Module, we ran the same task once per candidate module, and report the best-rated one-module variant for each condition. Data Collection Tasks. We evaluated the generic chat agent, pow- ered by the shared personal-context bus, using a frozen three-week dataset collected in a single-user, self-authored deployment setting. The dataset included both proactively logged personal data (e.g., food intake and diary entries) and passively sensed data streams from a smartwatch, such as heart rate, location, and ambient noise levels. We then generated a set of synthetic user queries (N=50) at different time points to mimic real-world use, where the system has access to only the data available at that moment. The 50 evaluation queries were organized into three reasoning task categories commonly studied in prior work: cross-module syn- thesis [6,8,9,18,21] (e.g., “Given my today’s done, what is the best next step for tonight?”), temporal grounding and control [5,9,21] (e.g., “How has my heart rate changed over the last week?”), and chain of actions [22–24] (e.g., “Check my calorie intake, then activ- ity, then net balance.”). The distribution of simulated queries across these categories was derived from the questions the single user asked during the three-week deployment. Aside from reasoning tasks, we tested 20 write-back action tasks across five domains (parking, food, activity, diary, and dynamically generated modules), to assess whether state changes initiated from chat were correctly reflected in the corresponding GUIs, as validated by sandbox state inspection. Metrics and Results. We evaluated the resulting responses us- ing fulfillment, task success, and latency. Fulfillment measures the fraction of gold-specification criteria satisfied by a response (contin- uous, 0–1), operationalized as the proportion of relevant modules correctly identified among all ground-truth modules. Task success is a stricter binary metric that requires all relevant modules to be (a) Context Bus(b) Search-Only(c) Single-Module Figure 4: Benchmark illustrations. C = Context; M = Module. selected (0 or 1). Because these tasks require integrating longitu- dinal evidence across many modules and time windows, holistic human rating would itself require reconstructing fragmented per- sonal traces, a burden that personal informatics research has long identified as difficult in practice [5,9,21]. We therefore use an in- dependent language-model judge (Claude Opus 4.6) as a pragmatic proxy rather than a substitute for human evaluation. Shared personal-context achieved a mean fulfillment score of 0.88, substantially outperforming Search-Only (0.63) and Single- Module (0.27). Task success followed the same pattern at 0.68, 0.32, and 0.08, respectively. We also measured latency. End-to-end latency is not monotonic with context size. On reasoning tasks, the mean successful latency was 25 s for Shared Context, 29 s for Search-Only, and 23 s for Single-Module. On write-back actions, we measure the task success on whether the GUI task were precisely completed by the chat agent or not. The pattern reverses: Shared and Single-Module both achieved 19 of 20 validated state changes (95%), while Search-Only achieved 8 of 20 (40%). In summary, the gap between conditions reveals where shared context’s value lies: for reasoning, it enables cross-module synthesis that no single module can provide; for actions, it provides write- path discovery that ad hoc search cannot reliably achieve. Together, the reasoning and action results provide evidence for bidirectional information access: shared context lets chat both read personal state (grounded reasoning) and write it back (stateful actions). 5 Discussion and Conclusion From generated apps to coherent instruments. The deployment suggests that the value of PSI lies not in any single module but in the division of labor that instruments enable. Instruments supported glanceable monitoring and routine control—the BoBo timeline and parking controls were often useful without opening chat at all— while chat handled cross-module synthesis and stateful actions over the same shared state. Without the shared-context layer, each generated module would remain a local success but a system-level dead end; the provider contract is what turns isolated apps into instruments by making integration a local obligation rather than a pairwise problem. Current design bets and limitations. Unconditional injec- tion trades prompt length for recall. At current scale the injected context is still manageable, but routing and selection will mat- ter more as module counts grow. The deployment also surfaces a system-specific risk: context pollution [27]. Because all interfaces trust provider summaries, stale or misleading summaries can de- grade responses system-wide. As the number of modules grows, Conference’17, July 2017, Washington, DC, USAZhiyuan Wang, Erzhen Hu, Mark Rucker, and Laura Barnes overlapping or redundant entries logged across modules can intro- duce ambiguity into the assembled context, causing the chat agent to misattribute, double-count, or contradict itself, a class of failure inherent to shared-state architectures that would not arise in siloed apps. Future work. Our evidence comes from a single technically skilled user over three weeks, so the paper should be read as a proof-of-concept for versatile applications. The reusable instrument that includes persistent GUI and generic chat agent are intended to be user-agnostic, but the current summaries, modules, and deploy- ment practices are specific to one personal ecology. Important open questions include routing at larger module counts, privacy and au- thorization for person-scoped actions, and how to make compliant module generation accessible to non-programmers. We also plan to release the reusable artifact components after publication, which should make these follow-on questions easier to study. References [1]2025. OpenClaw: An Open-Source Framework for Personal AI Agents. https: //github.com/openclaw/openclaw. [2]Anthropic. 2025. Claude Code: Anthropic’s Agentic Coding Tool. https://docs. anthropic.com/en/docs/claude-code. [3]Michel Beaudouin-Lafon. 2000. Instrumental Interaction: An Interaction Model for Designing Post-WIMP User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’00). 446–453. doi:10.1145/332040. 332473 [4] Yining Cao, Peiling Jiang, and Haijun Xia. 2025. Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Model. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Article 686, 20 pages. doi:10.1145/3706598.3713285 [5] Eun Kyoung Choe, Nicole B. Lee, Bongshin Lee, Wanda Pratt, and Julie A. Kientz. 2014. Understanding quantified-selfers’ practices in collecting and exploring personal data. In Proceedings of the SIGCHI Conference on Human Factors in Com- puting Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 1143–1152. doi:10.1145/2556288.2557372 [6] Akshat Choube, Ha Le, Jiachen Li, Kaixin Ji, Vedant Das Swain, and Varun Mishra. 2025. GLOSS: Group of LLMs for Open-ended Sensemaking of Passive Sensing Data for Health and Wellbeing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 9, 3, Article 76 (Sept. 2025), 32 pages. doi:10.1145/3749474 [7]Audrey Desjardins and Aubree Ball. 2018. Revealing Tensions in Autobiographical Design in HCI. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong, China) (DIS ’18). Association for Computing Machinery, New York, NY, USA, 753–764. doi:10.1145/3196709.3196781 [8]Anind K. Dey. 2001. Understanding and Using Context. Personal Ubiquitous Comput. 5, 1 (Jan. 2001), 4–7. doi:10.1007/s007790170019 [9] Daniel A. Epstein, An Ping, James Fogarty, and Sean A. Munson. 2015. A lived informatics model of personal informatics. In Proceedings of the 2015 ACM Interna- tional Joint Conference on Pervasive and Ubiquitous Computing (Osaka, Japan) (Ubi- Comp ’15). Association for Computing Machinery, New York, NY, USA, 731–742. doi:10.1145/2750858.2804250 [10]Bill Gaver, Tony Dunne, and Elena Pacenti. 1999. Design: Cultural probes. Inter- actions 6, 1 (Jan. 1999), 21–29. doi:10.1145/291224.291235 [11]William W. Gaver. 2012. What Should We Expect from Research Through Design?. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12). 937–946. doi:10.1145/2207676.2208538 [12] GitHub. 2021. GitHub Copilot. https://github.com/features/copilot. [13]Saul Greenberg and Bill Buxton. 2008. Usability Evaluation Considered Harmful (Some of the Time). In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 111–120. doi:10.1145/1357054.1357074 [14]Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Pittsburgh, Pennsylvania, USA) (CHI ’99). Association for Computing Machinery, New York, NY, USA, 159–166. doi:10.1145/302979.303030 [15]Hilary Hutchinson, Wendy Mackay, Bo Westerlund, Benjamin B. Bederson, Al- lison Druin, Catherine Plaisant, Michel Beaudouin-Lafon, Stéphane Conversy, Helen Evans, Heiko Hansen, Nicolas Roussel, and Björn Eiderbäck. 2003. Technol- ogy probes: inspiring design for and with families. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA) (CHI ’03). Association for Computing Machinery, New York, NY, USA, 17–24. doi:10.1145/642611.642616 [16]Peiling Jiang, Jude Rayan, Steven P. Dow, and Haijun Xia. 2023. Graphologue: Exploring Large Language Model Responses with Interactive Diagrams. In Pro- ceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST ’23). Association for Computing Machinery, New York, NY, USA, Article 3, 20 pages. doi:10.1145/3586183.3606737 [17]Daniel R. Olsen Jr. 2007. Evaluating User Interface Systems Research. In Proceed- ings of the 20th Annual ACM Symposium on User Interface Software and Technology. 251–258. doi:10.1145/1294211.1294256 [18] Young-Ho Kim, Bongshin Lee, Arjun Srinivasan, and Eun Kyoung Choe. 2021. Data@Hand: Fostering Visual Exploration of Personal Data on Smartphones Leveraging Speech and Touch Interaction. In Proceedings of the 2021 CHI Con- ference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 462, 17 pages. doi:10.1145/3411764.3445421 [19]Andrew J. Ko, Robin Abraham, Laura Beckwith, Alan Blackwell, Margaret Burnett, Martin Erwig, Chris Scaffidi, Joseph Lawrance, Henry Lieberman, Brad Myers, Mary Beth Rosson, Gregg Rothermel, Mary Shaw, and Susan Wiedenbeck. 2011. The State of the Art in End-User Software Engineering. Comput. Surveys 43, 3, Article 21 (2011). doi:10.1145/1922649.1922658 [20]David Ledo, Steven Houben, Jo Vermeulen, Nicolai Marquardt, Lora Oehlberg, and Saul Greenberg. 2018. Evaluation Strategies for HCI Toolkit Research. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Article 36, 17 pages. doi:10.1145/3173574.3173610 [21]Ian Li, Anind Dey, and Jodi Forlizzi. 2010. A stage-based model of personal informatics systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 557–566. doi:10.1145/1753326.1753409 [22] Jiahao Nick Li, Yan Xu, Tovi Grossman, Stephanie Santosa, and Michelle Li. 2024. OmniActions: Predicting Digital Actions in Response to Real-World Multimodal Sensory Inputs with LLMs. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 8, 22 pages. doi:10.1145/ 3613904.3642068 [23]Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. 2017. SUGILITE: Creating Multimodal Smartphone Automation by Demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 6038–6049. doi:10.1145/3025453.3025483 [24] Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, and Brad A. Myers. 2019. PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 577–589. doi:10.1145/3332165.3347899 [25]Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al.2024. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459 (2024). [26] Geoffrey Litt. 2023. Malleable Software in the Age of LLMs. https://w. geoffreylitt.com/2023/03/25/llm-end-user-programming.html. [27] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics 12 (2024), 157–173. [28]Wendy E. Mackay and Michel Beaudouin-Lafon. 2025. Interaction Substrates: Combining Power and Simplicity in Interactive Systems. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, Article 687, 16 pages. doi:10. 1145/3706598.3714006 [29]Pattie Maes. 1994. Agents that reduce work and information overload. Commun. ACM 37, 7 (July 1994), 30–40. doi:10.1145/176789.176792 [30]Damien Masson, Sylvain Malacria, Géry Casiez, and Daniel Vogel. 2024. Direct- GPT: A Direct Manipulation Interface to Interact with Large Language Models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 975, 16 pages. doi:10.1145/3613904.3642462 [31]Bonnie A. Nardi. 1993. A Small Matter of Programming: Perspectives on End User Computing. MIT Press. [32] Carman Neustaedter and Phoebe Sengers. 2012. Autobiographical design in HCI research: designing and learning through use-it-yourself. In Proceedings of the Designing Interactive Systems Conference (Newcastle Upon Tyne, United Kingdom) (DIS ’12). Association for Computing Machinery, New York, NY, USA, 514–523. doi:10.1145/2317956.2318034 [33]Don Norman. 2013. The Design of Everyday Things: Revised and Expanded Edition. Basic Books. [34]Joon Sung Park, Joseph C. O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User PSI: Shared State as the Missing Layer for Coherent AI-Generated Instruments in Personal AI AgentsConference’17, July 2017, Washington, DC, USA Interface Software and Technology (UIST ’23). Article 2, 22 pages. doi:10.1145/ 3586183.3606763 [35]Omar Shaikh, Shardul Sapkota, Shan Rizvi, Eric Horvitz, Joon Sung Park, Diyi Yang, and Michael S. Bernstein. 2025. Creating General User Models from Com- puter Use. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST ’25). doi:10.1145/3746059.3747722 [36] Ben Shneiderman. 1983. Direct Manipulation: A Step Beyond Programming Languages. Computer 16, 8 (1983), 57–69. [37] Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, and Haijun Xia. 2024. Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 644, 26 pages. doi:10.1145/3613904.3642400 [38]Priyan Vaithilingam, Elena L. Glassman, Jeevana Priya Inala, and Chenglong Wang. 2024. DynaVis: Dynamically Synthesized UI Widgets for Visualization Editing. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 985, 17 pages. doi:10.1145/3613904.3642639 [39]Mark Weiser. 1991. The Computer for the 21st Century. Scientific American 265, 3 (1991), 94–104. [40] Dora Zhao, Diyi Yang, and Michael S. Bernstein. 2025. Knoll: Creating a Knowl- edge Ecosystem for Large Language Models. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST ’25). Article 140, 23 pages. doi:10.1145/3746059.3747711 [41] John Zimmerman, Jodi Forlizzi, and Shelley Evenson. 2007. Research Through Design as a Method for Interaction Design Research in HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07). 493–502. doi:10.1145/1240624.1240704 A Evaluation Task Set The evaluation comprises 50 reasoning tasks across three fami- lies (cross-module synthesis, temporal grounding, and multi-step chains) and 20 write-back action tasks across five domains (parking, food, activity, diary, and dynamically generated modules). The full task set with queries, gold-specification criteria, and per-task scores is available as supplementary material. Table 2 lists representative action tasks. Table 2: Action tasks (푁=20, representative subset). Validated by sandbox state inspection. Shared = 19/20, Single = 19/20, Search = 8/20. QueryDomainSh. Se. Skip parking for tomorrow.Parking✓– Actually restore parking for to- morrow. Parking✓ Skip parking for all of next week. Parking✓ Log lunch: egg and chicken curry with rice. Health (food)✓– Log a 30 minute run, 300 cal.Health (activity)✓– Log my weight: 87.5 kg.Health (weight)✓– Add a diary entry: great work- out today. Diary✓ Log 8 glasses of water today.Dynamic mod- ule ✓ I slept 7.5 hours, quality good.Dynamic mod- ule ✓ Track vitamin D this morning.Dynamic mod- ule ✓ B Per-Task Results The full evaluation set with per-task scores across all conditions is available as supplementary material. Table 3 shows representative examples from each task family. Table 3: Example tasks from each family with Shared Context fulfillment scores. QueryFamily Ful. Given my day so far, what is the best next step? Synth.1.00 My day was noisy and I hit the gym— evening plan? Synth.0.67 What do I have tomorrow afternoon?Temp.1.00 Will parking auto-purchase next week? Temp.0.50 Check calorie intake, then activity, then net balance. Chain1.00 Look for stress signs, then food, then rest vs. exercise. Chain0.75 C Module Coverage Table 4 lists all modules deployed during and after the pilot period, with their integration surface. Table 4: Module coverage. Pilot = generated during the three-week pilot; Post = generated afterward using the same pipeline. Ctx = publishes shared context; GUI = persistent GUI; Write = chat-invocable write-back. ModulePeriod Ctx GUI Write BoBo (behavioral sensing)Pilot✓ Health (food/activity)Pilot✓ Calendar (scheduling)Pilot✓ Parking (automation)Pilot✓ BookFactory (reading)Pilot✓ Fluent (vocabulary)Pilot✓ Sleep TrackerPost✓ Medication TrackerPost✓ Spending TrackerPost✓ Mood JournalPost✓ Hydration TrackerPost✓ Habit TrackerPost✓ Reading TrackerPost✓ DashboardPost✓ D Integration Surface The core shared-context mechanism spans 173 lines across four files: •Provider protocol (38 lines): definesToolkitDataProvider withbuildContextSummary() -> String?and relevance keywords. •Context assembly (60 lines): iterates over all registered providers, concatenates summaries inside[Personal Context] delimiters, and prepends to chat messages. Conference’17, July 2017, Washington, DC, USAZhiyuan Wang, Erzhen Hu, Mark Rucker, and Laura Barnes • Dynamic registry (59 lines): maintains an in-memory dic- tionary ofDynamicModuleDescriptorentries, each holding a view builder and provider type. •Bootstrap (16 lines): calls each module’s registration func- tion at app startup. A provider summary follows a simple tagged format: [Health Data] Today: 1030 cal, 62g protein Gym: 12 min, 65 cal burned [End Health Data]

Instant research discovery