Paper deep dive

Motif: Discovering and Automating Personal Web Workflows

Shaokang Jiang, Daye Nam

Year: 2026Venue: arXiv preprintArea: cs.HCType: PreprintEmbeddings: 88

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 7/18/2026, 12:08:15 PM

Summary

The paper introduces Motif, a system for ambient automation discovery that passively observes user browser activity to identify recurring, programmable interaction patterns. Unlike traditional programming by demonstration or 'vibe coding' which require users to initiate automation, Motif automatically detects patterns, generates program designs using LLMs, and allows users to review, refine, and deploy deterministic scripts. The system addresses the 'initiation gap' in end-user programming by surfacing hidden routines, as validated in a study with eight participants where Motif discovered more automatable patterns than users recognized on their own.

Entities (9)

Motif · system · 98%LLM · technology · 95%Ambient Automation Discovery · concept · 93%Vibe Coding · concept · 92%Initiation Gap · problem · 91%Programming by Demonstration · concept · 90%Chrome extension · software-component · 85%Gemini · service · 80%Gmail · service · 80%

Relation Signals (6)

Motif → uses → LLM

confidence 95% · Motif leverages LLM multimodal capabilities as the primary method for understanding user interactions

Motif → implements → Ambient Automation Discovery

confidence 93% · We propose Ambient Automation Discovery: a paradigm in which the system observes everyday user behavior

Motif → addresses → Initiation Gap

confidence 91% · The concept of ambient automation discovery from end-user workflows, addressing the initiation gap between programming by demonstration and end-user programming.

Motif → generates → Program

confidence 90% · generate a program to install after user confirmation

Motif → comparesfavorablyto → Vibe Coding

confidence 85% · comparing its ambient discoveries against automations users attempted to build via “vibe coding.”

Motif → runson → Chrome extension

confidence 85% · provides a Chrome extension frontend

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Recent advances in LLMs and existing work on programming by demonstration have made it possible for end users to create automations by explicitly demonstrating their behavior to LLMs. However, these approaches rely on the assumption that users know what to automate and what is capable of being automated. Additionally, automation via LLM agents is often expensive compared with programs. We introduce Motif, a system that passively observes everyday browser activity to discover recurring interaction patterns that are programmable, makes recommendations to users whenever a pattern is discovered and generate a program to install after user confirmation. Users can review, and refine the program using natural language. We evaluated Motif in a multi-day study, comparing its ambient discoveries against automations users attempted to build via ``vibe coding.'' With eight participants, Motif discovered more automatable patterns than users recognized. Most of them matched participants' routines and were useful. Follow-up surveys showed most would continue using Motif-generated programs.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

87,046 characters extracted from source content.

Expand or collapse full text

Motif: Discovering and Automating Personal Web Workflows Shaokang Jiang shj@uci.edu University of California, Irvine Irvine, California, USA Daye Nam daye.nam@uci.edu University of California, Irvine Irvine, California, USA Program Generation User Review Data Collection UI Events, Screenshots, Network Interaction Action Abstraction Enter a prompt Switch to Gemini Copy email draft Paste email draft Draft email ⋮ ⋮ Pattern Mining Gemini Email Refiner Google Calendar Quick Zoom ⋮ Program Design Trigger [Refine with Gemini] button in Gmail UI Program Automates steps required to refine an email draft using Gemini. Notification Stataus message Gemini Email Refiner Gemini Email Refiner Google Calendar Quick Zoom ⋮ Summer Time Schedule Update Usefulness score Ranking Trigger [Refine with Gemini] button in Gmail UI → Automatic Gemini Email Refiner ✅✍  ⋮ Gemini Email Refiner Figure 1: Overview of Motif. Motif supports ambient automation discovery by streaming a pipeline of passive data collection, LLM-based pattern mining, and program generation. The interface lets users review, refine, and deploy discovered automations. Abstract Recent advances in LLMs and existing work on programming by demonstration have made it possible for end users to create automa- tions by explicitly demonstrating their behavior to LLMs. However, these approaches rely on the assumption that users know what to automate and what is capable of being automated. Addition- ally, automation via LLM agents is often expensive compared with programs. We introduce Motif, a system that passively observes ev- eryday browser activity to discover recurring interaction patterns that are programmable, makes recommendations to users when- ever a pattern is discovered and generate a program to install after user confirmation. Users can review, and refine the program using natural language. We evaluated Motif in a multi-day study, compar- ing its ambient discoveries against automations users attempted to build via “vibe coding.” With eight participants, Motif discovered more automatable patterns than users recognized. Most of them matched participants’ routines and were useful. Follow-up surveys showed most would continue using Motif-generated programs. 1 Introduction The rise of LLM (Large Language Model)-powered code generation is making programming more accessible. “Vibe coding [16,32]”, promises that end users may soon be able to describe what they want in natural language, and receive working code in return, us- ing LLM-powered tools like GitHub Copilot, Cursor, Claude Code, Antigravity, and many more. These tools have demonstrated that natural language could function as a way of programming, holding particular promise for end-user programming. However, vibe coding retains a critical assumption from tradi- tional software engineering: the user must know what to build first [22,32,38,44]. For example, a business analyst must recognize that their daily routine of checking three dashboards and compiling a summary email is a candidate for automation. Or, a real estate agent must recognize that their workflow of copying property de- tails into a comparison spreadsheet from multiple listing services and pasting the batches into an email could be a script, rather than feeling like “doing my job.” Some of these tasks are so embedded in daily practice that they feel like work itself rather than overhead that could be eliminated, and even developers are not immune. Fur- thermore, for end users without programming experience, these patterns are invisible because they lack a mental model of what a program can do. Programming by demonstration (PBD) systems have sought to make automation accessible to end users [30]. However, they also require the user to initiate the process, and users must first be aware that such tools exist, then recognize a task as worth recording, and then initiate the recording. Whether through vibe coding or PBD, the bottleneck is not how to program, but knowing what to program in the first place. In this paper, we propose a new way of end-user programming, through ambient automation discovery; rather than requiring users to identify and describe what to program, we build a system that passively observes user actions, automatically discovers recur- ring multistep patterns, and surfaces them as candidate programs. The system automatically identifies what is programmable and “program-worthy” so users do not have to. By converting these pat- terns into deterministic programs, the discovered automations be- come artifacts that can be deployed, revisited, and shared with oth- ers [27]. Moreover, because all programs are executed locally, they can reduce privacy risks [21] and be more cost-efficient than LLM agents that incur inference costs with each execution [12, 29, 33]. This paradigm of including both textual requirement and behav- ior incidence produces better programs even when people know what they want. For example, when a user wants a tab manager, arXiv:2607.10531v1 [cs.HC] 12 Jul 2026 Conference’17, July 2017, Washington, DC, USAShaokang Jiang and Daye Nam the user would likely under-specify their needs, as describing their routine is a big challenge. The user would describe their routine as “I want to clean up my tabs sometimes” rather than articulating the precise six-step sequence they follow across specific sites. The users may be able to build a generic tab manager through vibe coding, but it will not be grounded in what they actually did, and they will need to spend a long time refining it as shown in previ- ous work [32,40,44]. Under this paradigm, because the discovered patterns are grounded in the user’s actual behavior, the resulting specifications are inherently personalized. To realize this vision, we present Motif, a system that passively records everyday browsing activity, uses LLMs to identify recurring automatable patterns, design user-friendly programs, and provides a Chrome extension frontend where users can review, refine, and deploy them. We evaluated Motif in a study with eight participants over an average of 5.5 (M = 5, SD = 1.2) days of real browser usage. In total, the system identified 175 patterns, averaging 22 per person. During the user study, participants went through a total of 40 patterns. Among the 40 patterns, the participants agreed that 34 (85%) corre- sponded to their daily routine. Participants tried to build programs for these 40 patterns, and 24 (60%) were successfully deployed and run as expected. Notably, participants were able to recognize only three (7.5%) of the 40 patterns as automatable on their own before using our tool, confirming the what-to-build problem. A follow-up survey revealed that most of the programs remain functional and will continue to be used. In summary, this paper makes the following contributions: • The concept of ambient automation discovery from end-user workflows, addressing the initiation gap between program- ming by demonstration and end-user programming. • A technical pipeline that combines textual requirements with user-grounded behavioral evidence mined from observed actions, enabling LLM-based code generation to produce better personalized, deterministic, and deployable programs. •An interaction design for reviewing, previewing, and itera- tively updating discovered programs. •An empirical evaluation with eight participants demonstrat- ing that the system discovers meaningful, automatable pat- terns that users would not have identified on their own. 2 Backgrounds and Related Work 2.1 End User Programming End-user programming, enabling people without professional soft- ware development training to create programs that meet their own needs, has been a long-standing goal in HCI. Over the past decades, significant progress has been made in lowering the barriers to program creation. One noteworthy example is Programming by Demonstration (PBD), where users specify their needs by showing the system what to do [15,30,31,40]. Trigger-action programming further simplifies creation by decomposing automations into if-this- then-that rules [4,7,48,50]. More recently, the LLM-powered code generation tools allow users to describe programs in natural lan- guage and receive working code in return, or a practice increasingly referred to as “vibe coding.” [16] These approaches have lowered the barrier to creating programs for end users. 2.2 Initiation Gap However, these systems share a common upstream assumption: the user must first recognize that a task is worth automating and then initiate the process. This assumption, however, is often far from trivial, due to several compounding factors that form what we call the initiation gap: 1) Task habituation: Repetitive tasks become invisible through familiarity. When a task is performed frequently enough, it can feel like “just doing work” rather than overhead that could be eliminated, even when they require significant time or effort when aggregated. Research on habituation suggests that people systematically fail to notice their own repeated patterns, and self-tracking studies consistently show that users are surprised by their own behavioral data [3,28]. 2) Mental model gap: End users without programming experience lack a mental model of what a program can do. Without this mental model, end users cannot recognize which of their workflows could be handled by a program, even when those are clear to a programmer [22,32,38]. 3) Specification gap: Even when a user can recognize what to program, articulating it precisely enough can be difficult for end users. Studies on prompt engineering show that non-programmers struggle to specify their needs in natural language [20,44,49], and even studies on PBD tools report the challenges of end users in demonstrating their needs in a generalizable manner [11,30]. 4) Automation overhead: Even when users overcome the above barriers, users, even programmers, sometimes rationally avoid ini- tiating automation: the upfront cost of setting up tools for end-user programming, or writing programs, can feel higher than simply per- forming the task one more time [9,34], even when the cumulative cost is significant [14, 19, 24]. Together, these factors constitute the what-to-automate problem. Although existing tools have solved how to create programs to automate workflows (i.e., automation) [18,29,33], and commercial product have enable end-user to generate replable script [8,41], it remains open to identify which tasks are candidates for which to program and when to actually initiate the programming process. 2.3 Usage Pattern Mining If users cannot identify what to automate, can systems do it for them? For well-defined business processes, this idea has been used in Robotic Process Automation (RPA) discovery [26]. These systems actively look for “what can be automated?” by extracting workflows from event logs. However, these approaches assume well-defined business processes with clean event schemas, typically evaluated and operated by domain experts [1,25,45], which is a fundamentally different setting from everyday end-user automation. Previous work on usage mining and clickstream analysis has studied extracting user behavior patterns from logs [10,13,42], pre- collected interaction traces within a single app [5,17]. These tech- niques have been applied to product evaluation [23], user modeling for personalization [35,46] or UX optimization [6,47], semantic interpretation of user actions [36,37], behavior prediction [43], and validation of the semantic meaning of extracted patterns [17]. However, to the best of our knowledge, prior work has not fully addressed the complexity of real-world user interactions or explic- itly designed approaches for surfacing automation opportunities Motif: Discovering and Automating Personal Web WorkflowsConference’17, July 2017, Washington, DC, USA from end users’ daily behavior. Existing approaches typically re- quire users to possess sufficient background knowledge to recognize potential automation opportunities and to initiate data collection themselves [11,17]. In contrast to those post-hoc recognition, which often depends on user initiation and a well-curated corpus, we focus on discovering recurring patterns from real end users’ daily behav- ior across multiple applications and surfacing them as candidate programs for automation. One notable system is LiveAction [3], which introduced a fully automated approach for generating task models directly from real- world web usage logs. While it relied on classical sequence mining to model user behavior step-by-step, the advent of Large Language Models (LLMs) could enable a much richer, semantic approach to pattern extraction [2]. 3Ambient Automation Discovery for End Users As discussed in the previous section, existing approaches to end- user automation assume that the user must recognize a task is automatable and initiate the process [17,30,31]. We believe that this gap prevents broader adoption of end-user programming or automation. To address this, we propose Ambient Automation Discovery: a paradigm in which the system observes everyday user behavior, identifies recurring patterns that are candidates for automation, and deploy them as programs after users review and confirmation. We derive four design goals for this paradigm from the gaps we identified in our literature review: •DG1: Discover without user initiation. The system should identify automatable patterns without requiring users to rec- ognize or declare them. •DG2: Generate personalized specifications from ob- served behavior. The system should produce program based on what users do, not generic templates. • DG3: Keep users in control. Despite passive observation, users should be able to review, understand, edit, and reject discovered automations before deployment. • DG4: Produce deterministic and cost-efficient programs. The output should be a persistent artifact that users can re- visit, modify over time, stable, and low-cost. 3.1 Usage Scenario Consider Alex, a graduate teaching assistant who regularly re- ceives emails from students requesting accommodations for missed quizzes. For each request, Alex drafts a reply in Gmail, then opens Google Gemini in a new tab, copies the draft, pastes it with a prompt asking Gemini to make the reply more professional and empathetic, waits for the result, copies the refined text, and pastes it back into Gmail. Alex does this several times a week, but has never thought of it as something a program could handle, as it simply feels like part of replying to students. After a few days of normal browsing with Motif installed, Alex opens the Motif interface and sees a ranked list of discovered pro- grams. Near the top is one titled “Student Accommodation Email Refiner.” The system has identified the recurring pattern, drafting a reply, switching to Gemini, and copy-pasting text back and forth, specifically in the context of accommodation request threads. Alex is surprised, because they never considered this a “task” that could be automated. Alex opens and reviews the program design generated by Motif: (Figure 1-Program Design): a “Refine with Gemini” button injected into Gmail’s compose toolbar that, when clicked, automatically copies the current email draft to the Gemini web interface and trans- fers the polished version back. The trigger, program behavior, and notification are all described in natural language. The description is specific to accommodation emails, but Alex realizes this would be useful for all email drafts, not just accommodation replies. Alex edits the program design to remove the accommodation-specific language and generate the program. After confirming the updated design, Alex clicks “Generate Program,” installs the script, and from that point on, a single click in Gmail around the original send button replaces the multi-step, multi-tab routine. In this scenario, Alex never had to recognize this workflow as au- tomatable, describe it from scratch, or initiate any recording. Motif surfaced the opportunity from observed behavior (DG1), generated a specification grounded in what Alex actually did (DG2), and let Alex review and refine it before deployment (DG3), producing a deterministic script that runs locally at no ongoing cost (DG4). Notably, the observation-grounded starting point, a specific pattern tied to accommodation emails, gave Alex a concrete foundation to generalize from, rather than having to articulate a vague request like “help me polish my emails” to generate and reiterate the pro- gram from scratch, nor needing to specify personal preferences, such as they prefer Gemini over ChatGPT. 4 Motif: Ambient Automation Discovery Motif transforms raw browser activity into deployable automation programs through a pipeline described in Figure 1. 4.1 Data collection To capture the web usage with enough context, Motif collects four sets of logs. First is UI interactions, such as clicks and copy-pastes. When extracting these interactions, Motif also captures surround- ing text and corresponding IDs or ARIA labels to provide sufficient context. Second is browser system interactions, such as switching or opening tabs, along with the URLs. Motif also collects OS-level interactions, including screenshots of the Chrome window with absolute mouse position, the time when user leaves the browser, refocuses the browser, etc. As a backbone of the information, Mo- tif also collects lite network interactions, including API requests and responses, with corresponding URLs, request bodies, response bodies, etc., within 1.5 seconds after the UI interactions. 4.2 Action Abstraction Due to the complexity of user interactions in each chunk, Mo- tif abstracts low-level interactions into high-level actions. Motif leverages LLM multimodal capabilities as the primary method for understanding user interactions and summarizes detailed inter- action logs into a natural-language description with metadata to prevent hallucination, such as “clicking the ’Post’ button on Ed Discussion, with metadata such as copied text content and clicked page URLs.” Motif builds an action database containing a sequence of user actions with metadata. Conference’17, July 2017, Washington, DC, USAShaokang Jiang and Daye Nam α Controller Control the monitor β Blacklist Privacy control 1 Menu Showing available programs 2 Program Tag A program has been generated for this pattern 4 A Pattern A recognized pattern 3 Reviewed Pattern A pattern that has already been reviewed A Navigator Go back to menu, go to the next, prev pattern B When the program will be invoked. User can switch to None or reset to the initial value Trigger C Description How does the program work, modifiable D Notification When user should get informed, modifiable E Tutorial Help user know how to use the generated program H Generate program Export the program to a usable tampermonkey script F Regenerate program design Generate trigger, description, and notification again. G Remove program Remove the program from the entire list Figure 2: User interface. Right top (훼−훽): controls for managing system monitoring and data collection. Right bottom (1-4): menu for reviewing generated programs. Left (A-H): an interface showing discovered programs once the user clicks on a pattern. 4.3 Pattern Mining Motif then uses LLMs to identify all programmable sequences as patterns. The LLM is instructed to focus on automatable sequences, find and merge newly added programming sequences to the previ- ous similar patterns. This approach differs from traditional sequence mining algorithms, which rely purely on sequence frequency. While traditional sequence mining algorithms could be an alternative, it is easier to filter out noise and identify more meaningful and de- terministic patterns that are programmable using LLM, leveraging programming as a reasoning mechanism. Throughout this step, Motif builds a pattern database containing all identified patterns with metadata such as each individual occurrence. 4.4 Program design Patterns obtained from the previous step are usually descriptions of sequences of user interactions, which are not directly programmable, as they are just plain English descriptions of actions users per- formed. For patterns that are likely to be frequent, we leverage LLMs as a designer to analyze the identified patterns and gener- ate user-friendly program designs without technical details while considering the technical feasibility, which include the following components: Trigger. A button or a shortcut. For some tasks, it can be automatic. Program Description. It provides detailed and non-technical de- scriptions that are necessary for program generation. Notification. It shows the situation that user is going to be notified, such as successfully completed. Given that some programs are automatically inferred without actions, it is important to notify users when any automation occurs. 4.5 Program Ranking Given that many patterns can be discovered through passive obser- vation and that users typically review only the top selections [39], Motif displays the candidate programs sorted by a score determined by the LLM based on a combination of frequency and the likelihood of future occurrence. 4.6 User Review To ensure we give users control, Motif only generates programs after the user confirms the program design. More details in sup- porting user review can be found in Section 4.8. 4.7 Program Generation Once the user confirms the program design, Motif sends a request to an LLM with the program design and context collected in the previous steps, to generate a program to automate the pattern. Among many approaches that can achieve automation, including agent workflow as in [29], we chose to implement automation programs to achieve high cost-efficiency and reusability (DG4). Programs are sustainable at scale, run faster, privacy-guarded, and are significantly less costly compared to agent workflows, which require agent API usage and incur charges per execution. Motif: Discovering and Automating Personal Web WorkflowsConference’17, July 2017, Washington, DC, USA 4.8 User Interface To ensure users still have control over the ambient discovery pro- cess, as well as what to automate, we carefully designed a front-end interface for Motif (Figure 2). System controller (훼−훽). To address privacy concerns, we give users control during the monitoring process. Users can pause moni- toring at any time and add URLs to a blacklist to prevent the system from monitoring their interactions on those pages. A status indi- cator is also displayed to show whether the system is currently monitoring or paused, together with information such as the num- ber of patterns discovered. Menu for reviewing generated patterns (1-4). Once discovered patterns are available, the menu displays an entry for each pattern, presenting a list of “Available Programs.” Each entry contains a descriptive title, a color-coded difficulty indicator on the left, and a blue “Program” tag if there is a program associated with the pattern generated earlier. Patterns that were reviewed before appear faded. Program design review and refinement (A-H). When users click on a pattern entry, a detailed interface will be displayed, showing the program design generated by the LLM. Users can modify all as- pects of the program design, as well as to request a new design from the LLM, which will generate a brand-new program design that still meets the logic. Once the user is satisfied with all descriptions, they can deploy it by clicking "Generate Program." Privacy and Security. We designed Motif as a research prototype to validate ambient discovery rather than a production-ready sys- tem. As a result, we implemented only a minimal set of privacy safeguards, including filtering passwords and hidden fields before recording, truncating network request bodies and URLs, discarding responses prior to LLM calls, requiring human validation of de- signs before program generation, and asking participants to avoid sensitive content while running the study. Each generated Tamper- monkey script also includes several LLM-determinedgrantscopes, which ensure the script only has access to the scopes for which it is intended. All data are stored locally on the user’s machine, and users can inspect, review, revise, and delete data at any time. 4.9 Implementation Motif is implemented as a Chrome browser extension with a cross- platform native host, given the limited, inefficient storage available in Chrome extensions, the limited screenshot capabilities of Chrome extensions. The frontend was built with JavaScript, HTML, and CSS. To ensure good cross-platform compatibility, the native host was built with Rust and communicates with the frontend through Chrome Native Messaging. Users can inspect, review, and manage all data stored locally. For most of the pipeline, when LLM was involved, Gemini 3 Flash Preview was used with high-thinking enabled. For action abstraction, all raw events are unified with a single Unix timestamp and grouped into several small chunks based on a maximum of 60 seconds of continuous activity, whenever a user switches away from and returns to the browser, 15 seconds of inactivity, or 8 consecutive screenshots. Then, each chunk was sent to LLM for abstraction, with the maximum context size to be under 3K tokens. Small context windows also allow future deployment to consider local LLM deployment, which usually has more limited context windows, more sustainable, and cheaper. For pattern identification, we enabled tool usage of code in- terpreter, URL scraping, and Google Search to help the LLM bet- ter understand user interactions and find more accurate patterns. Whenever 500 new actions are discovered, we send a single request to the LLM to perform pattern identification. 500 was selected after several iterations. When generating responses for the previous two steps, we used a temperature of 0.1 to enable stable and accurate abstraction. To ensure full streaming and asynchronous processing, Motif was set to automatically check for new data using the above two procedures every 5 minutes and update the relevant database ac- cordingly. All uploaded images were cleaned right after finishing those two procedures to prevent caching and protect user privacy. During program design, we sent the patterns with the detailed previous actions and raw events to the LLM so it understands the context in which this pattern should occur, and can generate a program design that is more grounded in real user behavior. We set an occurrence threshold to be 5, so that only the frequent patterns will be displayed as program candidates. We used a temperature of 0.2 to allow more creativity in the design while still maintaining relatively stable and accurate results. To optimize the user experience, we run program design and ranking every 5 minutes, right after pattern identification, ensuring users get the designs as soon as possible. In program generation, Motif generates script supported by Tam- permonkey and will be installed by the user after manually review and confirmation. Each generated program declares explicit @grant scoping for its capabilities. 4.9.1 Limitations. Due to the capabilities of Chrome browser ex- tensions, the generated programs can only run within the scope of the page. Therefore, any pattern that requires interaction with the Chrome browser itself, such as changing system settings, is not supported. Another limitation is that the complexity of the gen- erated programs is restricted. For a single program, the sequence of actions needs to be well-defined, so any pattern that requires complex UI interaction, for example, drawing a graph on a canvas, is not supported. We consider these to be less automatable and less common, so we focus on patterns that can be automated with the capability of a program that can make people’s lives easier. 5 Evaluation Rationale, Design, and Context We conducted a lab user study to evaluate the usefulness, effective- ness, and usability of Motif in real-world tasks performed by diverse users, and to determine whether the design goals can be met. Motif is designed to be highly personalized, supporting diverse usage scenarios. As such, we took a primarily exploratory and qualitative approach. Overall, we wanted to answer the following research questions: •RQ1: What are the difference between automatable patterns recognized by Motif and users? (DG1) •RQ2: How does Motif support the generation, review, and refinement of program design from discovered patterns? (DG2, DG3) Conference’17, July 2017, Washington, DC, USAShaokang Jiang and Daye Nam •RQ3: What kinds of programs can Motif generate success- fully from program design? (DG4) • RQ4: How do users experience and perceive Motif in com- parison with vibe coding? (DG3, DG4) • RQ5: How do users incorporate Motif-generated programs into their daily workflows over time? (DG4) 5.1 Study Design The study included three parts: passive data collection, a lab session, and an optional longitudinal follow-up, all conducted in compliance with our institution’s IRB requirements. Part 1: Passive Data Collection. For the first part of the study, participants met with a researcher to provide consent for the entire study, then installed the Motif browser extension during a brief meeting. The participants were asked to use their Chrome browser as usual, with Motif passively recording their browsing activity and capturing their usage patterns. All raw data was stored locally on the participant’s device. The participants were also informed about privacy concerns and how to block recordings by pausing or using a block list of URLs. Once they have enough data (12 patterns), usually after 3-7 days of usage, the participants were asked to contact the research team to schedule Part 2. They were asked to continue using the browser until the Part 2 session. The extension displayed a readiness indicator to signal the participants, but other features (1-4 in Figure 2) were not displayed to the participants. Part 2: Lab session. Participants then attended a one-hour virtual lab session with a researcher. Participants were asked to record their screen and audio during this session. a) Reflection (5 mins). First, participants reflected on their routine browser workflows and identified tasks they believed could be automated. The researcher asked guided questions to help them refine and improve their ideas into more clearly defined, potentially automatable tasks. The resulting list was later used to confirm the initiation gap and to evaluate the patterns identified by Motif. b) Vibe coding (10 mins). Then, participants spent 10 minutes attempting to build a program to automate their chosen regular routine, using an AI-powered tool of their choice to help them maximize AI capabilities rather than overcome learning barriers when learning new coding tools. We asked participants to think aloud and did not provide any help other than when they were at risk of making serious, irreversible errors. Participants were allowed to give up in the middle. If participants were actively exploring or trying one solution at the time limits, we allowed them to spend an additional two minutes. c) Motif walkthrough (5 mins). Then, participants were intro- duced to the Motif interface and reviewed the programs it had discovered. We presented a Hello World program and let partici- pants interact with it to understand the interface and the process of reviewing and generating programs. d) Motif review (30 mins). Participants were asked to review and try five programs they would like to use, and assess the accuracy, usefulness through a survey form implemented as part of the Motif interface. They were allowed to generate the program multiple times after editing the program description. We also asked the par- ticipants to think aloud. We did not provide any help during this process, except when participants are at risk of making serious, irreversible errors. We provided guidance if participants are unsure how to proceed within the interface. After each review, participants were asked to either share or remove the program and complete a questionnaire about the identified pattern and the program’s useful- ness. Throughout the process, we encouraged participants to refine the program description and regenerate the design if the initial attempt is unsuccessful. To evaluate the usefulness and accuracy of the program generation process, participants were given sufficient time to explore and understand each program, while we kept a soft cutoff of five minutes per program to ensure balanced time allocation across all programs. e) post study survey and semi-structured interview (12 mins). Fi- nally, we had a post-study survey with parallel 5-point Likert scale items for both vibe coding and the Motif, covering dimensions such as ease of use, output accuracy, confidence, and usefulness. We ended the session with a semi-structured interview to further un- derstand their thoughts on automation blindness, the specification process, trust, and privacy. Participants kept the Tampermonkey scripts if they wanted, and all other programs and the Motif exten- sion were fully removed. Part 3: Longitudinal Follow-up. Before the end of Part 2, we asked participants if they are interested in participating in the optional longitudinal study. If they agreed, they received a follow- up survey on continued script usage, real-world usefulness, and issues encountered. 5.2 Recruitment We recruited ten participants 1 , through social media and university forums. All participants were at least 18 years old, used Google Chrome daily, and were proficient in English. Prior programming and AI tool experience were collected during recruitment, and we balanced participants by programming experience level, slightly favoring those with no experience. Participants were compensated with $45 gift cards after completing Parts 1 and 2, and an additional $5 for the optional Part 3 survey. If parts 2 ran longer than an hour, we paid them an additional $12.5 per 30 minutes. All participants were all students (4 undergraduates, 5 gradu- ates). P6 and P8 used Windows, while the rest used macOS. The remaining participants had a good mix of programming and AI tool experience. Detailed demographics and the programs generated by each participant can be found in Table 1. 5.3 Threats to Validity The vibe coding task was limited to 10 minutes, which may not have given participants sufficient time to fully iterate on their prompts and produce a polished result. However, our focus was on the specification process, rather than the final quality of the vibe-coded artifact. Capturing a complete vibe coding session would require substantially more time than is feasible in a controlled lab setting. Motif’s data collection period spanned approximately 4-8 days of active browsing. While this was sufficient to surface frequently re- curring patterns, it may not capture important but infrequent work- flows such as monthly reports or seasonal tasks, and likely over- represents short-term daily routines. This constraint was driven by practical considerations for study scheduling. We note that the 1 Two participants dropped out midway in data collection due to schedule changes. Motif: Discovering and Automating Personal Web WorkflowsConference’17, July 2017, Washington, DC, USA Table 1: All participants’ demographics and programs generated. IDProgramAI toolMajorDays# Act.# Pat.Worked Programs P1IntermediateAdvancedSWE5265624googleAiOverviewExpander, youtubeAutoFullscreen P3None IntermediatePharm. Sci.5245013forwardGmailEnrollmentProof,automateTubiPlayback,automateY- ouTubeEngagement P4 IntermediateAdvancedSWE5566630overleafAutoRecompile, acmDlAutoOpenPdf, arxivAutoOpenPdf P5BeginnerBeginner Bus.Econ., Psychology 6258321 facebookMarketplaceAutoInsights,facebookMarketplaceAutoRenewer, americaLearnsAutoTimesheet, whentoworkKeyboardNav P6NoneIntermediateBio. Sci.6329319googleSearchTabNavigator, stembleQuizNavigator, googleSearchAIEx- pander, wikipediaBirdRedirect P7 IntermediateAdvancedBiostatistics4159521chatGptResponseInteraction, chatGptProjectNavigator P8NoneBeginnerLaw5111712googleNewsNavigateCategory, zotGptAutoSendMessage P9ExpertExpertSWE8339735arxivAutoOpenPdf, googleScholarAutoSortYear, githubCopyCloneUrl, googleSearchExpandAIOverview limited collection period likely underestimates Motif’s potential: with longer logging and continuous use, the system would accu- mulate more diverse patterns and produce a richer set of program recommendations. We allowed participants to use the vibe coding tool of their choice, introducing variability across participants. This was a de- liberate decision: the learning curve for these tools is steep, and given the 10-minute time constraint, requiring all participants to use an unfamiliar tool would conflate tool learnability with the specification process we aimed to study. Letting participants use tools they were already comfortable with provided a more realistic picture of how they express their needs. During the Motif review phase, participants selected at least five programs they were most interested in from the full list of discovered programs, rather than reviewing all of them exhaustively. While this is based on the practical considerations to keep the lab session length to be reasonable, this reflects realistic usage behavior. In practice, users engage selectively with system recommendations rather than reviewing every result, much as they do with search engine results in information retrieval. Even though we designed some privacy protection as descibed at section 4.8, we acknowledge that the system still has residual privacy and security risks, including limited PII redaction, the po- tential for behavioral profiling or workplace surveillance misuse, prompt injection, malicious or buggy script generation despite scoped prompts, and risks from local compromise. Future work could further explore security-related warning interface design and PII data reduction under this scenario. Meanwhile, any future de- ployment of Motif would require additional operational safeguards, such as organizational policies, audit logging, and tighter grant sandboxing. The security scope of the generated programs is determined by the LLM and manually reviewed by humans. Although our prompt is designed to avoid generating programs that perform actions be- yond program design, it remains possible that a generated program may perform harmful operations. In future work, a non-LLM-based deterministic mechanism could be implemented to determine and validate the program’s security scope before deployment. To ensure Motif is widely available with all participants, we used a cloud-based LLM (Gemini) for the study. However, in practice, most LLM calls made by our program, such as action abstraction, are small, typically invoking fewer than 3K tokens per request. Based on our evaluation using a local LLM (Qwen3.6-35B-A3B), we can achieve performance comparable to that of cloud-based LLMs. This makes local deployment a viable solution for addressing some privacy concerns also. 6 Study Findings We analyzed both quantitative and qualitative data to answer the research questions. For qualitative data, including the study tran- scripts and the programs generated, one author performed initial open coding on the transcripts, and the codes were refined through discussion with another author to derive key themes. 6.1 RQ1: Pattern Identification Figure 3 summarizes the number of patterns that participants could come up with during the self-reflection session, and that the Motif passively identified from the web usage logs. At the beginning of the Part 2 lab session, participants were asked to self-reflect on their routine browser workflows and identify tasks they believed could be automated. All participants could come up with at least one routine to automate with only P4 explicitly saying no idea initially. However, except for P7, every participant origi- nally only came up with a very broad idea, such as “browse a series of pages” (P3), that cannot be automated with any tools available. The researcher then helped participants develop ideas by asking their routine in more details, providing in-context examples, or asking guiding questions, such as "Which pages do you visit?" and "What actions do you perform frequently?" The researcher also provided in-scenario examples to help clarify. Finally, three partic- ipants developed two ideas each and five participants developed one idea each. Motif, on average, could identify 22 usage patterns that occurred more than 5 times during Part 1 recording, that are automatable. Of the 5 patterns each participant reviewed in detail, participants agreed with an average of 4.75 pattern matches or partial matches Conference’17, July 2017, Washington, DC, USAShaokang Jiang and Daye Nam P1P3P4P5P6P7P8P9 CorrectUsefulWorking (program) 35 12 21 19 21 30 13 24 1 2 11 2 0 2 1 Self ReflectionMotif (Ambient Discovery) Identified pattern: 40 32 24 16 8 0 5 4 3 2 1 0 Out of 5 patterns reviewed: # Pattern# Reviewed Figure 3: Motif identified more automatable patterns (avg 22) than users’ self-reflection (avg 1.3). Participants agreed that most of the patterns identified by Motif matched their routine (avg 4.75 out of 5) and are useful for automation (avg 4.6 out of 5). their daily routine, and found an average of 4.6 patterns to be potentially useful if the program works perfectly. This confirms the initiation gap, especially the task habituation and the mental model gap, and shows that the users often do not realize what can be automatable. 6.2 RQ2: Pattern Usefulness Pattern Correctness. We evaluated the correctness of the iden- tified patterns based on the participants’ responses to the ques- tionnaire when they shared or removed the reviewed patterns. All participants agreed that at least four patterns they reviewed matched their actual routine and that creating a program for these would be useful. Five participants agreed that all five patterns they tried matched their routine. Two participants identified one pattern that matched their routine but deemed it not useful for automation, even if the program worked perfectly. Result as shown in fig. 3. Most participants appreciated the patterns identified through self-reflection that were not recognizable to them. P6 suggests, “I think it’s surprising how I could track what you normally browse and come up with shortcuts for those. I think that’s pretty cool”. Similarly, P4 also suggests “I found that the pattern identification in motif is very useful, because it’s very subtle, so I didn’t realize it, but if I can have a program to help me do that, I feel it’s very useful.” Longer-term usage. During the interview, six out of eight partici- pants indicated that they want to continue using Motif, and they are interested in seeing more patterns that the system can discover. Some participants (P4 and P6) indicated that a longer observation would be more useful to discover more patterns. “It will be interest- ing to have a leaderboard [so] I can know what I’ve done last month” (p4) However, participants also pointed out privacy concerns for the longer-term usage; “then there’s other things, like security-type concerns” (p8). Gap between reflected patterns and Motif detected patterns. Interestingly, most of the patterns that participants reflected on during the reflection phase were not detected by Motif, and most of the patterns that Motif detected were not reflected by participants. Diving deeper, the differences mostly stem from the scope of the patterns. Our Motif focuses more on automatable patterns that are well-scoped, predictable, and shorter in sequence, while participants reflected on patterns that are larger in scope, more open-ended, and sometimes all-in-one solutions. For example, P7 reflected on a tool that automatically generates a job summary, performs translation based on the job description, and evaluates her own fitness for the position. In contrast, Motif detected several job refinement-related patterns. P7 suggested “[Motif] break it down into very, small steps, small patterns”. Intention≠Execution. Users’ intended requirements were some- times much more abstract than the actions they actually performed. For example, P8 initially identified a pattern regarding "ease of access" across different result categories on Google News. How- ever, while engaging with the vibe coding prompt, she reflected and narrowed this abstract idea into a conceptual need: an auto- updating news summary. Even though Motif successfully generated a program based on her original recorded actions, P8 deemed the result unhelpful because it failed to reach this broader conceptual idea, noting: “It saves time jumping between pages, but the top bar is present with the floating bar.” Hesitate of Modification. Motif was designed to provide precise automation while leaving room for users to edit; however, par- ticipants generally preferred regenerating programs rather than modifying textual descriptions to match their needs. For example, P7 tested a button-invoked program designed to transform Gemini into a "career station" by appending tailored prompts to copied job descriptions. During the trial, she noted the program’s narrow scope, stating, “In this description, it seems like it just tells Gemini to be the career coach, not that if I switch to any AI agent, it will automatically tell them to be my career coach for me.” Although P7 could have modified the trigger condition to generate a more generalized program that matched her broader intention, she did not do so. Overall, four participants modified the descriptions of just eight programs in total. Early Feasibility Judgement. We found that some users prefer to assess the automation’s feasibility based on the description. By assuming that the functionality would not work without trying or modifying any program description, P8 rejected a pattern that could automatically highlight text on some websites, presuming that “[The program] need to understand the text in order for the highlights to work.” Sorting by Usefulness Score is not enough. Despite presenting patterns ranked by usefulness scores from high to low, partici- pants did not simply select the top-ranked programs. Instead, they browsed the entire list and selected programs based on personal interest, with final selections likely distributed normally across the entire list. P4 even asked, ‘Do you have some sorting for the avail- able programs?” Upon knowing the list was ranked by usefulness, P4 expressed surprise at patterns Motif identified as most useful but had not recognized in self-reflection. This suggests that while ranking provides structure, participants rely on personal judgment and curiosity rather than algorithmic scores when selecting which automations to explore. It may also suggest a frequency and useful- ness combined sorting is not enough for recommendation the best patterns to users. Motif: Discovering and Automating Personal Web WorkflowsConference’17, July 2017, Washington, DC, USA 6.3 RQ3: Program Usefulness Out of five programs each participant reviewed, three worked as expected on average. Among the 40 programs participants reviewed, the LLM labeled 1 as high difficulty, 22 as medium difficulty, and 17 as low difficulty. Low difficulty programs are more likely to work as expected, with 82% working as expected, compared with only 45% for medium difficulty programs. The only high difficulty program did not work as expected. 6.3.1 Programs that function as expected. From the entire work- flow, locating and designing a correct trigger is critical for the overall success of the generated program. Among the 40 programs generated by participants, 30 were designed to use a button as the trigger, seven were designed to run automatically when reach- ing a certain page, and five were designed to be triggered by key- board shortcuts. Among these, on page load and shortcut triggers showed higher success rates: 86% of on page load-triggered pro- grams worked as expected, and 67% of shortcut-triggered programs worked as expected, while only 54% of button-triggered programs worked as expected. This may be due to the fact that most existing Tampermonkey scripts used for LLM training were designed for full automation rather than user interaction. Since the built programs were personalized for each user based on their browsing history, the accepted programs were highly di- verse by type. Some accepted programs were for routine work that people needed to perform frequently, like automatically filling out the timesheet with a template and submitting that (americaLearn- sAutoTimesheet). Or highly repetitive work that was personalized to each user, such as automatically opening a PDF page whenever they reach the ACM Digital Library (acmDlAutoOpenPdf ), or au- tomatically sorting by years whenever they open Google Scholar (googleScholarAutoSortYear). Some interaction with a video stream- ing website, such as automatically selecting the next episodes and automatically playing the video (automateTubiPlayback), is also highly accepted. A last but not least type is a shortcut program that helps people jump to the right place faster, such as automatically navigating between different types of Google search results with a shortcut (googleSearchTabNavigator). 6.3.2 Programs that did not work. Only one participant removed a program because it did not work. This program was a script designed to join Zoom meetings without user confirmation. It failed because the task requires interacting with the Chrome browser itself by interacting with a browser pop-up window, which is outside the scope of a Tampermonkey script, since it only runs in the isolated context of a specific webpage. Among all programs, we did not observe any program that will perform a non-irreversible action without user confirmation, such as sending emails automatically without the user clicking send. We believe the program design prompts are in effect to prevent those security vulnerable programs from being generated. Some websites have complex or dynamically loaded DOM struc- tures that the original website intentionally designed to prevent au- tomation. For example, P8 evaluated a pattern called calendlySched- uleEvent, which is designed to automatically fill out the scheduling form on Calendly and submit it, but it did not work because the form is dynamically loaded, and the program cannot locate the right element to fill out and submit. Sometimes, the pattern users selected are too complex for the program to work; for example, P7 selected geminiWorkflowAutoma- tion, which requires switching between Gemini models (Fast/Think- ing) and streamlines the process of updating previous prompts by feedback and automatically submit the new version. P7 tried to regenerate the design several times, did not attempt to correct the program description, and could not make the program work as expected. We think this is because the program is too complex, making it hard for the LLM to capture the real user intent when designing and generating the program. 6.3.3 Perceived Usefulness. Overall, five participants (P3, P4, P5, P6, P9) had positive experiences and generated three to four programs that are very useful. P3 suggested that, “With Motif, I thought that it was pretty easy (to generate the program) ... I was surprised that they would think some of them could be automated, like, [describing the forwardGmailEnrollmentProof pattern].” Three participants (P1, P7, P8) were able to generate only two not very useful programs and had negative experience with the entire process. We think this is due to participants’ usage patterns. Two participants had concerns about privacy and did not use Motif for all of their work; they created multiple Chrome profiles and only used the one with Motif for some specific tasks to prevent privacy issues. One of these two was also concerned about privacy issues of trying some patterns during the study in front of us and chose to try rather simple programs with limited utility, such as automatically sending a chatbot message. Even for people who stand strongly for privacy concerns, they are still willing to share the generated program with others, which confirms our design goal of using programs to maintain privacy. Another participant did not use English in their daily workflow, given the majority focusing of Gemini and our workflow is on English content, the entire workflow may be less accurate when the user is not using English, which may cause the entire Motif workflow doesn’t run as expected. Potential time saving and personalization. Among the gener- ated programs, participants appreciate the time it could potentially save, with P3 suggesting, “It took me a while to draft the forwarding email to my employers, so it’s nice to have it automated as I need to do this every quarter for my job.” Sometimes, the program can personalize and remember the last step of the user and continue from there, such as P3 suggests, “This is useful because it remembers what episode I am at in the series.” Too small action may not be beneficial. If the action the program performs was too small, the participant complains about the actual effort saved by it, such as P8 suggested “[the program] duplicates the send button. They are in the same location.” Beginners, no experience, and experts benefit more than intermediate experience people. For people with less program- ming experience, they appreciate the ease of use, and specifically thanks to the regeneration feature, which allows them to easily modify the program description without going through the entire description. P3 has no experience in programming and suggested “I also like the regenerate design, how I can just press it, and then it Conference’17, July 2017, Washington, DC, USAShaokang Jiang and Daye Nam Easy to use Got wanted program Understand code Verify correctness Fix mistakes Useful for daily browsing Would use again Enjoyed using 1 2 3 4 5 Score (1-5, higher is better) Vibe coding Motif Figure 4: Post-study survey results. Most participants re- ported higher satisfaction with Motif compared to vibe cod- ing across all dimensions. would try to fix it, so I don’t really have to do anything too difficult.” Beginners and no experience people also appreciated the tutorial with program design, which tells them how to use the program, as P5 suggests “it easy for people to use and understand, because I don’t really have a background in code, but, like, having that tutorial thing and, like, having you explain how it works, I think it was really helpful.” Expert care more aspects about the program being generated than other people, such as the performance, and appreciate the user under control design and easy to modify, test and try of Mo- tif. P9 is an expert programmer who has the same pattern from self-reflection and Motif, he recognized that “Motif’s program is faster than [the vibe coding solution] I installed. It’s very fast.” He also recognizes the ease of modification, saying "Directly edit the program by natural language is easy to use... [Motif ] is easy to correct the program." However, people with intermediate experience are more likely to be critical of the program generation process and wish to stick with their original solution. P1 suggested “I think programming in natural language. . . but usually I would ask another. . . AI to, like, generate the exact prompt of the function I want". They also wish to have natural language to modify the program description, as P7 suggested “If you can add an AI toolset, help me translate my natural language into a more specific prompt to the program description, it will be easier.” They also complain about the waiting time of program generation, as P1 suggested, “Sometimes the waiting time is different from each trial, so it kind of makes me confused, like, is it working or not?” 6.4 RQ4: Comparison with Vibe Coding The overall subjective user feedback is shown in fig. 4. Overall, participants reported high satisfaction with Motif’s usability and effectiveness in generating automation programs for their routine web tasks, compared with vibe coding. Motif received higher ratings than vibe coding across all dimensions, with larger differences in understanding the code, verifying correctness, and enjoyment. This aligns with interview; for example, P4 said, “I think the title and the description are good, because it’s very specific, and it just describes a very, very specific behavior, not very general, so I can understand.” People generally enjoyed using Motif, which was further confirmed in the subsequent interview where six out of eight participants wished to continue using Motif, and the others expressed interest in using it if the privacy concerns or program accuracy could be improved. Vibe coding vs. Motif. Comparing vibe coding and Motif in general, most users with no experience or beginners found vibe coding difficult and prefer to use Motif. P6 was able to get a working program and suggested that “I would rather not do the ChatGPT one. Cause I like how this one just. . . Generate stuff for you, and you don’t have to manually type it in.” Fortunately, Motif also detected the same pattern for her. When comparing the entire process of both approaches in terms of trustworthiness, she suggested “probably [Motif ], because ChatGPT made, like, 3 mistakes.” Surprisingly, only two out of eight people were able to use the vibe coding tool they selected to build the workflow successfully. Participants used a variety of vibe coding tools, like ChatGPT, codex, which recommended a wide range of solutions, such as Playwright (P4), the Claude Code in-app widget (P1, P7), a third-party no-code platform (P3), Chrome extension (P9) and Tampermonkey (P6). P6 and P9 were able to successfully set up the environment following LLM guidance and finish all desired functionality, most likely due to the correct selection of the tool should be used (e.g., Tamper- monkey is more suitable for web automation in user’s personal browser). Four participants had setup issues that prevented them from successfully using the vibe coding tool, with two of them (P1, P7) experiencing issues with the Claude Code widget, likely due to incorrect or incomplete Claude Code setup. The other two participants (P3, P5) had issues with setting up the recommended platform or toolset on their local machine, which may be due to the complexity of the setup process or lack understanding of computer architecture. Although P6 successfully setup the environment, she still had to debug the setup 2 times and ask for guidance, which sug- gests that the setup process is still not smooth and straightforward in the vibe coding environment. Unlike providing a fixed pattern description in Motif, the vibe coding process is more iterative and open-ended, offering a sig- nificant benefit. Participants (P1, P4, P7) utilize the iterative chat feature to refine and formulate more fine-grained ideas after seeing the system’s initial responses, establishing a collaborative dynamic between themselves and the vibe coding platform. Users also ap- preciate the high flexibility of vibe coding, which allows them to express their needs in a more open-ended manner and receive more comprehensive solutions. For example, P7 suggested, “If the vibe coding one did work, I might prefer that one, because that looks like a really big and comprehensive app that I can use” and P4 suggest "[vibe coding tool] is more straightforward for me to interact with". Besides the setup challenging, when comparing the core pro- grams successfully generated by both approaches, the vibe coding solutions are significantly shorter in terms of lines of code, but they are also less informative and provide less feedback to the end-user. For example, the vibe coding solution for P6 is a 3-line Tampermonkey script that automatically redirects from one page to another without any user confirmation or notification. In contrast, the program generated by Motif is much longer and provides an interactive entry point for the user to initiate the transition. The Motif: Discovering and Automating Personal Web WorkflowsConference’17, July 2017, Washington, DC, USA 010 Plan to Keep Using Saved Time Worked as Expected Strongly disagree Disagree Neutral Agree Strongly agree Figure 5: Followup survey results. extreme automation by vibe coding could pose a malicious program by essentially preventing the user from accessing the original page. 6.5 RQ5: Long-term usefulness We sent a follow-up survey 2–4 days after participants completed the study session, asking about their continued use of the worked programs generated by the Motif. Six participants (P1, P3, P4, P5, P8, P9) responded to the follow-up survey. Among 18 programs they had kept and used, they regularly used 30% of programs, used 40% of programs a few times, and had not yet had the opportunity to use the remaining programs. Regarding reasons for non-use, two participants (P3, P8) reported four programs to be less useful than expected, and P5 reported not having had the opportunity to use two programs. We asked participants, using a Likert scale, whether the pro- gram continued to work as expected, saved time, and whether they would continue using it. The results are shown in fig. 5. Overall, most of the programs that participants kept continued to work as expected, saved time, and participants were willing to continue using them. By analyzing the programs that participants were less likely to continue using and that were deemed not to save time, such as zotGptAutoSendMessage, we observed that participants were less likely to continue using programs that performed only minor actions or interfered with the original functionality. An- other pattern among programs that participants were less likely to continue using is those that have too many actions at once, such as automateYouTubeEngagement, which automatically likes, sub- scribes, and shares content in one click. These behaviors can be overwhelming for users and may even cause technical issues. One of the most far-reaching influences of this experiment is that it might reshape how people think about automation and patterns. In the follow-up survey, all participants agreed that using Motif changed how they think about their browser workflow, with four participants strongly agreeing. Two participants also came up with new tasks that they could automate after using Motif. However, upon reviewing the new patterns that participants self-reflected on, only one of them was a valid, automatable pattern, suggesting that people’s thinking about their workflow is still limited by the mental model gap and task habituation, even after using Motif. 7 Discussion and Implications Mixed method in automation. One drawback of the program generation is its lack of compatibility with complex DOM websites, such as Google Docs, games, or canvas drawing pages. P1 tried to write a program to automatically download a Google Doc as a PDF, but it did not work because the program could not locate the element. One possible solution is to combine the program with a computer-use agent to recognize and perform clicks in real time, which can significantly increase the program’s compatibility while still maintaining its benefits. Higher level of abstraction and fine-grained program gen- eration. From the gap between the reflected patterns and those detected by Motif, we can see that people tend to think in terms of high-level, open-ended, all-in-one solutions, while Motif focused on automatable patterns that are well-scoped, predictable, and shorter in sequence. Although users could modify the program design in our interface, they often seemed unwilling to do so, they preferred to click “regenerate” The reasons are twofold: first, users with lim- ited programming experience often found it difficult to understand the program description; second, users with intermediate experi- ence perceived the cognitive load of modifying it as too high. This highlights the need for a higher level of abstraction in program design, which could better capture user intent and generate pro- grams more closely aligned with their needs. Given that the goal of the current work was to test the feasibility and benefits of the new paradigm of ambient automation discovery, we plan to keep this as future work and explore it further. More Intuitive Program Refinement. P1, P7, and P8 expressed a desire for a system that allows them to modify program descriptions using natural-language prompts, or a place where they can indicate what went wrong. As P8 suggested, “I would say maybe just a place to input what’s going wrong, because right now you only have whether this program works or it doesn’t.” This could be a very intuitive feature for users to quickly improve the generated programs. 8 Conclusion In this paper, we presented Motif, a system that automatically iden- tifies routine web usage patterns and generates personalized au- tomation programs. Through a user study with eight participants, we found that Motif can identify many more automatable patterns than users can self-reflect on, and that most of the identified pat- terns were useful for automation. We also found that most gener- ated programs worked as expected and were perceived as useful by participants in long term. Our findings suggest that Motif can effectively recognize and automate routine web usage patterns, thereby significantly enhancing user productivity and experience. Future work should explore ways to further abstract user intent, improve program generation, and streamline the program adoption workflow. References [1]Simone Agostinelli, Marco Lupia, Andrea Marrella, and Massimo Mecella. 2022. Reactive synthesis of software robots in RPA from user interface logs. Computers in Industry 142 (2022), 103721. [2] Siraaj Akhtar, Saad Khan, and Simon Parkinson. 2025. LLM-based event log analysis techniques: A survey. arXiv preprint arXiv:2502.00677 (2025). [3]Saleema Amershi, Jalal Mahmud, Jeffrey Nichols, Tessa Lau, and German Attana- sio Ruiz. 2013. LiveAction: Automating Web Task Model Generation. ACM Trans. Interact. Intell. Syst. 3, 3, Article 14 (Oct. 2013), 23 pages. doi:10.1145/2533670. 2533672 [4]Fraser Anderson, Tovi Grossman, and George Fitzmaurice. 2017. Trigger-Action- Circuits: Leveraging Generative Design to Enable Novices to Design and Build Circuitry. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association Conference’17, July 2017, Washington, DC, USAShaokang Jiang and Daye Nam for Computing Machinery, New York, NY, USA, 331–342. doi:10.1145/3126594. 3126637 [5]Deniz Arsan, Ali Zaidi, Aravind Sagar, and Ranjitha Kumar. 2021. App-based task shortcuts for virtual assistants. In The 34th Annual ACM Symposium on User Interface Software and Technology. 1089–1099. [6]Richard Atterer, Monika Wnuk, and Albrecht Schmidt. 2006. Knowing the user’s every move: user activity tracking for website usability evaluation and implicit interaction. In Proceedings of the 15th International Conference on World Wide Web (Edinburgh, Scotland) (W ’06). Association for Computing Machinery, New York, NY, USA, 203–212. doi:10.1145/1135777.1135811 [7]Timothy J. Aveni, Hila Mor, Armando Fox, and Björn Hartmann. 2025. Generative Trigger-Action Programming with Ply. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST ’25). Association for Computing Machinery, New York, NY, USA, Article 33, 17 pages. doi:10.1145/ 3746059.3747638 [8]Axiom AI Limited. 2025. Axiom.ai: Browser Automation for Humans and AI. https://axiom.ai/. Accessed: 2026-06-29. [9] Alan F Blackwell. 2002. First steps in programming: A rationale for attention in- vestment models. In Proceedings IEEE 2002 Symposia on Human Centric Computing Languages and Environments. IEEE, 2–10. [10]Antonio Bosco, Adriano Augusto, Marlon Dumas, Marcello La Rosa, and Gian- carlo Fortino. 2019. Discovering automatable routines from user interaction logs. In International conference on business process management. Springer, 144–162. [11]Sarah E. Chasins, Maria Mueller, and Rastislav Bodik. 2018. Rousillon: Scrap- ing Distributed Hierarchical Web Data. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18). Association for Computing Machinery, New York, NY, USA, 963–975. doi:10.1145/3242587.3242661 [12]Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176 (2023). [13] Daehyoun Choi, Hind R’bigui, and Chiwoon Cho. 2021. Candidate digital tasks selection methodology for automation with robotic process automation. Sustain- ability 13, 16 (2021), 8980. [14] Vincent J Hellendoorn, Sebastian Proksch, Harald C Gall, and Alberto Bacchelli. 2019. When code completion fails: A case study on real-world completions. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 960–970. [15] Jaylin Herskovitz, Andi Xu, Rahaf Alharbi, and Anhong Guo. 2024. ProgramAlly: Creating Custom Visual Access Programs via Multi-Modal End-User Program- ming. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. 1–15. [16]Marko Horvat. 2025. What is Vibe coding and when should you use it (or not)? Authorea Preprints (2025). [17]Forrest Huang, Gang Li, Tao Li, and Yang Li. 2024. Automatic macro mining from interaction traces at scale. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–16. [18]Faria Huq, Zora Zhiruo Wang, Zhanqiu Guo, Venu Arvind Arangarajan, Tianyue Ou, Frank Xu, Shuyan Zhou, Graham Neubig, and Jeffrey P. Bigham. 2026. Mod- eling Distinct Human Interaction in Web Agents. arXiv:2602.17588 [cs.CL] https://arxiv.org/abs/2602.17588 [19]Shaokang Jiang and Michael Coblenz. 2024. An analysis of the costs and benefits of autocomplete in ides. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1284–1306. [20]Shaokang Jiang and Daye Nam. 2025. Beyond the Prompt: An Empirical Study of Cursor Rules. arXiv preprint arXiv:2512.18925 (2025). [21]Martin Kleppmann, Adam Wiggins, Peter Van Hardenberg, and Mark Mc- Granaghan. 2019. Local-first software: you own your data, in spite of the cloud. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software. 154–178. [22]Amy J Ko, Brad A Myers, and Htet Htet Aung. 2004. Six learning barriers in end- user programming systems. In 2004 IEEE Symposium on Visual Languages-Human Centric Computing. IEEE, 199–206. [23]Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M Henne. 2009. Controlled experiments on the web: survey and practical guide. Data mining and knowledge discovery 18, 1 (2009), 140–181. [24]Tessa Lau. 2009. Why Programming by Demonstration Systems Fail: Lessons Learned for Usable AI. AI Magazine 30, 4 (2009), 65–67. [25] Van-Hoang Le and Hongyu Zhang. 2023. Log parsing with prompt-based few-shot learning. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2438–2449. [26]Volodymyr Leno, Artem Polyvyanyy, Marlon Dumas, Marcello La Rosa, and Fabrizio Maria Maggi. 2021. Robotic Process Mining: Vision and Challenges: V. Leno et al. Business & Information Systems Engineering 63, 3 (2021), 301–314. [27]Gilly Leshed, Eben M. Haber, Tara Matthews, and Tessa Lau. 2008. CoScripter: automating & sharing how-to knowledge in the enterprise. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08). Association for Computing Machinery, New York, NY, USA, 1719–1728. doi:10.1145/1357054.1357323 [28]Ian Li, Anind Dey, and Jodi Forlizzi. 2010. A stage-based model of personal informatics systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 557–566. doi:10.1145/1753326.1753409 [29]Jiawen Li, Zheng Ning, Yuan Tian, and Toby Jia-jun Li. 2025. ALLOY: Gen- erating Reusable Agent Workflows from User Demonstration. arXiv preprint arXiv:2510.10049 (2025). [30] Toby Jia-Jun Li, Amos Azaria, and Brad A Myers. 2017. SUGILITE: creating multimodal smartphone automation by demonstration. In Proceedings of the 2017 CHI conference on human factors in computing systems. 6038–6049. [31]Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M Mitchell, and Brad A Myers. 2019. Pumice: A multi-modal agent that learns concepts and conditionals from natural language and demonstrations. In Proceedings of the 32nd annual ACM symposium on user interface software and technology. 577–589. [32]Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Benjamin Zorn, Jack Williams, Neil Toronto, and Andrew D. Gordon. 2023. “What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code- Generating Large Language Models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). ACM, 1–31. doi:10.1145/3544548. 3580817 [33] Yimeng Liu, Misha Sra, Jeevana Priya Inala, and Chenglong Wang. 2025. ReUseIt: Synthesizing Reusable AI Agent Workflows for Web Automation. arXiv preprint arXiv:2510.14308 (2025). [34]Wendy E Mackay. 1991. Triggers and barriers to customizing software. In Proceed- ings of the SIGCHI conference on Human factors in computing systems. 153–160. [35]Bamshad Mobasher, Robert Cooley, and Jaideep Srivastava. 2000. Automatic personalization based on Web usage mining. Commun. ACM 43, 8 (Aug. 2000), 142–151. doi:10.1145/345124.345169 [36] Nuria Oliver, Greg Smith, Chintan Thakkar, and Arun C. Surendran. 2006. SWISH: semantic analysis of window titles and switching history. In Proceedings of the 11th International Conference on Intelligent User Interfaces (Sydney, Australia) (IUI ’06). Association for Computing Machinery, New York, NY, USA, 194–201. doi:10.1145/1111449.1111492 [37] Seokhyeon Park, Yumin Song, Soohyun Lee, Jaeyoung Kim, and Jinwook Seo. 2025. Leveraging Multimodal LLM for Inspirational User Interface Search. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, Article 579, 22 pages. doi:10.1145/3706598.3714213 [38]Roy D Pea. 1987. User centered system design: new perspectives on human- computer interaction. Journal educational computing research 3 (1987), 129–134. [39]K. Pernice, K. Whitenton, J. Nielsen, and Nielsen Norman Group. 2014. How People Read on the Web: The Eyetracking Evidence. Nielsen Norman Group. https://books.google.com/books?id=nQ2-oQEACAAJ [40]Kevin Pu, Jim Yang, Angel Yuan, Minyi Ma, Rui Dong, Xinyu Wang, Yan Chen, and Tovi Grossman. 2023. DiLogics: Creating Web Automation Programs with Diverse Logics. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST ’23). Association for Computing Machinery, New York, NY, USA, Article 74, 15 pages. doi:10.1145/ 3586183.3606822 [41] Road to Ramen LLC. 2025. Browserflow: Web Scraping & Web Automation. https://browserflow.app/. Accessed: 2026-06-29. [42]Jianqiang Shen, Erin Fitzhenry, and Thomas G. Dietterich. 2009. Discovering frequent work procedures from resource connections. In Proceedings of the 14th International Conference on Intelligent User Interfaces (Sanibel Island, Florida, USA) (IUI ’09). Association for Computing Machinery, New York, NY, USA, 277–286. doi:10.1145/1502650.1502690 [43] Jianqiang Shen, Lida Li, Thomas G. Dietterich, and Jonathan L. Herlocker. 2006. A hybrid learning system for recognizing user tasks from desktop activities and email messages. In Proceedings of the 11th International Conference on Intelligent User Interfaces (Sydney, Australia) (IUI ’06). Association for Computing Machinery, New York, NY, USA, 86–92. doi:10.1145/1111449.1111473 [44]Pradyumna Shome, Sashreek Krishnan, and Sauvik Das. 2025. Why Johnny Can’t Use Agents: Industry Aspirations vs. User Realities with AI Agent Software. arXiv preprint arXiv:2509.14528 (2025). [45]Rehan Syed, Suriadi Suriadi, Michael Adams, Wasana Bandara, Sander J Leemans, Chun Ouyang, Arthur HM Ter Hofstede, Inge Van De Weerd, Moe Thandar Wynn, and Hajo A Reijers. 2020. Robotic process automation: contemporary themes and challenges. Computers in industry 115 (2020), 103162. [46] A Aylin Tokuç and Tamer Dag. 2025. Predicting user purchases from clickstream data: A comparative analysis of clickstream data representations and machine learning models. IEEE Access 13 (2025), 43796–43817. [47] Kelsey Turbeville, Jennarong Muengtaweepongsa, Samuel Stevens, Jason Moss, Amy Pon, Kyra Lee, Charu Mehra, Jenny Gutierrez Villalobos, and Ranjitha Kumar. 2024. LLM-powered Multimodal Insight Summarization for UX Testing. In Proceedings of the 26th International Conference on Multimodal Interaction. 4–11. Motif: Discovering and Automating Personal Web WorkflowsConference’17, July 2017, Washington, DC, USA [48]Blase Ur, Elyse McManus, Melwyn Pak Yong Ho, and Michael L. Littman. 2014. Practical trigger-action programming in the smart home. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 803–812. doi:10.1145/2556288.2557420 [49]J Diego Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI conference on human factors in computing systems. 1–21. [50]Lefan Zhang, Weijia He, Olivia Morkved, Valerie Zhao, Michael L. Littman, Shan Lu, and Blase Ur. 2020. Trace2TAP: Synthesizing Trigger-Action Programs from Traces of Behavior. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 3, Article 104 (Sept. 2020), 26 pages. doi:10.1145/3411838 A Prompt used Abstracting actions You are an AI analyzing a user’s web activity. Based on the provided UI events and the attached screenshots, sum- marize the step-by-step actions the user took in this time period. NOTE: The ’path’ in the ’screenshot_taken’ events corre- sponds to the URIs of the images attached to this message. Use the timestamps and sequence of events to map the ac- tions to the correct images. The red dot inside the images stands for the cursor position at the time of the screenshot. === UI EVENTS === $uiEventsString === CORRELATED NETWORK CONTEXT === $net- workContextString Output your response as a concise list of sequential ac- tions meets the requested JSON schema. For each action: - "action": Provide a concise, human-readable description of what the user did. - "meta": Extract relevant technical iden- tifiers to help locate elements or replicate the action (e.g., target URL, exact text typed, element IDs, DOM classes, or correlated network payload keys). Identifying patterns Analyze raw browser logs and compile deterministic se- quences into Tampermonkey scripts. === RULES === 1. EXTRACT CODE: Write robust scripts (with // ==UserScript==) for automatable tasks (clicks, forms, APIs). Ignore random noise or visual reading. Sum- marize the code in plain English. 2. MAP TRACES (Ar- ray of Arrays): Chronologically group log ’id’s that the script automates. Maintain temporal isolation. Example: If a task happens twice, output [["1", "2"], ["8", "9"]]. NEVER merge distinct executions. 3. MATCH STATE: If the sequence matches a KNOWN PATTERN, output its exact name in ’existing_pattern_match’. If it is new, output ’NEW_PATTERN’ and provide a camelCase ’pat- tern_name’. === DATA === KNOWN PATTERNS: $existingPattern- Summaries NEW BATCH TO ANALYZE: $newLogs Program design You are an Expert Browser Automation Architect and UX Designer. I am providing you with multiple histori- cal execution traces of a repetitive user workflow ("$pat- tern.name"). Yourtaskistoanalyzetheseexecutions (’all_historical_executions’) to understand the core user intent. Identify the common denominator and abstract away any noise, misclicks, or highly specific one-off values (e.g., dynamic IDs, specific search strings). CRITICAL INSTRUCTIONS: 1. Optimize for User Experi- ence (UX): Do not just blindly automate the exact sequence. Think about the most user-friendly way for someone to in- teract with this script. Instead of aggressive auto-execution on page load, consider better patterns like injecting a float- ing "Execute Task" button, using a keyboard shortcut, or waiting for a specific contextual UI element to appear be- fore acting. 2. NO TECHNICAL JARGON: Describe the user journey in plain English (e.g., "clicks the search bar"). You are strictly forbidden from using HTML tags, CSS classes, XPaths, or DOM locators. 3. Concise Description: Provide a short, clear description of the logical workflow and how the user will trigger it. Focus entirely on the human-computer interaction. 4. Feedback Mechanism: Decide exactly when the script should trigger a message notification to inform the user of the script’s state (e.g., completion, error, need attention). Finally, evaluate how feasible this is to automate via Tam- permonkey, keeping in mind out of scope, complex iframes, or SPAs (Single Page Applications). Pattern Ranking You are an Automation Architect. Evaluate the following $totalPatterns patterns and predict their future automa- tion ROI (1-10). DATA CONTEXT: - ’occurrences’: A 2D array. Each inner array is a single execution. The timestamps inside repre- sent individual steps taken during that execution. CRITERIA for your score and reasoning: 1. Volume & Pace: High frequency, dense clusters, and rapid steps indicate high ROI. 2. Predictability: Ongoing, continuous needs score higher than one-off transient events. 3. Likelihood of happening in the future: Patterns that are fundamental to user workflows or show signs of increasing frequency are more likely to be automated and thus score higher. Provide an automation_value_score (1-10) and a brief rea- soning addressing the 3 criteria above. There are exactly $totalPatterns patterns in the data be- low. You MUST evaluate every single one. Do not skip, summarize, or omit any patterns. Your final JSON array must contain exactly $totalPatterns objects. Data: $dataForLLM Conference’17, July 2017, Washington, DC, USAShaokang Jiang and Daye Nam Program Generation You are an Expert Tampermonkey Developer. Write a ro- bust userscript based on the provided BLUEPRINT and TRACES. === REQUIREMENTS === 1. Behavior & UX: The BLUE- PRINT ’program_description’ dictates the final user experi- ence and goal. You MUST adhere to the intended UX (how it triggers and notifies). For ’notification’ situations, prefer Tampermonkey’s ‘GM_notification‘ (include ‘// @grant GM_notification‘). 2. Execution Toolkit: You decide the best technical ap- proach to achieve the program_description goal based on the provided traces. Choose the most reliable and ef- ficient strategy from the following viable options: - Net- work Requests: Consolidate manual clicks into background ` fetch()òr ` GM_xmlhttpRequest ` calls. Safeguard: Only simu- late state-changing requests (POST, PUT) if the exact end- point and payload are captured in the traces. Do not guess payloads. - Shortcuts: Leverage official keyboard short- cuts if they are faster/safer, invoking them via keyboard events. - DOM Targeting & Manipulation: When locating UI elements is required, you MUST use an **Anchor-First (Bottom-Up)** approach. Locate a highly stable element first, then traverse relatively if needed. - Identify multi- ple high-quality candidate anchors from the TRACES (e.g., Content-Based like aria-labels/visible text via Xpath, and Property-Based like semantic classes/IDs). All selectors MUST be semantically meaningful. - Then evaluate these candidates and explicitly select the SINGLE best, most ro- bust selector to implement. Prefer content based selector. Any ID, class, or text used MUST be explicitly captured and proven from the TRACES. *Note: ALWAYS wrap DOM selections and interactions in async wait functions.* 3. Planning First: BEFORE writing any code, you MUST output your thought process using two strict XML blocks: <plan> and <automation_steps>. - Inside <plan>: Analyze the traces and the scenario, evaluate the trade-offs of the strategies in your toolkit, and explicitly state which ap- proach is best for this specific task and why. - Inside <au- tomation_steps>: Map out the technical sequence for all scenarios based strictly on the TRACES. You should also plan for SPA rendering delays. For each step, use this ex- act pipe-separated format: [Action Name] | [Operation: Detail the exact execution strategy. If DOM, list final se- lected anchor. If Network, list the exact endpoint/payload. If Shortcut, list the exact keys. If the previous action trig- gered a UI change or network request, this step MUST explicitly define a Wait/Sleep operation.] 4. Format Requirements: You MUST output in exactly two parts. PART 1: The <plan> block, followed by the <au- tomation_steps> block. PART 2: Immediately following the </automation_steps> tag, output ONLY the raw JavaScript code starting with the standard // ==UserScript== header. Do not add any conversational text before or after the code. === BLUEPRINT (WHAT TO BUILD) === $blue- print.trigger_condition, blueprint.program_description, blueprint.notification === TRACES (HOW THE USER DID IT) === $allExecu- tions Tutorial Generation You are an expert Technical Writer. Your task is to write a clear, user-friendly Markdown tutorial for a newly gener- ated Tampermonkey userscript called "$patternName". === CONTEXT === BLUEPRINT (What the script was designed to do): $blueprint GENERATED SCRIPT (The actual code): “‘javascript $gen- eratedScript “‘ === REQUIREMENTS === Write a concise, easy-to-read Markdown document that includes: 1. **Overview:** A brief description of what the script does. 2. **How to Trig- ger It:** Step-by-step instructions on how the user activates the script (e.g., clicking a specific button, pressing a hotkey, or if it runs automatically on page load). 3. **What to Ex- pect:** What the UI or page will look like after the script successfully executes. Keep the tone helpful and accessible for a standard web user. Be concise and only provide the Markdown tutorial.

Instant research discovery