Paper deep dive
PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents
Yuxiang Chai, Shunye Tang, Han Xiao, Rui Liu, Hongsheng Li
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/13/2026, 12:39:07 AM
Summary
PIRA-Bench is a novel benchmark designed to evaluate Multimodal Large Language Models (MLLMs) on their ability to perform proactive intent recommendation in GUI environments. Unlike reactive agents that wait for explicit instructions, PIRA-Bench tests an agent's capacity to anticipate user goals from continuous, noisy visual streams, accounting for interleaved tasks and user-specific profiles. The paper also introduces the PIRF baseline, a memory-aware framework that uses state-tracking and reflection to manage long-horizon trajectories and reduce hallucinations.
Entities (4)
Relation Signals (3)
PIRF â empowers â MLLMs
confidence 95% ¡ a memory-aware, state-tracking framework that empowers general MLLMs
PIRA-Bench â evaluates â MLLMs
confidence 95% ¡ a novel benchmark for evaluating multimodal large language models (MLLMs)
PIRA-Bench â contains â PIRF
confidence 85% ¡ To establish a strong foundation for this challenging task, we further propose the Proactive Intent Recommendation Framework (PIRF).
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Current Graphical User Interface (GUI) agents operate primarily under a reactive paradigm: a user must provide an explicit instruction for the agent to execute a task. However, an intelligent AI assistant should be proactive, which is capable of anticipating user intentions directly from continuous visual inputs, such as mobile or desktop screenshots, and offering timely recommendations without explicit user prompting. Transitioning to this proactive paradigm presents significant challenges. Real-world screen activity is rarely linear; it consists of long-horizon trajectories fraught with noisy browsing, meaningless actions, and multithreaded task-switching. To address this gap, we introduce PIRA-Bench (Proactive Intent Recommendation Agent Benchmark), a novel benchmark for evaluating multimodal large language models (MLLMs) on continuous, weakly-supervised visual inputs. Unlike reactive datasets, PIRA-Bench features complex trajectories with multiple interleaved intents and noisy segments with various user profile contexts, challenging agents to detect actionable events while fitting to user preferences. Furthermore, we propose the PIRF baseline, a memory-aware, state-tracking framework that empowers general MLLMs to manage multiple task threads and handle misleading visual inputs. PIRA-Bench serves as an initial step toward robust and proactive GUI-based personal assistants.
Tags
Links
- Source: https://arxiv.org/abs/2603.08013v1
- Canonical: https://arxiv.org/abs/2603.08013v1
Trouble viewing inline? Open PDF directly â
Full Text
46,160 characters extracted from source content.
Expand or collapse full text
PIRA-Bench: A Transition from Reactive GUI Agents to GUI-based Proactive Intent Recommendation Agents Yuxiang Chai 1 , Shunye Tang 2 , Han Xiao 1 , Rui Liu 3 , Hongsheng Li 1â 1 MMLab @ CUHK, 2 Nankai University, 3 Huawei Research, â Corresponding Author https://w.pira-bench.top Abstract. Current Graphical User Interface (GUI) agents operate pri- marily under a reactive paradigm: a user must provide an explicit in- struction for the agent to execute a task. However, an intelligent AI assistant should be proactive, which is capable of anticipating user inten- tions directly from continuous visual inputs, such as mobile or desktop screenshots, and offering timely recommendations without explicit user prompting. Transitioning to this proactive paradigm presents significant challenges. Real-world screen activity is rarely linear; it consists of long- horizon trajectories fraught with noisy browsing, meaningless actions, and multithreaded task-switching. To address this gap, we introduce PIRA- Bench (Proactive Intent Recommendation Agent Benchmark), a novel benchmark for evaluating multimodal large language models (MLLMs) on continuous, weakly-supervised visual inputs. Unlike reactive datasets, PIRA-Bench features complex trajectories with multiple interleaved in- tents and noisy segments with various user profile contexts, challenging agents to detect actionable events while fitting to user preferences. Fur- thermore, we propose the PIRF baseline, a memory-aware, state-tracking framework that empowers general MLLMs to manage multiple task threads and handle misleading visual inputs. PIRA-Bench serves as an initial step toward robust and proactive GUI-based personal assistants. Keywords: Proactive Agents¡ GUI assistants¡ Benchmark 1 Introduction The rapid evolution of Multimodal Large Language Models (MLLMs) [1,2,15] has fundamentally transformed the landscape of human-computer interaction, paving the way for sophisticated Graphical User Interface (GUI) agents [5,6,12,13]. By leveraging strong visual understanding and reasoning capabilities, these agents can navigate complex operating systems and execute tasks across mobile and desktop environments. Recent industry and academic advancements, exemplified by integrated systems like the Doubao Phone, have demonstrated that modern GUI agents are highly capable of completing a wide array of user tasks accurately when provided with natural language instructions. Despite these impressive capabilities, the current paradigm of GUI automation remains fundamentally reactive. Existing state-of-the-art frameworks, such as arXiv:2603.08013v1 [cs.AI] 9 Mar 2026 2Authors Suppressed Due to Excessive Length Fig. 1: A comparison between the traditional reactive GUI agent paradigm and the proposed proactive intent recommendation agent. Reactive agents (left) function as passive executors, remaining idle until they receive instructions to initiate device control. In contrast, a proactive agent (right) continuously monitors the visual context to autonomously infer the userâs latent goals before the user even formulates a command. UI-TARS and UI-Venus, function primarily as passive executors. They require human users to specify instructions meticulously, often necessitating a high degree of detail to ensure successful task completion. This reliance on explicit prompting places a significant cognitive burden on the user. In dynamic, real-world scenarios, a user might easily forget specific context or omit crucial details, such as the exact time, location, or name of a restaurant during a conversation, which causes reactive agents to either fail or require tedious, step-by-step clarification. To bridge the gap between reactive automation and a "true" AI assistant, agents must evolve to anticipate user needs without requiring explicit, highly detailed prompts. We introduce the concept of the Proactive Intent Recommen- dation (PIR) agent, illustrated in Fig 1. A PIR agent continuously monitors a contextual stream of GUI screenshots (a sequence ofNimages) from a device and autonomously predicts what the user will likely want to do next. For example, if a user is chatting with a friend about planning a meal for the upcoming weekend, a PIR agent will seamlessly analyze the visual context, extract the relevant details, and proactively recommend a set of actionable instructions, such as booking a table at the mentioned restaurant, setting a reminder, and adding a calendar event. By independently formulating the exact instructions the user intended but had not yet typed, the agent acts as a comprehensive, forward-thinking assistant. To catalyze research and development in this promising new direction, we propose PIRA-bench, a novel benchmark designed to systematically evaluate Proactive Intent Recommendation in current models and agents. PIRA-bench comprises 100 annotated trajectories, with each trajectory containing an average PIRA-Bench3 of 32 sequential screenshots and paired with 3 distinct user profiles. To com- prehensively assess an agentâs reasoning capabilities, the benchmark evaluates performance across three scenario types: direct intent recommendation, profile- dependent prediction, and noise rejection. In direct recommendation scenarios, the agent must infer the userâs future actions relying exclusively on the visual con- text without profile distraction. Crucially, to reflect complex human multitasking, these trajectories frequently contain multiple interleaved and crossed intents. For instance, a user might alternate between chatting with a friend about a weekend meal and studying course materials; the agent must successfully disentangle these concurrent activities to proactively recommend a composite set of actions, such as booking a restaurant and creating a calendar event, alongside summarizing study materials. The second scenario introduces deep personalization, where the agent must condition its recommendations on specific user profiles. For example, given a similar housing search trajectory, an agent should recommend buying premium real estate for a wealthy user, while recommending affordable rentals for a student. Finally, to rigorously simulate real-world conditions where users are often idle or distracted, we insert noises to those trajectories and also include trajectories composed entirely of pure noise. This requires agents to not only extract hidden intents from cluttered streams but also strictly demonstrate operational restraint by preventing false predictions when no actionable intent exists. To establish a strong foundation for this challenging task, we further propose the Proactive Intent Recommendation Framework (PIRF). PIRF serves as a baseline designed to empower general MLLMs to iteratively process long sequences of visual context. The framework features a dedicated memory module that dynamically records and tracks ongoing multitasking states and specific user profile contexts. To effectively mitigate hallucinations, which is a common failure mode when general models process noisy trajectories, PIRF incorporates a built- in reflection mechanism. By continuously evaluating the memorized tasks and executing an auto-deletion process for outdated or completed intents, PIRF maintains a clean, accurate representation of the userâs true goals. To summarize, our main contributions are three-fold: â˘We introduce the Proactive Intent Recommendation (PIR) task, shifting the focus of GUI agents from purely reactive, instruction-following executors to forward-thinking assistants capable of anticipating user needs from continuous visual streams. â˘We construct PIRA-bench, a comprehensive dataset containing 100 real-world, multi-step trajectories with 3 user profiles each (averaging 33 screenshots). The benchmark is uniquely designed with interleaved multi-tasking scenarios, user profile context, and intentional noise to rigorously evaluate an agentâs ability to disentangle concurrent activities and filter out distractions. â˘We propose PIRF, a novel architecture that equips general MLLMs with iterative processing capabilities, a dynamic memory module, and a reflection- based auto-deletion mechanism. This framework effectively tracks complex, crossed intents and significantly reduces hallucinations when processing lengthy, noisy visual trajectories. 4Authors Suppressed Due to Excessive Length 2 Related Work 2.1 GUI Agent The rapid advancement of Multimodal Large Language Models (MLLMs) [?,1, 2,14] has catalyzed significant progress in the development of Graphical User Interface (GUI) agents [5, 6, 12, 13]. These systems are designed to perceive visual screen states and execute complex action sequences across mobile and desktop operating systems. Recent pioneering works have heavily optimized the instruction-following and visual-grounding capabilities of these agents. For instance, UI-TARS [10,17] and Mobile-Agent-V3/3.5 [19,21] have demonstrated robust, end-to-end navigation and task completion on smartphone interfaces, effectively mapping natural language commands to precise screen interactions. Similarly, models like UI-Venus [4, 16], UI-Genie [18] and MAI-UI [22] have introduced enhanced capability for better UI element localization and action planning, significantly improving the success rates of complex, multi-step tasks. Furthermore, the integration of advanced reinforcement learning and post training, as seen in models like InfiGUI-R1 [7], UI-R1 [8] and GUI-R1 [9], allows agents to have stronger reasoning capabilities. 2.2 General AI Assistant Beyond the specialized domain of GUI navigation, the broader trajectory of artificial intelligence research is shifting from passive tools toward autonomous, proactive personal agents. This paradigm shift is heavily exemplified by recent open-source initiatives such as OpenClaw 1 . Operating as a self-hosted, continuous automation engine, OpenClaw bridges the gap between digital environments and general task execution by integrating Large Language Models directly with local operating systems and cross-platform messaging APIs. Unlike traditional chatbots restricted to isolated, stateless sessions, these agents maintain persistent memory and operate continuously in the background. They demonstrate the viability of autonomous execution, managing calendars, summarizing communications, and running scheduled background tasks based on historical context without requiring immediate human prompting. In the academic pursuit of this proactive ideal, recent literature has begun exploring intent awareness on digital devices [3,20]. For instance, the FC-MIR framework [20] leverages screen context to detect user goals and recommend relevant actions. However, FC-MIR fundamentally focuses on identifying the intent of a task the user is currently executing or actively interacting with to accelerate in-progress workflows. Our proposed Proactive Intent Recommendation (PIR) task diverges significantly by focusing on the inference of future, latent goals. For example, a PIR agent must autonomously propose booking a restaurant based on the context of an ongoing messaging chat, rather than merely assisting within a restaurant application the user has already explicitly opened. 1 https://openclaw.ai/ PIRA-Bench5 3 Proactive Intent Recommendation Graphical User Interface (GUI) agents have emerged as powerful tools for digital automation, significantly reducing the manual effort required to navigate complex software ecosystems. By accurately mapping natural language instructions to executable system actions, these agents excel at completing well-defined tasks, from web browsing to application management. However, their utility is funda- mentally bottlenecked by their reactive nature. Users are required to explicitly formulate detailed instructions, a process that imposes a cognitive burden and interrupts the natural flow of work. In real-world scenarios, users frequently multitask, switch between applications, or simply forget specific context, such as the exact time of an appointment or the name of a venue mentioned in a passing conversation. To transcend the limitations of passive execution, there is a critical need for Proactive Intent Recommendation (PIR) agents. Rather than waiting for explicit commands, PIR agents continuously observe the userâs visual context, anticipate their latent goals amidst noise and distractions, and au- tonomously propose timely, actionable tasks. This shift from instruction-following to intent-anticipation represents the crucial next step in evolving GUI agents into comprehensive, intelligent assistants. 3.1 Reactive GUI Agent Paradigm Standard GUI agents operate as instruction-following executors within a sequen- tial decision-making framework, typically modeled as a Partially Observable Markov Decision Process (POMDP). Formally, given a specific, user-provided natural language instructionI, an initial visual states 0 , and a predefined action spaceA, the agentâs objective is to generate a sequence of actions to fulfillI. At any time stept, the agent receives a visual observations t (a GUI screenshot) and samples an action a t âA from its policy Ď: a t âź Ď(a| s t ,I,H <t ) whereH <t = (s 0 ,a 0 ,s 1 ,a 1 ,...,s tâ1 ,a tâ1 ) represents the historical trajectory of states and actions. In this paradigm, the instructionIis a strictly necessary prerequisite; without explicit human prompting, the policy cannot determine the goal state, rendering the agent idle. 3.2 Proactive Intent Recommendation (PIR) Paradigm In contrast, a comprehensive AI assistant must anticipate user needs before an explicit instructionIis given. We define the Proactive Intent Recommendation task as the continuous analysis of a passive observation stream to predict the userâs latent future goals, as illustrated in Fig 1. LetT= (s 1 ,s 2 ,...,s N ) denote a trajectory ofNsequential GUI screenshots passively captured from the userâs device. LetPdenote the user profile, encapsulating personalized preferences and socio-economic status. The objective of a PIR agent is to learn a mapping 6Authors Suppressed Due to Excessive Length functionf θ that predicts a set of future, actionable intentsI â =i 1 ,i 2 ,...,i K that the user is highly likely to execute or desire next. We formulate this mapping as finding the intent set that maximizes the conditional probability P θ : I â = f θ (T ,P) = arg max I P θ (I |T ,P) where each intenti k âI â is a natural language instruction or structured command (e.g., "book a table for two at Grnd Restaurant at 7 PM"). 3.3 Real-World Complexity: Interleaved Intents and Noise Real-world trajectories are rarely monolithic. Due to user multitasking and distractions, the trajectoryTis often a disjointed union of sub-trajectories corresponding to multiple concurrent tasks, alongside irrelevant noise. We can formulate the observed trajectory as: T =T task 1 âŞT task 2 ¡âŞT task M âŞT noise whereT task m represents the subset of non-contiguous frames relevant to a specific latent intenti m , andT noise represents frames resulting from app-switching, random browsing, or idle scrolling. Therefore, a successful PIR agent must not only maximizeP(I | T ,P) but also perform temporal credit assignment and disentanglement to successfully map interleaved state subsets (T task 1 ,...,T task M ) to their corresponding distinct intents, while aggressively driving the probability of generating intents from T noise to zero. 4 PIRA-Bench While the rapid advancement of Multimodal Large Language Models (MLLMs) has spurred the creation of numerous environments and datasets for GUI automa- tion, the research community currently lacks a standardized evaluation framework dedicated to proactive assistance. Existing benchmarks and mobile GUI datasets predominantly focus on reactive, instruction-following capabilities. In these tradi- tional setups, the evaluation metric is strictly tied to how accurately an agent executes a pre-defined, explicit natural language command. Consequently, current benchmarks or environments are structurally unsuited to assess an agentâs ability to infer latent, future goals from a passive, continuous stream of visual context. To accurately measure and drive progress in this paradigm shift, it is impera- tive to establish a rigorous evaluation standard. We need a framework that tests not just what a model can do when told, but how well it can anticipate user needs, handle interleaved multi-tasking, and filter out real-world noise to propose actionable tasks. To address this critical gap, we introduce PIRA-Bench, the first benchmark explicitly designed to evaluate Proactive Intent Recommendation Agents. By shifting the evaluation criteria from instruction-execution accuracy to intent-prediction relevance, PIRA-Bench provides the foundational infras- tructure necessary to systematically assess, compare, and improve the proactive capabilities of current and future MLLMs. PIRA-Bench7 4.1 Dataset Composition To systematically assess the proactive capabilities of MLLMs in a realistic envi- ronment, PIRA-Bench is constructed with a total of 100 meticulously curated GUI trajectories. These trajectories represent sequential, passive observation streams captured from diverse real-world device usage, encompassing both mobile phone and desktop environments. Rather than isolating capabilities into rigid subsets, the dataset presents a unified, mixed distribution of challenges that mirrors the unpredictability of actual user behavior. To ensure robust evaluation across different dimensions of proactivity, the 100 trajectories are designed with the following universal and variable features: ⢠Universal Noise Injection and Profiling: To simulate the clutter of realistic usage, every trajectory in the dataset is injected with noise frames, such as irrelevant app switching, idle screens, or random browsing at leisure, which requires agents to continuously filter out distractions. Furthermore, to enable scalable personalization testing, each of the 100 trajectories is paired with three distinct user profiles, encapsulating varying socio-economic statuses, preferences, and characteristics. â˘Spectrum of Intent Dependencies: Within the dataset, the trajectories cover a spectrum of dependency on user context: 1.Direct Recommendation Cases: In these instances, the visual context pro- vides sufficient information to infer the userâs future goals (e.g., "Schedule a meeting based on the chat content"). The latent intents are broadly applicable and can be predicted directly from the screenshots, testing the agentâs ability to disentangle interleaved tasks. 2.Profile-Dependent Cases: These trajectories present scenarios where the visual context alone is ambiguous or insufficient. Here, the agent must cross-reference the visual cues with the assigned user profile to gener- ate the correct intent (e.g., distinguishing whether a user intends to "buy a luxury apartment" or "rent a budget studio" based on financial constraints), thereby evaluating deep personalization capabilities. ⢠Negative Rejection Samples (Pure Noise): To rigorously test operational safety and hallucination resistance, a portion of the trajectories are designed as negative samples. Despite being paired with user profiles and containing valid GUI interactions, these sequences consist entirely of aimless actions or fragmented browsing with no actionable latent goal. For these cases, the ground-truth expectation is for the agent to correctly identify the absence of intent and strictly propose no action. By integrating these diverse scenarios, ranging from context-rich tasks to profile-dependent decisions and pure noise, into a single unified dataset, PIRA- Bench provides a holistic testbed for evaluating the predictive accuracy, person- alization capability, and operational restraint of proactive agents. 8Authors Suppressed Due to Excessive Length 4.2 Evaluation To ensure a robust and standardized assessment of proactive agents, PIRA-Bench employs a rigorous ground-truth annotation process coupled with an automated, scalable evaluation pipeline using state-of-the-art models. Ground Truth Annotation Establishing objective ground truth (GT) for latent intents is inherently challenging due to the subjective nature of human goals and the specific constraints of user profiles. To mitigate individual bias, we employ a consensus-based human annotation strategy. For each of the 100 trajectories, three independent humans are provided with the visual sequence and the specific user profile. They separately annotate the explicit, actionable intents they infer. The final GT intent set, denoted asI GT , is constructed by aggregating these annotations and retaining only the intents that reach a majority agreement (i.e., identified by at least two out of the three annotators). For trajectories designed as pure noise or where the profile constraints negate any actionable task, the consensus GT correctly maps to an empty set (I GT =â ). LLM-as-a-Judge Evaluation Evaluating the exact lexical match of predicted intents against the GT is sub-optimal. Therefore, we adopt an LLM-as-a-judge paradigm. Gemini-3-flash is utilized to semantically compare the agentâs predicted intent set,I pred , againstI GT . Crucially, the judge is also provided with the user profile context to supplementarily verify that the predicted intents satisfy the specific socio-economic and preference constraints of the user. Metrics Formulation To comprehensively capture an agentâs ability to accu- rately anticipate profile-aligned intents while resisting hallucinations, we define three core metrics based on the nature of the ground truth: â˘Average Intent F1 Score (F1 avg ): This metric assesses the modelâs accuracy on all trajectories where actionable intents exist (i.e.,I GT ̸=â ). We compute the F1 score independently for each such trajectoryjby comparing the semantic alignment ofI (j) pred andI (j) GT . The final score is the macro-average across all positive samples: F1 avg = 1 |T pos | X jâT pos F1 (j) where T pos represents the set of trajectories containing valid latent intents. ⢠Normalized False Positive Score (FPS norm ): To rigorously assess robust- ness, we evaluate performance on all trajectories where no actionable intent exists. We first calculate the False Positive Score (FPS) as the average number of hallucinated intents per negative sample. To map this unbounded count to a normalized metricFPS norm â(0,1], we apply a logarithmic dampening function: FPS norm = 1 1 + ln(1 + FPS) PIRA-Bench9 This formulation mimics the diminishing returns of error perception, which strictly penalizes the initial hallucinations while preventing the metric from vanishing too rapidly for models with moderate noise levels. A perfect model (FPS = 0) achieves a score of 1.0. â˘Final Score (S final ): To provide a unified assessment of an agentâs overall reliability, we compute the Final Score as the product of the proactive capability and the hallucination penalty: S final = F1 avg ¡ FPS norm By formulating the Final Score multiplicatively,FPS norm acts as a strict reliability scaling factor. This ensures that a top-performing agent must simultaneously master complex, profile-aware intent prediction and maintain operational restraint in noisy environments. 5 Proactive Intent Recommendation Framework (PIRF) To establish a robust baseline for the PIRA-Bench dataset, we propose the Proactive Intent Recommendation Framework (PIRF). Standard Multimodal Large Language Models, while highly capable of single-turn visual understanding, often struggle with long-horizon context retention and are highly susceptible to hallucinations when presented with noisy, continuous visual streams (see in Section 6). PIRF addresses these limitations by wrapping a general MLLM in a structured, state-tracking cognitive architecture designed specifically for continuous intent disentanglement and reflection, as illustrated in Fig. 2. 5.1 Architecture and Dynamic Memory Module At its core, PIRF treats the intent recommendation task as a continuous state- updating process. Instead of processing the entire trajectoryTat once, which is computationally expensive and inapplicable to the real-world scenario, PIRF processes the visual stream sequentially. To handle the complex, interleaved multi- tasking scenarios in PIRA-Bench and the real world, PIRF employs a multifaceted dynamic Memory Module. This module anchors the user profileP, capturing essential personalization constraints, socio-economic context, and individual preferences. Concurrently, it maintains a dynamic list of active "threads," where each thread represents a distinct, suspended user intent. At any time stept, the framework injects this combined structured memory state, which pairs the static user profile with the ongoing tasks (e.g., Intent 1: Study machine learning, Intent 2: Reserve a table), directly into the MLLMâs context. To provide immediate temporal context without maintaining a huge context window, PIRF also utilizes a sliding conversational window, retaining only theKmost recent frames and reasoning steps (where K = 10 in our baseline implementation). 10Authors Suppressed Due to Excessive Length Fig. 2: The overall architecture of the Proactive Intent Recommendation Framework (PIRF). The system processes continuous visual inputs sequentially by integrating a sliding window of historical context with a dynamic Memory Module that maintains both static user profiles and a continuously updated intent bank. Driven by a Multimodal Large Language Model, the framework evaluates the current state against a structured action space (i.e., Create, Resume, Update, or IDLE) to autonomously refine its memory and generate a final set of actionable, proactive intents. 5.2 Intent Action Space and State Transitions At each frame, the framework prompts the underlying MLLM to analyze the current visual observation against the Suspended Intents memory and output a structured state transition. The action space is defined as follows: ⢠CREATE: Triggered when the visual context indicates the initiation of a novel task. PIRF instantiates a new thread ID and generates an abstract, proactive intent description. ⢠RESUME: Triggered when the user switches back to a previously suspended task. The framework updates its active pointer to the corresponding thread ID, effectively disentangling interleaved multi-tasking. ⢠UPDATE : Triggered when the current screen represents a continuation or progression of the currently active intent, allowing the framework to refine the intent description as more context is revealed. ⢠IDLE: A critical action for hallucination mitigation. When the framework detects that the current screen is meaningless noise (e.g., idle scrolling, application homepages, or random browsing), it outputs IDLE. This explicit rejection class prevents the model from over-triggering and ensures safety on the pure noise subsets of PIRA-Bench. PIRA-Bench11 5.3 Reflection and Auto-Deletion Mechanism A significant challenge in processing long GUI trajectories is memory bloat: as users abandon tasks due to context changing, stale intents remain in the context window, confusing the model and degrading the average intent F1 score. To combat this, PIRF incorporates a Reflection and Auto-Deletion Mechanism. Independent of the primary state transitions (CREATE, RESUME, etc.), the framework enforces a continuous reflection protocol at every time step. The model must explicitly evaluate if the visual evidence suggests any intent in the memory bank has been either modified or abandoned. If so, it issues adelete_intent_id parameter. PIRF immediately purges these obsolete threads from the active memory. This auto-deletion ensures that the cognitive load remains low and the predicted intent pool strictly reflects the userâs current latent goals, maximizing the final reliability score S final . 6 Experiments 6.1 Settings We evaluate the performance of state-of-the-art Multimodal Large Language Models (MLLMs) on PIRA-Bench using two distinct experimental settings (i.e., a naive baseline and our proposed Proactive Intent Recommendation Framework (PIRF)) alongside a human performance reference: â˘Naive MLLM Baseline: To assess the inherent proactive capabilities of current models without specialized architecture, we employ a standard "sliding context" approach. In this setting, the MLLM is fed a sequence of N= 10 resized screenshots at each turn. To serve as a naive baseline, the model is instructed via the system prompt to "remember the context and observe user behavior" for the initial frames and is strictly forbidden from outputting predictions until the final turn of the window. This baseline tests the modelâs raw ability to retain visual context and identify intents without external memory modules or structured state tracking. â˘PIRF (Ours): We evaluate the models within the Proactive Intent Rec- ommendation Framework (PIRF) described in Section 5. In this setup, the models utilize the dynamic Memory Module to track suspended intents and the Reflection Mechanism to auto-delete completed tasks. The context win- dow is similarly maintained at 10 frames, but the model is queried at every step to update its memory state (CREATE, RESUME, UPDATE) or remain IDLE, rather than just at the end of the sequence. â˘Human Performance: To establish an empirical upper bound and validate the solvability of the benchmark, we conducted a human performance study. Human evaluators were presented with the same sequential visual streams and user profiles as the models. They were asked to identify actionable intents and determine when no action was required (noise). This baseline serves as a reference for the ideal balance between intent discovery (Recall) and hallucination resistance (F1 and FPS). 12Authors Suppressed Due to Excessive Length Table 1: Main results on PIRA-Bench. We report the average Precision, average Recall, and average F1 (F1 avg ) for intent detection, alongside the Normalized False Positive Score (FPS norm ) for noise robustness. The Final Score (S final ) is the product ofF1 avg and FPS norm . All values are reported as percentages (%). MethodModelPrecisionRecallF1 avg FPS norm S final Naive Gemini-3.1-Pro48.5769.9745.0849.6122.36 GPT-5.231.9583.3740.7531.3112.76 Qwen3.5-Plus40.4769.3646.5845.4621.18 Seed-1.843.5368.7447.6848.8523.29 PIRF Gemini-3.1-Pro53.0578.9756.5845.3925.68 GPT-5.250.5284.5454.6843.9024.00 Qwen3.5-Plus42.0770.2149.8747.6023.73 Seed-1.851.8272.6755.7150.3628.05 Human-98.7689.6793.8996.2390.35 For each MLLM setting, we conduct our evaluation on four leading MLLMs to represent a diverse range of capabilities: Gemini-3.1-Pro, GPT-5.2, Qwen3.5- Plus [15], Seed-1.8 [11]. The quantitative results for the Naive Baseline, PIRF, and Human Performance are presented in Table 1. The performance of GPT-5.2 in the Naive setting serves as a stark illustration of the "over-proactivity" trap. While it achieves a remarkable recall of 83.37%, which is the highest among all naive baselines, this sensitivity proves deceptive. The accompanying precision is critically low (31.95%), and its noise robustness score (FPS norm ) is the worst in the table at 31.31%. This pattern characterizes a "trigger-happy" model: it correctly identifies most true user intents simply by aggressively predicting intents from many noise frames. In a real-world proactive assistant, this behavior is catastrophic; the utility of correct suggestions is drowned out by a flood of hallucinations during idle moments. Consequently, despite having high recall, its final score (S final ) is penalized heavily (12.76), accurately reflecting that a hyper-active assistant is functionally unusable. From the table, a notable observation is the consistent superiority of the PIRF framework over the Naive baseline across all evaluated models. By integrating a structured dynamic memory module and a reflection mechanism, PIRF signifi- cantly improves the final score (S final ) for every model. For GPT-5.2, PIRF acts as a cognitive filter. While recall remains high (increasing slightly to 84.54% due to better context tracking), the reflection mechanism drives a dramatic improvement in precision (+18.57 points to 50.52%) and noise robustness (+12.59 points to 43.90%). This indicates that the framework successfully disentangles "eagerness" from "accuracy," allowing the model to retain its reasoning power while learning to remain silent during noise. A comparison between Gemini-3.1-Pro and Seed-1.8 highlights a fundamental trade-off in proactive agent design: Gemini-3.1-Pro (PIRF) achieves the highest F1 avg (56.58%) and precision (53.05%). It can formulate the most accurate intent PIRA-Bench13 descriptions when tasks are present. Seed-1.8 (PIRF), however, achieves the highest final score (28.05). This victory is driven by its superiorFPS norm (50.36%). This result suggests that for the PIRA-Bench metric, operational restraint is weighted as heavily as capability. Seed-1.8 acts as a "conservative" agent: it misses some intents (lower Recall of 72.67%), but it rarely hallucinates during noise. In the context of an always-on background assistant, this conservative behavior yields a higher overall reliability score than a more capable but "noisier" model. The data suggests that future research must focus not just on boosting recall, but on teaching frontier models when not to act. The human performance baseline reveals a substantial gap between current MLLMs and human capability, with the humanS final of 90.35 far surpassing the best model performance of 28.05 (Seed-1.8). Crucially, this disparity is driven not merely by the ability to identify intents, where GPT-5.2âs recall of 84.54% approaches the human benchmark of 89.67%, but by the profound difference in operational restraint. Humans demonstrate near-perfect precision (98.76%) and noise robustness (96.23%), indicating an almost flawless ability to distinguish true latent goals from background noise. In contrast, even the most capable models struggle significantly with false positives, underscoring that the primary challenge for future proactive agents is not just increasing sensitivity, but mastering the discernment to remain silent when no action is required. However, this superior discernment comes at a significant cost, as human testers required approximately 15 to 20 times the inference time of the models to process the same trajectories. 6.2 Ablation Study To quantify the "distraction cost" of real-world visual clutter, we conducted an ablation study analyzing the impact of noise injection on agent performance. In this experiment, we compared the performance of the PIRF framework across two dataset variations on the positive samples which have intents: â˘Clean Trajectories: The original sequences with all noise frames (idle scrolling, etc.) manually removed, leaving only the intent-relevant frames. â˘Noised Trajectories (Standard): The standard PIRA-Bench evaluation subset, containing the full mixture of relevant actions and injected noise. The ablation study results in Table 2 isolate the "distraction cost" of real- world visual environments. By comparing performance on Clean trajectories (where only intent-relevant frames are retained) versus Noised trajectories (with noise frames injected), we observe a distinct failure mode in current MLLMs: high susceptibility to visual distraction. The most significant finding is the drastic collapse in precision across all models when noise is introduced. GPT-5.2, which achieves a near-perfect Precision of 92.23% on clean trajectories, plummets to 50.52% on Noised trajectories, which is a degradation of over 40 percentage points. Similarly, Gemini-3.1-Pro drops from 85.28% to 53.05%. This indicates that while current MLLMs are highly capable of interpreting user behavior in idealized, noise-free environments (CleanF1 avg >74%), they lack the robustness 14Authors Suppressed Due to Excessive Length Table 2: Ablation study on the impact of noise injection using the PIRF framework. We compare performance on Clean trajectories (noise frames removed) versus Noised trajectories (standard PIRA-Bench). All values are reported as percentages (%). TrajModelPrecisionRecallF1 avg Noised Gemini-3.1-Pro53.0578.9756.58 GPT-5.250.5284.5454.68 Qwen3.5-Plus42.0770.2149.87 Seed-1.851.8272.6755.71 Clean Gemini-3.1-Pro85.2874.4476.42 GPT-5.292.2383.5784.46 Qwen3.5-Plus81.7872.6274.02 Seed-1.883.3471.8674.39 to function reliably in the wild. The introduction of irrelevant frames (e.g., idle scrolling or random browsing) confuses the models, causing them to misinterpret noise as meaningful trigger signals and hallucinate intents that do not exist. Counter-intuitively, the Recall for the top-performing models (Gemini-3.1- Pro and GPT-5.2) actually increases slightly in the Noised setting (e.g., GPT- 5.2 improves from 83.57% to 84.54%). This phenomenon suggests that noise triggers an "over-proactive" behavior. Confronted with a cluttered visual stream, these powerful models tend to lower their detection thresholds and generate more predictions to ensure they donât miss potential tasks. While this strategy successfully captures the true intents (maintaining or boosting Recall), it comes at the devastating cost of Precision described above. The models effectively become "trigger-happy," failing to exercise the restraint necessary for a helpful assistant. 7 Conclusion In this work, we introduce PIRA-bench, a novel benchmark designed to evaluate GUI-based proactive assistants. Through 100 diverse, real-world trajectories paired with distinct user profiles and realistic visual noise, we provide the first systematic assessment of an agentâs ability to autonomously discover latent intents, disentangle interleaved tasks, and personalize recommendations. To address these complexities, we propose the Proactive Intent Recommendation Framework (PIRF), a baseline architecture integrating dynamic memory with reflection. Our extensive evaluation reveals that while frontier models demonstrate high recall, they suffer from significant "over-proactivity" and hallucinate intents when confronted with visual noise. However, the consistent gains achieved by PIRF demonstrate that structured state tracking and self-reflection are viable pathways to mitigate these failures, establishing a rigorous standard for future research into agents that are not only smarter but also more discerning about when to act. PIRA-Bench15 References 1.Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y., Tang, J., Tu, J., Wan, J., Wang, P., Wang, P., Wang, Q., Wang, Y., Xie, T., Xu, Y., Xu, H., Xu, J., Yang, Z., Yang, M., Yang, J., Yang, A., Yu, B., Zhang, F., Zhang, H., Zhang, X., Zheng, B., Zhong, H., Zhou, J., Zhou, F., Zhou, J., Zhu, Y., Zhu, K.: Qwen3-vl technical report (2025), https://arxiv.org/abs/2511.21631 2.Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang, P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., Lin, J.: Qwen2.5-vl technical report (2025), https://arxiv.org/abs/2502.13923 3. Cohen, D., Halpern, Y., Kahlon, N., Oren, J., Berkovitch, O., Caduri, S., Dagan, I., Efros, A.: Small models, big results: Achieving superior intent extraction through de- composition. In: Christodoulopoulos, C., Chakraborty, T., Rose, C., Peng, V. (eds.) Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. p. 18780â18799. Association for Computational Linguistics, Suzhou, China (Nov 2025).https://doi.org/10.18653/v1/2025.emnlp-main.949,https: //aclanthology.org/2025.emnlp-main.949/ 4. Gu, Z., Zeng, Z., Xu, Z., Zhou, X., Shen, S., Liu, Y., Zhou, B., Meng, C., Xia, T., Chen, W., Wen, Y., Dou, J., Tang, F., Lin, J., Liu, Y., Guo, Z., Gong, Y., Jia, H., Gao, C., Guo, Y., Deng, Y., Guo, Z., Chen, L., Wang, W.: Ui-venus technical report: Building high-performance ui agents with rft (2025),https: //arxiv.org/abs/2508.10833 5.Hu, X., Xiong, T., Yi, B., Wei, Z., Xiao, R., Chen, Y., Ye, J., Tao, M., Zhou, X., Zhao, Z., Li, Y., Xu, S., Wang, S., Xu, X., Qiao, S., Wang, Z., Kuang, K., Zeng, T., Wang, L., Li, J., Jiang, Y.E., Zhou, W., Wang, G., Yin, K., Zhao, Z., Yang, H., Wu, F., Zhang, S., Wu, F.: Os agents: A survey on mllm-based agents for general computing devices use (2025), https://arxiv.org/abs/2508.04482 6.Liu, G., Zhao, P., Liang, Y., Liu, L., Guo, Y., Xiao, H., Lin, W., Chai, Y., Han, Y., Ren, S., Wang, H., Liang, X., Wang, W., Wu, T., Lu, Z., Chen, S., LiLinghao, Wang, H., Xiong, G., Liu, Y., Li, H.: LLM-powered GUI agents in phone automation: Surveying progress and prospects. Transactions on Machine Learning Research (2025), https://openreview.net/forum?id=yWQqoi1G1K 7. Liu, Y., Li, P., Xie, C., Hu, X., Han, X., Zhang, S., Yang, H., Wu, F.: Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners (2025), https://arxiv.org/abs/2504.14239 8.Lu, Z., Chai, Y., Guo, Y., Yin, X., Liu, L., Wang, H., Xiao, H., Ren, S., Xiong, G., Li, H.: Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning (2025), https://arxiv.org/abs/2503.21620 9.Luo, R., Wang, L., He, W., Chen, L., Li, J., Xia, X.: Gui-r1 : A generalist r1-style vision-language action model for gui agents (2025),https://arxiv.org/abs/2504. 10458 10. Qin, Y., Ye, Y., Fang, J., Wang, H., Liang, S., Tian, S., Zhang, J., Li, J., Li, Y., Huang, S., Zhong, W., Li, K., Yang, J., Miao, Y., Lin, W., Liu, L., Jiang, X., Ma, Q., Li, J., Xiao, X., Cai, K., Li, C., Zheng, Y., Jin, C., Li, C., Zhou, X., Wang, M., Chen, H., Li, Z., Yang, H., Liu, H., Lin, F., Peng, T., Liu, X., Shi, G.: Ui-tars: Pioneering automated gui interaction with native agents (2025), https://arxiv.org/abs/2501.12326 16Authors Suppressed Due to Excessive Length 11.Seed, B.: Seed1.8 model card: Towards generalized real-world agency. Tech. rep., Technical report (model card), December 2025. (2025) 12.Shi, Y., Yu, W., Huang, J., Yao, W., Chen, W., Liu, N.: Towards trustworthy gui agents: A survey (2026), https://arxiv.org/abs/2503.23434 13.Tang, F., Xu, H., Zhang, H., Chen, S., Wu, X., Shen, Y., Zhang, W., Hou, G., Tan, Z., Yan, Y., Song, K., Shao, J., Lu, W., Xiao, J., Zhuang, Y.: A survey on (m)llm-based gui agents (2025), https://arxiv.org/abs/2504.13865 14.Team, K., Bai, T., Bai, Y., Bao, Y., Cai, S.H., Cao, Y., Charles, Y., Che, H.S., Chen, C., Chen, G., Chen, H., Chen, J., Chen, J., Chen, J., Chen, J., Chen, K., Chen, L., Chen, R., Chen, X., Chen, Y., Chen, Y., Chen, Y., et al.: Kimi k2.5: Visual agentic intelligence (2026), https://arxiv.org/abs/2602.02276 15. Team, Q.: Qwen3.5: Accelerating productivity with native multimodal agents (February 2026), https://qwen.ai/blog?id=qwen3.5 16.Team, V., Gao, C., Gu, Z., Liu, Y., Qiu, X., Shen, S., Wen, Y., Xia, T., Xu, Z., Zeng, Z., Zhou, B., Zhou, X., Chen, W., Dai, S., Dou, J., Gong, Y., Guo, Y., Guo, Z., Li, F., Li, Q., Lin, J., Zhou, Y., Zhu, L., Chen, L., Guo, Z., Meng, C., Wang, W.: Ui-venus-1.5 technical report (2026), https://arxiv.org/abs/2602.09082 17.Wang, H., Zou, H., Song, H., Feng, J., Fang, J., Lu, J., Liu, L., Luo, Q., Liang, S., Huang, S., Zhong, W., Ye, Y., Qin, Y., Xiong, Y., Song, Y., Wu, Z., Li, A., Li, B., Dun, C., Liu, C., Zan, D., Leng, F., Wang, H., Yu, H., Chen, H., Guo, H., Su, J., Huang, J., Shen, K., Shi, K., Yan, L., Zhao, P., Liu, P., Ye, Q., Zheng, R., Xin, S., Zhao, W.X., Heng, W., Huang, W., Wang, W., Qin, X., Lin, Y., Wu, Y., Chen, Z., Wang, Z., Zhong, B., Zhang, X., Li, X., Li, Y., Zhao, Z., Jiang, C., Wu, F., Zhou, H., Pang, J., Han, L., Liu, Q., Ma, Q., Liu, S., Cai, S., Fu, W., Liu, X., Wang, Y., Zhang, Z., Zhou, B., Li, G., Shi, J., Yang, J., Tang, J., Li, L., Han, Q., Lu, T., Lin, W., Tong, X., Li, X., Zhang, Y., Miao, Y., Jiang, Z., Li, Z., Zhao, Z., Li, C., Ma, D., Lin, F., Zhang, G., Yang, H., Guo, H., Zhu, H., Liu, J., Du, J., Cai, K., Li, K., Yuan, L., Han, M., Wang, M., Guo, S., Cheng, T., Ma, X., Xiao, X., Huang, X., Chen, X., Du, Y., Chen, Y., Wang, Y., Li, Z., Yang, Z., Zeng, Z., Jin, C., Li, C., Chen, H., Chen, H., Chen, J., Zhao, Q., Shi, G.: Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning (2025), https://arxiv.org/abs/2509.02544 18. Xiao, H., Wang, G., Chai, Y., Lu, Z., Lin, W., He, H., Fan, L., Bian, L., Hu, R., Liu, L., Ren, S., Wen, Y., Chen, X., Zhou, A., Li, H.: UI-genie: A self-improving approach for iteratively boosting MLLM-based mobile GUI agents. In: The Thirty- ninth Annual Conference on Neural Information Processing Systems (2025),https: //openreview.net/forum?id=3uUmJzSSOW 19.Xu, H., Zhang, X., Liu, H., Wang, J., Zhu, Z., Zhou, S., Hu, X., Gao, F., Cao, J., Wang, Z., Chen, Z., Liao, J., Zheng, Q., Zeng, J., Xu, Z., Bai, S., Lin, J., Zhou, J., Yan, M.: Mobile-agent-v3.5: Multi-platform fundamental gui agents (2026), https://arxiv.org/abs/2602.16855 20.Yang, Z., Sheng, X., Zhang, Z., Wu, J., Wang, Z., He, X., Xu, S., Xiong, G.: Fc-mir: A mobile screen awareness framework for intent-aware recommendation based on frame-compressed multimodal trajectory reasoning (2025),https://arxiv.org/ abs/2512.19107 21.Ye, J., Zhang, X., Xu, H., Liu, H., Wang, J., Zhu, Z., Zheng, Z., Gao, F., Cao, J., Lu, Z., Liao, J., Zheng, Q., Huang, F., Zhou, J., Yan, M.: Mobile-agent-v3: Fundamental agents for gui automation (2025), https://arxiv.org/abs/2508.15144 22.Zhou, H., Zhang, X., Tong, P., Zhang, J., Chen, L., Kong, Q., Cai, C., Liu, C., Wang, Y., Zhou, J., Hoi, S.: Mai-ui technical report: Real-world centric foundation gui agents (2025), https://arxiv.org/abs/2512.22047