Paper deep dive

LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents

Weiming Li, Helen Paik, Yulei Sui

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 95

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 8/29/2026, 3:28:21 AM

Summary

The paper introduces LocalLSTC, a training-free architecture for locally deployed GUI agents that addresses the performance drop of local models (like Qwen3.5-9B) compared to frontier API models (like GPT-5). LocalLSTC organizes control by temporal scope, separating Long-Term Control (persistent state, subgoals, evidence) from Short-Term Execution (bounded commitments). It uses Long-to-Short Planning (L2S) and Short-to-Long Control (S2L) to maintain explicit cross-step state, reducing control failures. With Qwen3.6-27B, it achieves state-of-the-art local results on OSWorld (64.7% SR-100) and WindowsAgentArena (65.3% SR-100).

Entities (14)

LocalLSTC · method · 98%OSWorld · benchmark · 98%WindowsAgentArena · benchmark · 98%GPT-5 · model · 95%Qwen3.5-9B · model · 95%Qwen3.6-27B · model · 95%Long-Term Control · component · 92%Short-Term Execution · component · 92%Long-to-Short Planning · mechanism · 90%Short-to-Long Control · mechanism · 90%Agent S3 · method · 85%CoAct-1 · method · 85%GTA1 · method · 85%HISA · method · 85%

Relation Signals (10)

LocalLSTC → achievesscoreon → OSWorld

confidence 95% · With Qwen3.6-27B, LocalLSTC reaches 64.7% SR-100 on OSWorld

LocalLSTC → achievesscoreon → WindowsAgentArena

confidence 95% · LocalLSTC reaches ... 65.3% on WindowsAgentArena

LocalLSTC → uses → Long-to-Short Planning

confidence 95% · Long-to-Short Planning forms each commitment from persistent state

LocalLSTC → uses → Short-to-Long Control

confidence 95% · Short-to-Long Control integrates execution outcomes back into that state

LocalLSTC → uses → Long-Term Control

confidence 95% · LocalLSTC ... Long-Term Control maintains the active subgoal

LocalLSTC → uses → Short-Term Execution

confidence 95% · LocalLSTC ... Short-Term Execution realizes bounded commitments

GPT-5 → outperforms → Qwen3.5-9B

confidence 90% · replacing GPT-5 with Qwen3.5-9B reduces average OSWorld SR-100 from 60.9% to 37.7%

Qwen3.5-9B → usedin → LocalLSTC

confidence 90% · Under the same Qwen3.5-9B backbone, LocalLSTC reaches 49.1% SR-100

GTA1 → →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Modern GUI-agent frameworks achieve strong desktop task performance with frontier API models, yet persistent control information often remains implicit in growing interaction trajectories. At each step, the planner reconstructs the active task stage, accumulated evidence, and runtime feedback before deciding the next action. This dependence becomes more pronounced under weaker local reasoning backbones. Across four representative state-of-the-art frameworks, replacing GPT-5 with Qwen3.5-9B reduces average OSWorld SR-100 from 60.9\% to 37.7\%. Trajectory annotation further identifies at least one control failure in 91.6\% of failed trajectories. To address this problem, we introduce LocalLSTC, a training-free architecture that organizes control by temporal scope, maintaining persistent cross-step state to guide short-term execution commitments. Long-Term Control maintains the active subgoal, subgoal-aligned evidence, and runtime feedback across interactions, while Short-Term Execution realizes bounded commitments for the current step. Long-to-Short Planning forms each commitment from persistent state, and Short-to-Long Control integrates execution outcomes back into that state for progress assessment, recovery, and termination. With Qwen3.6-27B, LocalLSTC reaches 64.7\% SR-100 on OSWorld and 65.3\% on WindowsAgentArena, outperforming the strongest prior local results on both benchmarks. Ablations further support contributions from mechanisms on both sides of execution. These findings identify temporal organization of control information as a distinct architectural dimension for locally deployed GUI agents.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

91,682 characters extracted from source content.

Expand or collapse full text

LOCALLSTC: A LONG SHORT-TERM CONTROL AR- CHITECTURE FOR LOCALLY DEPLOYED GUI AGENTS Weiming Li, Helen Paik & Yulei Sui ∗ School of Computer Science and Engineering The University of New South Wales Sydney, Australia weiming.li1,h.paik,y.sui@unsw.edu.au ABSTRACT Modern GUI-agent frameworks achieve strong desktop task performance with frontier API models, yet persistent control information often remains implicit in growing interaction trajectories. At each step, the planner reconstructs the active task stage, accumulated evidence, and runtime feedback before deciding the next action. This dependence becomes more pronounced under weaker local reasoning backbones. Across four representative state-of-the-art frameworks, re- placing GPT-5 with Qwen3.5-9B reduces average OSWorld SR-100 from 60.9% to 37.7%. Trajectory annotation further identifies at least one control failure in 91.6% of failed trajectories. To address this problem, we introduce LocalLSTC, a training-free architecture that organizes control by temporal scope, maintain- ing persistent cross-step state to guide short-term execution commitments. Long- Term Control maintains the active subgoal, subgoal-aligned evidence, and run- time feedback across interactions, while Short-Term Execution realizes bounded commitments for the current step. Long-to-Short Planning forms each commit- ment from persistent state, and Short-to-Long Control integrates execution out- comes back into that state for progress assessment, recovery, and termination. With Qwen3.6-27B, LocalLSTC reaches 64.7% SR-100 on OSWorld and 65.3% on WindowsAgentArena, outperforming the strongest prior local results on both benchmarks. Ablations further support contributions from mechanisms on both sides of execution. These findings identify temporal organization of control infor- mation as a distinct architectural dimension for locally deployed GUI agents. 1INTRODUCTION GUI agents extend multimodal models from interface understanding to sequential action in computer-use tasks. Long-horizon tasks require agents to preserve the active task stage across in- teractions, interpret execution outcomes, recover from failures, and terminate only after task con- straints are satisfied. Recent systems achieve strong desktop task performance through hierarchical planning, specialized components, code execution, and test-time search (Agashe et al., 2025a;b; Gonzalez-Pumariega et al., 2025; Yang et al., 2026b; Song et al., 2026). Yet persistent control infor- mation often remains implicit in the interaction trajectory. At each new decision, the planner needs to reconstruct the active task stage, accumulated evidence, and relevant runtime feedback before choosing the next action. Local deployment changes this assumption. Keeping inference local allows screenshots, files, cre- dentials, browser state, and application data to remain within the local deployment boundary. It also reduces dependence on external-service latency, availability, and recurring invocation costs (Liu et al., 2024; Xue et al., 2024). Public benchmarks and open-model studies continue to report a capa- bility gap between locally deployed open models and strong API models on desktop tasks (Xie et al., 2024; Bonatti et al., 2025; Wang et al., 2025c). Recent reasoning models narrow this gap (Qwen Team, 2026a;b; Google DeepMind, 2026), while local hardware capacity, compute, and inference ∗ Corresponding author: y.sui@unsw.edu.au 1 arXiv:2608.25777v1 [cs.AI] 26 Aug 2026 ΔE t = SA(ĝ t , A t , o t ) Ẽ t+1 = Update(E t , ΔE t ) 5Step Abstraction (SA) PROBLEM SOLUTION ... Local GUI Agent (local backbone) Step t-1Step tStep 2Step 1 Trajectory history H t (b) Backbone Sensitivity (a) Implicit Trajectory-Level Control Burden (c) Control-Failure Diagnostics 7 Route z t → (C t+1 , q t+1 ) Long-Term Control across steps Long-to-Short Planning (L2S) Short-Term Execution within step (d) Long Short Term Control (LSTC) 6 Assign state z t = Assign(g t , K t , Ẽ t+1 ) Short-to-Long Control (S2L) → retain g t + append evidence → adopt ĝ t + refine context → return repair feedback → replan + suppress equivalents → invoke Final Verification continueadvance finishstallerror Compare candidate proposals P1P2Pn ... 2 Build subgoal- conditioned context 1 evidence E t domain skills subgoal g t feedback f t task instruction current screenshot 3 K t o t Implicit cross-step control in trajectory history imposes per-step reconstruction on local backbones, leading to control failures LocalLSTC makes cross-step control explicit with Long Short Term Control, coupling execution commitments with evidence-based state updates Cross-step control reconstructed from H t at every step Long-Term Control Recover task stage, progress evidence, and control intent Short-Term Execution Decide the current Step's execution Commit selected proposal proposed subgoal ĝ t thought ordered actions A t Execution Boundary Validate & Execute 4 WAIT (dynamic) / TERMINATE / INFEASIBLE GUI interaction bash execution software API Step t feedback f t evidence E t subgoal g t C t = (g t , E t , f t ) Step t+1 feedback f t+1 evidence E t+1 subgoal g t+1 C t+1 = (g t+1 , E t+1 , f t+1 ) Figure 1: Overview of the control problem, empirical evidence, and LocalLSTC. Panel (a) illustrates the implicit trajectory-level control burden. Panel (b) shows the SR-100 drop after replacing GPT-5 with Qwen3.5-9B across four GUI-agent frameworks. Panel (c) reports cross-step control-failure incidence under Qwen3.5-9B. Panel (d) presents the seven-step L2S–S2L control loop. throughput still constrain deployable model scale (Liu et al., 2024; Xue et al., 2024). Existing ap- proaches mainly rely on more complex system composition or GUI-specific training that changes model parameters (Agashe et al., 2025b; Song et al., 2026; Qin et al., 2025; Xu et al., 2025). Run- time interfaces for adapting off-the-shelf general backbones to open-ended desktop tasks remain less systematically studied (Xu et al., 2026b). Long-horizon GUI control requires information to persist beyond a single decision. Existing trajectory-conditioned interfaces leave the active task stage, accumulated evidence, and runtime feedback embedded in a growing interaction history. At each step, the backbone reconstructs this cross-step state before determining how to proceed. We call this repeated reconstruction the im- plicit trajectory-level control burden. Panel (a) of Figure 1 depicts this burden. As trajectories grow, reconstruction errors can manifest as loss of stage commitment, incorrect progress assess- ment, repeated equivalent actions after no-progress evidence, mismatched recovery, or premature termination without sufficient completion evidence. We refer to these observable manifestations as control failures. Controlled backbone replacement exposes this sensitivity. Across four representative state-of-the- art frameworks, replacing GPT-5 with local Qwen3.5-9B reduces average OSWorld SR-100 from 60.9% to 37.7%. Among failed trajectories from these local-backbone baselines, 91.6% contain at least one control failure. Under the same Qwen3.5-9B backbone, LocalLSTC reaches 49.1% SR-100 and reduces ANY control-failure incidence from the 59.4% baseline average to 43.1%. Panels (b) and (c) of Figure 1 summarize the backbone replacement and control-failure diagnostics. These results motivate explicit cross-step control state under constrained local reasoning backbones. To address this problem, we introduce LocalLSTC, a Long Short-Term Control Architecture for Lo- cally Deployed GUI Agents. LocalLSTC separates control by temporal scope. Long-Term Control preserves the active subgoal, subgoal-aligned evidence, and runtime feedback across interactions. Short-Term Execution represents a bounded commitment for the current step. This separation keeps 2 persistent control information available across decisions while limiting each execution commitment to the immediate interface state. Long-to-Short Planning (L2S) converts the persistent state and current observation into a subgoal- conditioned execution commitment. Short-to-Long Control (S2L) then abstracts the execution out- come into subgoal-aligned evidence and updates the persistent state for subsequent progress as- sessment, recovery, and termination. The resulting L2S–S2L loop carries the active subgoal, accu- mulated evidence, and runtime feedback across interactions without requiring each planner call to recover them from the full trajectory. With Qwen3.6-27B, LocalLSTC reaches 64.7% SR-100 on OSWorld and 65.3% on Win- dowsAgentArena, achieving the strongest literature-reported local results in our comparison on both benchmarks. Under the 50-step WindowsAgentArena setting, it reaches 63.4%, improving the pre- vious local result by 18.1 percentage points and coming within 0.1 points of the strongest reported API result. With Qwen3.5-9B fixed, removing L2S or S2L reduces OSWorld SR-100 from 49.1% to 36.4% and 41.5%, respectively, supporting contributions from mechanisms on both sides of exe- cution. Our contributions are as follows. • We quantify backbone sensitivity across four state-of-the-art GUI-agent frameworks and charac- terize failed local trajectories with five observable cross-step control-failure labels. • We introduce LocalLSTC, a training-free architecture that separates persistent control from step- local execution by temporal scope. Long-Term Control maintains cross-step state, Short-Term Execution realizes bounded commitments, and L2S and S2L connect the two scopes across exe- cution. • LocalLSTC achieves the strongest local results on OSWorld and WindowsAgentArena. Fixed- backbone analysis and ablations further isolate contributions from mechanisms on both sides of execution. 2RELATED WORK 2.1GUI-SPECIFIC MODELS AND PARAMETER ADAPTATION One route to improving locally deployable GUI agents is to strengthen the model itself through GUI-specific data and parameter adaptation. UI-TARS and Aguvis jointly train grounding, reason- ing, and GUI action generation (Qin et al., 2025; Xu et al., 2025). OS-ATLAS scales GUI grounding data to improve localization and out-of-distribution generalization, while AgentCPM-GUI combines grounding-aware pre-training, trajectory supervised fine-tuning, and reinforcement fine-tuning for mobile interaction (Wu et al., 2025; Zhang et al., 2025). These methods encode GUI-specific ca- pabilities into model parameters. LocalLSTC instead operates with an off-the-shelf backbone and focuses on how task stage, execution evidence, and runtime feedback are maintained across interac- tions. 2.2GUI AGENT FRAMEWORKS AND INFERENCE-TIME SCALING Beyond parameter adaptation, GUI-agent frameworks improve performance by strengthening the system around the backbone through composition, execution tools, and additional inference-time computation.Agent S and Agent S2 combine hierarchical planning, grounding, and execu- tion through an Agent-Computer Interface and generalist–specialist coordination (Agashe et al., 2025a;b). Agent S3 and GTA1 increase test-time computation through candidate generation, rollout selection, and judge-based ranking (Gonzalez-Pumariega et al., 2025; Yang et al., 2026b). CoAct- 1 extends GUI interaction with programmatic execution, while Avenir-Web and ClawGUI develop grounding and system infrastructure for broader computer-use settings (Song et al., 2026; Li et al., 2026a; Tang et al., 2026). These frameworks improve agent capability through richer system compo- sition, execution interfaces, or additional inference-time computation. LocalLSTC instead focuses on how persistent control information is organized across interactions around a fixed backbone. 3 2.3SEQUENTIAL PLANNING AND RUNTIME CONTROL Beyond these system-level approaches, sequential agents organize long-horizon decisions through search, subgoals, explicit execution states, and revisable plans. Language Agent Tree Search ex- plores candidate trajectories with inference-time search, while the Subgoal-driven Framework tracks task progress through online subgoal planning (Zhou et al., 2024; Wang et al., 2026). Agent-SAMA uses a finite-state machine for execution verification and recovery, AgentProg represents interaction history as programs with variables and control flow, and Plover exposes plans as inspectable and revisable objects (Guo et al., 2026; Tian et al., 2026; Venkatesan et al., 2026). LIFE-HARNESS adapts a runtime harness around a fixed model (Xu et al., 2026b). Unlike these approaches, Lo- calLSTC treats the temporal organization of control information across interactions as the primary architectural abstraction. 3METHOD 3.1PROBLEM FORMULATION AND STATE REPRESENTATION A common GUI-agent control interface conditions each decision on the current observation and accumulated interaction trajectory. At step t, a t = π(x t ,H t ), o t = Exec(a t ), H t+1 = H t ⊕ (a t ,o t ), where x t contains the task instruction and current observation, H t is the interaction trajectory accu- mulated through step t, a t is the action selected at the current step, and o t is the resulting execution outcome. This trajectory-conditioned interface leaves the active task stage, accumulated evidence, and runtime feedback implicit in H t , requiring them to be reconstructed before each new decision. LocalLSTC externalizes information that persists across interactions as a Long-Term Control State C t = (g t ,E t ,f t ), C 0 = (⊥,∅,⊥), where g t represents the active subgoal and unmet constraints, E t records subgoal-aligned execution evidence, and f t carries runtime feedback from errors, stalls, or rejected termination. Together, these variables preserve task stage, progress evidence, and control feedback across planner calls. The corresponding step-local object is a Short-Term Execution Commitment K t = (ˆg t ,A t ), where ˆg t is the execution subgoal selected for the current step and A t is the ordered action list generated under that subgoal. The commitment expires after its execution outcomes have been incorporated into the next control state. To connect these two temporal scopes across execution, LocalLSTC uses two complementary mech- anisms. Long-to-Short Planning (L2S) forms a Short-Term Execution Commitment from the persis- tent control state and current observation. Short-to-Long Control (S2L) incorporates the execution outcome into the next Long-Term Control State. The interaction between the two temporal scopes is K t = L2S(x t ,C t ), o t = Exec(A t ),(C t+1 ,q t+1 ) = S2L(C t ,K t ,o t ), where q t+1 ∈running,success,failure denotes task status after the state update. 3.2L2S–S2L CONTROL LOOP Panel (d) of Figure 1 instantiates the interaction between the two temporal scopes as a seven-step control cycle. Step 1 builds a subgoal-conditioned context from the task instruction, current ob- servation, domain skills, and persistent state C t . Step 2 compares candidate proposals under this shared context, and Step 3 commits the selected proposal as K t = (ˆg t ,A t ). Step 4 validates and executes the ordered actions in K t , producing the execution outcome o t . S2L then updates persis- tent control from this outcome. Step 5 applies Step Abstraction to convert o t into subgoal-aligned evidence, Step 6 assigns a transient routing state according to the resulting execution evidence, and Step 7 routes this state into C t+1 and task status q t+1 . The updated Long-Term Control State then conditions the next interaction. 4 3.2.1LONG-TO-SHORT PLANNING L2S converts persistent control state into the execution commitment for the current step. In step 1, g t provides the active stage reference, E t provides evidence accumulated from prior execution, and f t provides feedback from the preceding transition. The initial call derives the first subgoal from the task instruction and current observation. Subsequent calls condition planning on the persisted state. Domain skills are selected according to the active subgoal so that operational guidance remains aligned with the current task stage. Appendix D.1 details prompt construction. In step 2, the planner evaluates alternative proposals before committing to the next execution. A common inference-time approach improves selection by sampling candidates across multiple plan- ner calls and ranking them with a separate judge, as in GTA1 (Yang et al., 2026b). L2S instead performs proposal comparison within a single planner inference, with all candidates conditioned on the same persistent control state. In step 3, the selected proposal is serialized as a JSON object containing thought, subgoal, and actions. The proposal specifies the execution subgoal ˆg t , which may retain g t or propose a new subgoal, together with an action list A t . Actions can combine GUI, Bash, and software API channels as needed within the same commitment. LocalLSTC adopts programmatic Bash execution from CoAct-1 (Song et al., 2026), while its software API follows a schema-constrained interface similar to ComputerRL (Lai et al., 2026). Appendices D.2 and D.3 specify the action space and validation rules. 3.2.2EXECUTION Step 4 validates K t = (ˆg t ,A t ) before execution. The runtime checks the final JSON object, ac- tion schemas, and channel-specific parameters, then dispatches valid actions in list order through the corresponding GUI, Bash, or software API executor. Pre-execution fingerprint checks suppress repeated equivalent actions when the current state provides no basis for repetition. Reserved actions WAIT, TERMINATE, and INFEASIBLE are handled directly by the runtime. The resulting out- comes o t , together with K t , are passed to S2L. Appendices D.3 and D specify the validation and execution rules. 3.2.3SHORT-TO-LONG CONTROL S2L converts the outcome of the current commitment into the next persistent control state. In step 5, Step Abstraction (SA) evaluates the observed execution outcome against the intended effect of the current commitment. Given the execution subgoal ˆg t , ordered actions A t , and observed outcome o t , Step Abstraction determines whether execution produces task-relevant progress, no change, or an exception, and records this assessment as a subgoal-aligned evidence increment ∆E t = SA(ˆg t ,A t ,o t ), e E t+1 = Update(E t , ∆E t ). This process converts heterogeneous execution traces into evidence interpreted with respect to the action intent and current subgoal. The mechanism builds on HiSA’s Step Abstraction for GUI ex- ecution and extends the evidence interface to Bash and software API channels (Li et al., 2026b). Appendix D.4 details the channel-specific observation and abstraction mechanisms. In step 6, the runtime assigns a transient routing state z t = Assign(g t ,K t , e E t+1 ), where z t ∈ continue,advance,error,stall,finish. Assignment follows a fixed precedence. A TERMINATE request maps to finish. Execution failure maps to error, repeated no-progress maps to stall, and a changed execution subgoal maps to advance. All remaining cases map to continue. Appendix D.5 specifies the detection rules and thresholds. In step 7, the routing state determines the next control state (C t+1 ,q t+1 ) = Route(z t ,C t ,K t , e E t+1 ). The continue route retains the current subgoal and appends new evidence. The advance route adopts the proposed subgoal after execution provides supporting evidence and can trigger context 5 refinement, which compresses completed evidence into compact history while retaining recent ev- idence for subsequent planning and verification. LocalLSTC inherits this refinement mechanism from HiSA (Li et al., 2026b). The error route writes execution-specific feedback into the persis- tent state, while stall records no-progress or rejected-termination feedback for the next planner call. The finish route invokes Final Verification, which evaluates task completion against accu- mulated visual, textual, and software API evidence. A PASS sets task status to success, while a FAIL records the rejected termination, returns stall feedback, and resumes planning with status running. The INFEASIBLE action and predefined runtime failure boundaries set task status to failure. Step Abstraction and Final Verification use the planner backbone under dedicated prompts, while state assignment and routing are deterministic runtime operations. Appendices D.6, D.7, and D provide the implementation details. 4EXPERIMENTS Experiments evaluate three aspects of LocalLSTC. We first examine planner-backbone sensitivity and cross-step control failures in existing GUI-agent frameworks. We then evaluate LocalLSTC across two desktop benchmarks and multiple local backbones. Finally, fixed-backbone ablations isolate the contributions of L2S, S2L, and their constituent mechanisms, with runtime-event analysis examining how these mechanisms affect recovery after execution problems. 4.1EXPERIMENTAL SETUP Benchmarks. We evaluate LocalLSTC on OSWorld (Xie et al., 2024) and WindowsAgentArena (Bonatti et al., 2025). OSWorld contains 369 Linux tasks spanning single applications, file oper- ations, and cross-application workflows. Backbone-sensitivity experiments, control-failure anno- tation, fixed-backbone ablations, and runtime-event analyses are conducted on OSWorld, together with the main benchmark evaluation. WindowsAgentArena contains 154 Windows tasks cover- ing applications, system utilities, and browsers, and is used to evaluate performance in a distinct operating-system and application environment. Implementation. OSWorld and WindowsAgentArena run in benchmark-defined Ubuntu and Win- dows virtual machines, respectively, hosted in headless Docker containers. All experiments use screenshots at a resolution of 1280× 720. Local inference runs on a workstation with an Intel Core Ultra 7 265 processor and a 96 GB NVIDIA RTX Pro 6000 Blackwell Workstation Edition GPU. Unless stated otherwise, LocalLSTC uses Qwen3.5-9B (Qwen Team, 2026a) as the planner and GTA1-7B (Yang et al., 2026b) as the visual grounder. Additional experiments evaluate Qwen3.5- 4B, Qwen3.6-27B, Gemma-4-E4B-it, and Gemma-4-26B-A4B-it (Qwen Team, 2026a;b; Google DeepMind, 2026). The planner backbone is reused for SA, context refinement, and Final Verifica- tion under dedicated prompts. OSWorld backbone configurations are run independently twice with the same parameter settings while other results use one run. Appendix D.8 lists runtime parameters, Appendix D.9 specifies model roles and token accounting, and Appendix E provides the prompt templates. Control-failure annotation. To characterize the cross-step control failures associated with local- backbone execution, we use GPT-5.6-sol to perform multi-label annotation of OSWorld trajectories from Agent S3, CoAct-1, GTA1, HiSA, and LocalLSTC under the same Qwen3.5-9B planner back- bone. The labels STAGE, ASSESS, LOOP, RECOVER, and STOP capture unresolved failures in stage maintenance, progress assessment, repeated no-progress behavior, recovery, and termination. ANY indicates that a trajectory contains at least one of these five control failures, OTHER records a failure outside these categories, and NONE indicates that no final unresolved failure is present. To validate the reliability of the GPT annotations, we stratify OSWorld tasks by domain and ran- domly sample 37 tasks, yielding 185 trajectories across the five methods. Two PhD-level AI re- searchers independently annotate the trajectories under blinded method identities using the same protocol. 6 Metrics and statistics. We report task success rate (SR) under different step budgets. SR-100, SR- 50, and SR-15 impose maximum budgets of 100, 50, and 15 steps. Following OSWorld (Xie et al., 2024), one step corresponds to one planner decision. Control-failure comparisons use paired binary outcomes on the same tasks. Task-level 95% confidence intervals are estimated with a percentile bootstrap using 10,000 resamples and random seed 2027 (Efron, 1979). 4.2RESULTS 4.2.1BACKBONE SENSITIVITY AND CROSS-STEP CONTROL FAILURES Existing GUI-agent frameworks show a consistent sensitivity to reasoning-backbone strength. We evaluate this sensitivity on Agent S3, GTA1, CoAct-1, and HiSA through controlled reasoning- backbone replacement, substituting GPT-5 with Qwen3.5-9B while preserving each framework’s native prompts, grounder, action space, tools, search procedure, and 100-step environment bud- get. As shown in Figure 1 (b), SR-100 decreases by 21.1–25.0 points across the four frameworks, lowering their average from 60.9% to 37.7%. The consistency of this drop across different system compositions and inference-time strategies indicates that planner-backbone sensitivity is not specific to a single framework design. Under the same backbone, LocalLSTC reaches 49.1% SR-100, 11.4 points above the four-framework average. Trajectory annotation further localizes this sensitivity to cross-step control behavior. Among failed trajectories from the four baselines, 91.6% contain at least one control failure. RECOVER, STOP, and LOOP are the three most frequent categories, occurring in 64.2%, 41.1%, and 34.4% of failed trajec- tories, respectively. The dominant category varies across frameworks, indicating that the aggregate burden manifests through multiple failure modes. Human verification confirms the reliability of the aggregate annotation. The two annotators agree on 86.5% of ANY labels, with Cohen’s κ = 0.730. Against the two annotators, GPT-5.6-sol achieves ANY F1 scores of 86.1% and 85.2%, with κ = 0.686 and 0.647, respectively. Appendix A.4 reports the complete validation results. Across all OSWorld tasks, LocalLSTC reduces ANY control-failure incidence to 43.1%, compared with a 59.4% average across the four baselines. The 16.3-point reduction has a task-level boot- strap 95% CI of [8.3, 24.2]. LocalLSTC also achieves the highest NONE incidence at 42.3%. Ap- pendix A.5 reports the complete label distribution. 4.2.2BENCHMARK PERFORMANCE On OSWorld, LocalLSTC with Qwen3.6-27B and GTA1-7B reaches 64.7% SR-100, 62.7% SR- 50, and 51.4% SR-15, achieving the strongest local results at all three step budgets. These results improve the previous strongest local scores by 10.8, 6.0, and 5.1 points, respectively. The advantage persists as the step budget decreases, indicating that the gain does not depend on long trajectories or a large interaction budget. On WindowsAgentArena, the same configuration reaches 65.3% SR-100, 63.4% SR-50, and 48.5% SR-15, achieving the strongest local results on the benchmark. SR-50 improves the previous strongest local result by 18.1 points and matches the strongest API result within 0.1 points. The strong results across Ubuntu and Windows environments indicate that the control architecture trans- fers across distinct application ecosystems without benchmark-specific training. LocalLSTC also benefits consistently from stronger local reasoning backbones. Within both the Qwen3.5 and Gemma-4 families, increasing backbone scale improves SR at every evaluated step budget on both benchmarks. This scaling pattern places temporal control organization and backbone capability on complementary axes. Table 2 examines how the control structure is used across OSWorld domains with Qwen3.5-9B. Across the two runs, LocalLSTC averages 28.1 planner steps and 10.1 subgoal segments per task, with 3.7 actions per subgoal and 1.3 actions per planner step. The multi-action interface therefore supports short bounded commitments while each persistent subgoal spans multiple interactions. Execution channels also adapt to task structure. GUI actions account for 72.7% of execution overall, while Bash accounts for 15.1% and software APIs for 4.6%. OS tasks use Bash for 52.5% of actions, whereas Chrome and GIMP use GUI interaction for 86.1% and 85.5%, respectively. 7 Table 1: Success rates on OSWorld and WindowsAgentArena under step budgets of 100, 50, and 15. OSWorld results for LocalLSTC report mean±SD over two runs, and WindowsAgentArena results report one run. Other methods use their reported results. Bold and underlining mark the highest and second-highest values, and “-” indicates an unreported result. AccessMethod / Backbone OSWorldWindowsAgentArena SR-100↑SR-50↑SR-15↑ SR-100↑ SR-50↑ SR-15↑ API o3 (OpenAI, 2025b)23.017.29.1--- Agent S / GPT-4o (Agashe et al., 2025a)--20.6--18.2 computer-use-preview (OpenAI, 2025a)30.531.326.0--- Claude 3.7 Sonnet (Anthropic, 2025a)35.635.827.1--- Seed1.5-VL / Doubao-1.5-thinking-vision-pro (Guo et al., 2025a) 40.0-31.9--- Claude Sonnet 4 (Anthropic, 2025b)41.443.931.2--- Agent S2 / Gemini 2.5 Pro (Agashe et al., 2025b)-45.834.6-41.4- GTA1-32B / o3 (Yang et al., 2026b)53.148.6-51.2-- UI-TARS-2-2509 (Wang et al., 2025a)53.1---50.6- CoAct-1 / CUA 4o + o3 + o4-mini (Song et al., 2026)59.956.439.852.543.521.4 Agentic-Lybic-Maestro (Guo et al., 2025b)61.956.9---- Claude Sonnet 4.5 (Anthropic, 2025c)62.958.142.9--- GTA1 / GPT-5 (Yang et al., 2026b)63.4----- Agent S3 / GPT-5 (Gonzalez-Pumariega et al., 2025)63.5--50.249.0- Agent S3 / Opus 4.5 bBoN(N=1) (Gonzalez-Pumariega et al., 2025) 66.0----- UiPath Screen Agent / Opus 4.5 (Cristescu et al., 2025)67.164.4 ---- Agent S3 / GPT-5 bBoN (N=10) (Gonzalez-Pumariega et al., 2025) 69.9--56.654.1- VLAA-GUI / Gemini 3 Flash (Han et al., 2026)---61.0 60.4- OS-Symphony / GPT-5 (Yang et al., 2026a)-65.8--63.5- Claude Sonnet 4.6 (Anthropic, 2025d)72.1----- Agent S3 / Opus 4.5 + GPT-5 bBoN (N=10) (Gonzalez-Pumariega et al., 2025) 72.6----- VLAA-GUI / Opus 4.5 (Han et al., 2026)76.3----- LOCAL Qwen2.5-VL-7B (Bai et al., 2025)----3.43.4 Qwen2.5-VL-72B (Bai et al., 2025)5.0-4.4-9.7- Kimi-VL-A3B (Du et al., 2025)10.3-9.7--10.4 InternVL3.5-8B (Wang et al., 2025b)----10.5- InternVL3.5-38B (Wang et al., 2025b)----14.5- OpenCUA-Qwen2-7B (Wang et al., 2025c)23.120.619.9--- OpenCUA-7B (Wang et al., 2025c)26.628.224.3--- UI-TARS-72B-DPO (Qin et al., 2025)27.125.824.0-17.9- UI-TARS-1.5-7B (Qin et al., 2025)27.427.324.5-15.9- OpenCUA-32B (Wang et al., 2025c)34.834.129.7--- DeepMiner-Mano-7B (Fu et al., 2025)40.2----- OpenCUA-72B-Preview (Wang et al., 2025c)45.044.939.0--- OS-Symphony / Qwen3-VL-32B (Yang et al., 2026a)----45.3- EvoCUA-8B-20260105 (Xue et al., 2026)-46.1---- AutoGLM-OS-9B (Lai et al., 2026)-47.346.3 --- DeepMiner-Mano-72B (Fu et al., 2025)53.9----- GUI-Owl-1.5-32B (Xu et al., 2026a)-55.4---- EvoCUA-20260105 (Xue et al., 2026)-56.7---- OursLocalLSTC / Gemma-4-E4B-it + GTA1-7B28.1± 2.126.9± 2.720.3± 0.626.126.119.6 LocalLSTC / Qwen3.5-4B + GTA1-7B37.3± 4.835.5± 3.324.7± 1.230.423.921.7 LocalLSTC / Qwen3.5-9B + GTA1-7B49.1± 0.646.2± 1.134.8± 2.242.935.825.4 LocalLSTC / Gemma-4-26B-A4B-it + GTA1-7B60.3± 0.254.7± 2.442.2± 1.441.334.828.3 LocalLSTC / Qwen3.6-27B + GTA1-7B64.7± 0.662.7± 0.451.4± 1.965.363.448.5 Table 2: Domain-level runtime behavior of LocalLSTC with Qwen3.5-9B on OSWorld, averaged over two runs. Execution-channel columns report action shares. Pass, Fail, and Bypass denote accepted termination, rejected termination followed by recovery, and cutoff without verification. Recovery statistics are conditional on recovery entry. SummaryCommitmentExecution ChannelsEvidence Control Planning StructureChannel UseVerificationRecovery DomainTasks SR↑ (%) steps Subgoals / Task Actions / Subgoal Actions / step GUI (%) Bash (%) API (%) Context Refines Pass (%) Fail (%) Bypass (%) SR after↑ recovery (%) steps after recovery entry Chrome46 57.0 23.98.23.41.1 86.11.83.12.3 58.2 17.624.261.02.1 GIMP26 81.2 12.04.53.71.4 85.55.90.01.0 75.53.920.750.08.8 LibreOffice Calc47 44.1 28.510.63.91.5 74.48.59.32.9 39.0 24.136.987.56.9 LibreOffice Impress47 30.9 23.88.94.01.5 74.51.214.12.3 44.3 16.839.045.45.3 LibreOffice Writer23 47.5 31.510.84.31.5 80.86.84.43.0 43.2 25.231.765.03.9 Multi-apps101 32.7 38.114.03.41.3 67.2 25.21.54.1 52.3 19.728.137.25.5 OS24 72.9 24.06.44.41.1 40.9 52.50.02.2 66.7 16.716.7100.09.5 Thunderbird15 60.0 25.56.94.31.2 87.64.30.02.1 70.03.426.750.00.5 VLC17 62.1 31.711.43.21.1 71.7 14.93.43.4 68.7 14.317.274.711.5 VS Code23 77.7 19.58.13.31.4 69.9 20.40.21.8 38.2 29.532.385.71.6 All36949.128.110.13.71.372.715.14.62.853.018.328.863.86.0 The persistent-state mechanisms are exercised repeatedly during successful execution. LocalLSTC performs 2.8 context refinements per OSWorld task. Final Verification rejects termination on 18.3% 8 Table 3: Execution-boundary and component ablations on OSWorld with Qwen3.5-9B. Resource metrics report per-task averages. Bold marks the highest SR or lowest resource value in each column. CategoryMethodSR-100↑ Tokens (k)↓ Prompt (k)↓ Completion (k)↓ Steps↓ Time (s)↓ FullLocalLSTC49.1256.3240.615.728.1403.2 BOUNDARY w/o L2S36.4192.9179.113.831.4382.2 w/o S2L41.5246.9229.417.427.1473.6 LONG-TERM CONTROL w/o persistent subgoal38.0276.6257.319.329.6554.4 w/o Step Abstraction31.6270.4251.918.533.0550.8 w/o state-conditioned routing40.7215.2199.016.225.4431.3 w/o stall/loop handling45.1285.0266.918.129.8517.2 w/o Final Verification40.3268.7251.716.928.7510.7 SHORT-TERM EXECUTION w/o candidate proposals40.7273.0259.113.931.2412.6 w/o multi-action list43.5260.7243.017.828.8445.5 Table 4: Runtime-event diagnostics for Full, w/o L2S, and w/o S2L on OSWorld. Rate reports event incidence, and Score reports mean evaluator score conditional on entering the corresponding event. Bold marks the column maximum, and “–” indicates an event unavailable after removing the corresponding mechanism. Stall / repetitionRecovery entryRejected termination MethodRate (%) Score (%)↑ Rate (%) Score (%)↑ Rate (%) Score (%)↑ Full4.534.360.034.032.138.9 w/o L2S4.318.856.520.919.323.1 w/o S2L4.611.8– of tasks, and 63.8% of those tasks subsequently succeed after returning to execution. The same behavior appears on WindowsAgentArena, where verification enters recovery on 21.4% of tasks and 51.5% of those tasks subsequently succeed after an average of 5.2 additional steps. Appendix C reports the complete WindowsAgentArena runtime statistics. 4.2.3ABLATIONS AND MECHANISM ANALYSIS We next examine which mechanisms account for LocalLSTC’s gains under a fixed Qwen3.5-9B backbone. We first ablate L2S and S2L as grouped mechanisms, then remove individual components within Long-Term Control and Short-Term Execution. Appendix B specifies the exact w/o L2S and w/o S2L configurations. Removing L2S lowers SR-100 from 49.1% to 36.4%, and S2L to 41.5%. Both sides of the execu- tion boundary contribute under the same Qwen3.5-9B backbone. Among individual mechanisms, SA produces the largest effect, reducing SR-100 by 17.5 points. Removing the persistent subgoal re- duces SR by 11.1 points, while Final Verification, state-conditioned routing, candidate proposals, the multi-action list, and stall/loop handling contribute 8.8, 8.4, 8.4, 5.6, and 4.0 points, respectively. The largest effects occur in mechanisms that maintain or interpret cross-step evidence, consistent with the identified control burden. Runtime-event diagnostics separate event frequency from post-event behavior. Stall or repetition incidence is similar across Full, w/o L2S, and w/o S2L at 4.5%, 4.3%, and 4.6%. Conditional on such events, Full reaches a mean evaluator score of 34.3%, compared with 18.8% for w/o L2S and 11.8% for w/o S2L. The difference reflects how the agent responds after encountering no- progress behavior. The pattern extends to recovery and termination. After recovery entry, Full reaches 34.0% compared with 20.9% for w/o L2S. After rejected termination, the scores are 38.9% and 23.1%. L2S supplies the stage and evidence context for the next commitment, while S2L converts execution outcomes into recovery state. The event-conditioned results connect the grouped ablations to the runtime behavior producing the final SR differences. Appendix F provides trajectory-level examples. 5CONCLUSION LocalLSTC addresses the cross-step control burden in locally deployed GUI agents by separating persistent Long-Term Control from Short-Term Execution. L2S forms execution commitments from persistent state and current observations, while S2L incorporates outcomes into state for subsequent 9 planning, recovery, and termination. LocalLSTC improves task success and reduces control failures; ablations confirm contributions from both sides of the execution boundary. With Qwen3.6-27B, Lo- calLSTC reaches 64.7% SR-100 on OSWorld and 65.3% on WindowsAgentArena, the strongest local results on both benchmarks. These results establish temporal control organization as an archi- tectural dimension for locally deployed GUI agents. REPRODUCIBILITY STATEMENT Sections 3 and 4 describe the framework and experimental protocol. Appendix A details the control- failure annotation and human-verification procedures, and Appendix A.5 provides the corresponding task-level bootstrap analysis. Appendix D documents implementation details, while Appendix E includes the prompts for planning, repair, Step Abstraction, verification, and context refinement. The implementation, task-level outputs, and evaluation artifacts are released through an anonymous code repository for reproducibility. AI USE STATEMENT GPT-5.6-sol was used for post-hoc trajectory annotation. OpenAI Codex was used for language editing and L A T E X revision. All AI-assisted analyses and revisions were reviewed by the authors, who take responsibility for the final manuscript. REFERENCES Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like a human. In The Thirteenth International Conference on Learning Representations, 2025a. URL https://openreview.net/for um?id=lIVRgt4nLv. Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional generalist-specialist framework for computer use agents. In Second Conference on Language Modeling, 2025b. URL https://openreview.net/forum?id=zg5is4GJ 3R. Anthropic. Claude 3.7 Sonnet and Claude Code, February 2025a. URL https://w.anthro pic.com/news/claude-3-7-sonnet. Anthropic. Introducing Claude 4, May 2025b. URL https://w.anthropic.com/news /claude-4. Anthropic. Introducing Claude Sonnet 4.5, September 2025c. URL https://w.anthropi c.com/news/claude-sonnet-4-5. Anthropic. Introducing Claude Sonnet 4.6, 2025d. URL https://w.anthropic.com/ne ws/claude-sonnet-4-6. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025. doi: 10.48550/arXiv.2502.13923. URL https://arxiv.org/abs/2502.13923. Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Keunho Jang, and Zheng Hui. Windows Agent Arena: Evaluating multi-modal OS agents at scale. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 4874–4910. PMLR, 2025. URL https://proceedings.mlr.press/v2 67/bonatti25a.html. Horia Cristescu, Charles Park, Trong Canh Nguyen, Sergiu Talmacel, Alexandru-Gabriel Ilie, and Stefan Adam. UI-CUBE: Enterprise-grade computer use agent benchmarking beyond task accu- racy to operational reliability. arXiv preprint arXiv:2511.17131, 2025. doi: 10.48550/arXiv.251 1.17131. URL https://arxiv.org/abs/2511.17131. 10 Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, et al. Kimi-VL technical report. arXiv preprint arXiv:2504.07491, 2025. doi: 10.48550/arXiv.2504.07491. URL https://arxiv.org/ab s/2504.07491. Bradley Efron. Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1): 1–26, 1979. doi: 10.1214/aos/1176344552. URL https://doi.org/10.1214/aos/11 76344552. Tianyu Fu, Anyang Su, Chenxu Zhao, Hanning Wang, Minghui Wu, Zhe Yu, Fei Hu, Mingjia Shi, Wei Dong, Jiayao Wang, et al. Mano technical report. arXiv preprint arXiv:2509.17336, 2025. doi: 10.48550/arXiv.2509.17336. URL https://arxiv.org/abs/2509.17336. Gonzalo Gonzalez-Pumariega, Vincent Tu, Chih-Lun Lee, Jiachen Yang, Ang Li, and Xin Eric Wang. The unreasonable effectiveness of scaling agents for computer use. arXiv preprint arXiv:2510.02250, 2025. Google DeepMind. Gemma 4. https://deepmind.google/models/gemma/gemma-4/, 2026. Accessed: 2026-05-30. Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1.5-VL technical report. arXiv preprint arXiv:2505.07062, 2025a. doi: 10.48550/arXiv.2505.07062. URL https://arxiv.org/abs/2505.07062. Liangxuan Guo, Bin Zhu, Qingqian Tao, Kangning Liu, Xun Zhao, Xianzhe Qin, Jin Gao, and Guangfu Hao. Agentic lybic: Multi-agent execution system with tiered reasoning and orches- tration. arXiv preprint arXiv:2509.11067, 2025b. doi: 10.48550/arXiv.2509.11067. URL https://arxiv.org/abs/2509.11067. Linqiang Guo, Wei Liu, Yi Wen Heng, Tse-Hsun Peter Chen, and Yang Wang. Agent-sama: State-aware mobile assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, p. 29459–29467, 2026. doi: 10.1609/aaai.v40i35.40187. URL https: //ojs.aaai.org/index.php/AAAI/article/view/40187. Qijun Han, Haoqin Tu, Zijun Wang, Haoyue Dai, Yiyang Zhou, Nancy Lau, Alvaro A. Cardenas, Yuhui Xu, Ran Xu, Caiming Xiong, et al. VLAA-GUI: Knowing when to stop, recover, and search, a modular framework for gui automation. arXiv preprint arXiv:2604.21375, 2026. doi: 10.48550/arXiv.2604.21375. URL https://arxiv.org/abs/2604.21375. Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, and Jie Tang. ComputerRL: Scaling end-to-end online reinforcement learning for computer use agents. In The Fourteenth International Conference on Learning Representa- tions, 2026. URL https://iclr.c/virtual/2026/poster/10007435. Aiden Yiliu Li, Xinyue Hao, Shilong Liu, and Mengdi Wang.Avenir-web:Human- experience-imitating multimodal web agents with mixture of grounding experts. arXiv preprint arXiv:2602.02468, 2026a. doi: 10.48550/arXiv.2602.02468. URL https://arxiv.org/ab s/2602.02468. Weiming Li, Hye-young Paik, and Yulei Sui. HiSA: Hierarchical state abstraction for scalable GUI agents. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (eds.), Find- ings of the Association for Computational Linguistics: ACL 2026, p. 11965–11985, San Diego, California, United States, July 2026b. Association for Computational Linguistics. ISBN 979-8- 89176-395-1. doi: 10.18653/v1/2026.findings-acl.581. URL https://aclanthology.o rg/2026.findings-acl.581/. Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. MobileLLM: Optimizing sub-billion parameter language models for on-device use cases. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 32431–32454. PMLR, 2024. URL https: //proceedings.mlr.press/v235/liu24ce.html. 11 OpenAI. Computer-using agent, January 2025a. URL https://openai.com/index/compu ter-using-agent/. OpenAI. Introducing OpenAI o3 and o4-mini, April 2025b. URL https://openai.com/ind ex/introducing-o3-and-o4-mini/. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026a. URL https://qw en.ai/blog?id=qwen3.5. Qwen Team. Qwen3.6-27B: Flagship-level coding in a 27B dense model, April 2026b. URL http s://qwen.ai/blog?id=qwen3.6-27b. Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, Ran Xu, and Caiming Xiong. CoAct-1: Computer-using multi-agent system with coding actions. In The Fourteenth International Conference on Learning Represen- tations, 2026. URL https://iclr.c/virtual/2026/poster/10007725. Fei Tang, Zhiqiong Lu, Boxuan Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. Clawgui: A unified framework for training, evaluating, and deploying gui agents. arXiv preprint arXiv:2604.11784, 2026. doi: 10.48550/arXiv.2604.11784. URL https://arxiv. org/abs/2604.11784. Shizuo Tian, Hao Wen, Yuxuan Chen, Jiacheng Liu, Shanhui Zhao, Guohong Liu, Ju Ren, Yunxin Liu, and Yuanchun Li. Agentprog: Empowering long-horizon GUI agents with program-guided context management. In Proceedings of the 24th Annual International Conference on Mobile Systems, Applications and Services, p. 883–910. Association for Computing Machinery, 2026. doi: 10.1145/3745756.3809245. URL https://doi.org/10.1145/3745756.380924 5. Madhumitha Venkatesan, Shicheng Wen, Jiajing Guo, Jorge Piazentin Ono, Liu Ren, and Dongyu Liu. Plover: Steering GUI agents through plan-centric interaction. arXiv preprint arXiv:2607.15193, 2026. doi: 10.48550/arXiv.2607.15193. URL https://arxiv.or g/abs/2607.15193. Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, et al. UI-TARS-2 technical report: Advancing GUI agent with multi-turn reinforcement learning. arXiv preprint arXiv:2509.02544, 2025a. doi: 10.48550/arXiv.2509.02544. URL https://arxiv.org/abs/2509.02544. Taiyi Wang, Sian Gooding, Florian Hartmann, Oriana Riva, and Edward Grefenstette. A subgoal- driven framework for improving long-horizon LLM agents. arXiv preprint arXiv:2603.19685, 2026. doi: 10.48550/arXiv.2603.19685. URL https://arxiv.org/abs/2603.19685. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025b. doi: 10.48550/arXiv.2508.18265. URL https://arxiv.org/abs/2508.18265. Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongchan Shin, Martin Shin, Jiarui Hu, Yuyan Wang, Jixuan Chen, Yuxiao Ye, Danyang Zhang, Yipu Wang, Heng Wang, Diyi Yang, Victor Zhong, Y. Charles, Zhilin Yang, and Tao Yu. OpenCUA: Open foundations for computer-use agents. In Advances in Neural Information Processing Systems, volume 38, 2025c. doi: 10.52202/085713-4669. URL https://proceedings.neurips.c/paper_fil es/paper/2025/hash/c7ae529e945226b0d52ea4ac478c4f3-Abstract-Con ference.html. 12 Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. OS-ATLAS: Foundation action model for generalist GUI agents. In The Thirteenth International Conference on Learning Representa- tions, 2025. URL https://openreview.net/forum?id=n9PDaFNi8t. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Advances in Neural Information Processing Systems, volume 37, p. 52040–52094, 2024. doi: 10.52202/079017- 1650. URL https://mlanthology.org/neurips/2024/xie2024neurips-osworld/. Haiyang Xu, Xi Zhang, Haowei Liu, Junyang Wang, Zhaozai Zhu, Shengjie Zhou, Xuhao Hu, Feiyu Gao, Junjie Cao, Zihua Wang, et al. Mobile-Agent-v3.5: Multi-platform fundamental GUI agents. arXiv preprint arXiv:2602.16855, 2026a. doi: 10.48550/arXiv.2602.16855. URL https://arxiv.org/abs/2602.16855. Tianshi Xu, Huifeng Wen, and Meng Li. Adapting the interface, not the model: Runtime harness adaptation for deterministic LLM agents. arXiv preprint arXiv:2605.22166, 2026b. doi: 10.485 50/arXiv.2605.22166. URL https://arxiv.org/abs/2605.22166. Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous GUI inter- action. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 69772–69805. PMLR, 2025. URL https://proceedings.mlr.press/v267/xu25ae.html. Taofeng Xue, Chong Peng, Mianqiu Huang, Linsen Guo, Tiancheng Han, Haozhe Wang, Jianing Wang, Xiaocheng Zhang, Xin Yang, Dengchang Zhao, et al. EvoCUA: Evolving computer use agents via learning from scalable synthetic experience. arXiv preprint arXiv:2601.15876, 2026. doi: 10.48550/arXiv.2601.15876. URL https://arxiv.org/abs/2601.15876. Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. Powerinfer-2: Fast large language model inference on a smartphone. arXiv preprint arXiv:2406.06282, 2024. doi: 10.48550/arXiv.2406.06282. URL https://arxiv.org/abs/2406.06282. Bowen Yang, Kaiming Jin, Zhenyu Wu, Zhaoyang Liu, Qiushi Sun, Zehao Li, JingJing Xie, Zhoumi- anze Liu, Fangzhi Xu, Kanzhi Cheng, Yian Wang, Qingyun Li, Yu Qiao, Zun Wang, and Zichen Ding. OS-symphony: A holistic framework for robust and generalist computer-using agents. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 22300–22330, San Diego, California, United States, July 2026a. Association for Computational Linguistics. doi: 10.18653/v1/2026.acl- long.1021. URL https://aclanthology.org/2026.acl-long.1021/. Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, Ran Xu, Liyuan Pan, Caiming Xiong, and Junnan Li. GTA1: GUI test-time scaling agent. In The Fourteenth International Conference on Learning Represen- tations, 2026b. URL https://iclr.c/virtual/2026/poster/10011639. Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, Jie Xie, Wei Zhou, Wang Xu, Yuanheng Zhang, Zhou Su, Zhongwu Zhai, Xiaoming Liu, Yudong Mei, Jianming Xu, Hongyan Tian, Chongyi Wang, Chi Chen, Yuan Yao, Zhiyuan Liu, and Maosong Sun. AgentCPM-GUI: Building mobile-use agents with reinforcement fine-tuning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, p. 155–180, Suzhou, China, November 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.emnlp- demos.12. URL https: //aclanthology.org/2025.emnlp-demos.12/. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 62138–62160. PMLR, 2024. URL https://proceedings.mlr. press/v235/zhou24r.html. 13 APPENDIX ACONTROL-FAILURE ANNOTATION PROTOCOL This section presents the annotation protocol used for both GPT-5.6-sol annotation and human verifi- cation. Both settings follow the same evidence inputs, label definitions, and decision rules, ensuring consistent assignment of control-failure labels across trajectories. Control-Failure Annotation Guide A.1ANNOTATION OBJECTIVE AND SETUP Annotate unresolved cross-step control failures that remain at trajectory termination. Use the task instruction, complete execution logs, and available evaluator evidence to assess the full tra- jectory. Problems resolved during subsequent execution are excluded. When evaluator evidence conflicts with the execution record, follow the execution record. A.2LABEL DEFINITIONS The annotation set contains five control-failure labels, STAGE, ASSESS, LOOP, RECOVER, and STOP. Evaluate these labels independently, and record each applicable label at most once per trajectory. • STAGE records failure to maintain the task stage or its remaining requirements. It applies when execution moves to an incorrect stage, advances before the current stage is satisfied, returns to an earlier stage without justification, or omits a requirement needed to complete the current stage. • ASSESS records an incorrect assessment of the current execution state or action outcome. It applies when trajectory evidence indicates that an action failed or remained incomplete but the agent treats it as successful, or when a successful action is treated as failed. • LOOP records repetition of a materially equivalent action after evidence that the previous at- tempt made no progress, without new information, a relevant change in execution conditions, or an adjustment that could alter the outcome. • RECOVER records an ineffective response to a recognized failure or no-progress condition. It applies when a recovery, rollback, or alternative execution path does not address the observed problem and leaves execution unrecovered. • STOP records termination without sufficient evidence that the task has been completed. It applies when required task conditions remain unverified or unsatisfied and execution does not resume. • OTHER records a final unresolved failure that does not satisfy any of the five control-failure definitions. • NONE records a trajectory with no unresolved failure at termination. • ANY indicates that a trajectory contains at least one of the five control failures. OTHER and NONE are mutually exclusive with the five control-failure labels and with each other. A.3DECISION PROCEDURE AND BOUNDARY RULES 1. Review the complete trajectory and identify failures that remain unresolved at termination. 2. Evaluate STAGE, ASSESS, LOOP, RECOVER, and STOP independently, and record every applicable label. 3. Mark ANY if at least one control-failure label applies. If none applies, record OTHER when an unresolved failure remains, or NONE when no unresolved failure remains. 4. Assign multiple control-failure labels when the trajectory independently satisfies multiple definitions. For LOOP, an equivalent retry is labeled only when the previous attempt has explicit no-progress evidence. A retry that introduces new information, changes a relevant execution condition, or makes a corrective adjustment is not labeled as LOOP. For RECOVER, the failure or no-progress condition must first be recognized. The label applies when the subsequent response does not address that condition. ASSESS and STOP may co-occur when the agent incorrectly interprets the execution state and subsequently terminates without sufficient completion evidence. 14 Table 5: Inter-annotator agreement on 185 trajectories. A1 and A2 report label prevalence for the two indepen- dent human annotators. Agreement, F1, and Cohen’sκ are computed separately for each label. Label A1 prev. (%) A2 prev. (%) Agreement↑ (%) F1 (%)↑κ↑ STAGE20.020.588.672.00.649 ASSESS22.231.983.870.00.594 LOOP25.422.292.484.10.792 RECOVER28.118.487.072.10.641 STOP25.423.887.674.70.665 OTHER9.28.688.636.40.301 NONE40.537.387.083.30.727 ANY50.354.186.587.00.730 Table 6: Agreement between GPT-5.6-sol and the two independent human annotators on the same 185 trajec- tories. Precision, recall, F1, and Cohen’sκ are computed separately with each annotator as the reference. Annotator 1Annotator 2 LabelP (%)↑R (%)↑F1 (%)↑κ↑P (%)↑R (%)↑F1 (%)↑κ↑ STAGE68.470.369.30.61560.560.560.50.503 ASSESS69.873.271.40.63067.449.256.90.410 LOOP88.480.984.40.79574.478.076.20.692 RECOVER64.196.276.90.65239.791.255.40.400 STOP66.780.973.10.62752.668.259.40.445 OTHER63.682.471.80.68531.843.836.80.298 NONE93.658.772.10.59593.663.875.90.654 ANY77.696.886.10.68679.392.085.20.647 Macro74.079.975.70.66162.468.363.30.506 A.4HUMAN VERIFICATION We validate the GPT-5.6-sol annotations on a stratified sample of 37 OSWorld tasks shared across all five methods, yielding 185 trajectories. The tasks are sampled without replacement across OSWorld domains using random seed 42, with 10 multi-application tasks and 27 tasks from the nine single- domain strata. To prevent method identity from influencing annotation, we recode each trajectory and remove iden- tifying fields, including method names and result directories. Two PhD-level AI researchers inde- pendently annotate all 185 trajectories using the same label definitions and decision procedure. Each annotator receives the task instruction, complete execution logs, and evaluator evidence, without ac- cess to method identity or GPT-5.6-sol labels. The two annotators assign identical complete label sets to 61.1% of trajectories, with a mean Jaccard overlap of 70.1%. For ANY, they achieve 86.5% agreement, 87.0% F1, and Cohen’s κ = 0.730. Across the seven original labels, Cohen’s κ ranges from 0.301 for OTHER to 0.792 for LOOP, with a macro average of 0.624. Table 5 reports the complete label-wise inter-annotator agreement. We further compare the GPT-5.6-sol annotations with each human annotator on the same 185 trajec- tories. Table 6 reports precision, recall, F1, and Cohen’s κ for each label and for ANY, using each annotator independently as the reference. For ANY, GPT-5.6-sol achieves 86.1% F1 with κ = 0.686 against Annotator 1 and 85.2% F1 with κ = 0.647 against Annotator 2. The corresponding human–human comparison reaches 87.0% F1 with κ = 0.730. A.5COMPLETE ANNOTATION RESULTS Table 7 reports the complete task-level label distribution across all 369 OSWorld tasks under the fixed Qwen3.5-9B planner backbone. LocalLSTC has the lowest ANY incidence at 43.1% and the 15 Table 7: Task-level incidence of control-failure labels on OSWorld under the fixed Qwen3.5-9B planner back- bone. Each cell reports the percentage of all 369 tasks assigned the corresponding label. Bold marks the highest NONE incidence and the lowest ANY incidence. MethodSTAGE↓ ASSESS↓ LOOP↓ RECOVER↓ STOP↓ NONE↑ OTHER↓ ANY↓ HiSA32.5%5.7%20.3%46.3% 52.0%29.3%4.9% 65.9% CoAct-14.9%4.9%35.0%60.2%3.3%32.5%2.4% 65.0% GTA116.3%16.3%25.2%39.0% 16.3%29.3%14.6% 56.1% Agent S322.0%39.0%6.5%17.1% 39.0%35.8%13.8% 50.4% LocalLSTC9.8%18.7%28.5%35.0%21.1%42.3%14.6%43.1% Table 8: Grouped L2S and S2L ablation configurations under the fixed Qwen3.5-9B backbone. Full denotes the complete LocalLSTC configuration. ConfigurationFullw/o L2Sw/o S2L Planning contextPersistent g t , subgoal-aligned evi- dence, refined history, and recent ab- stracted steps Task-conditioned evidence and his- tory without a persistent subgoal in planner context Planner retains a subgoal proposal, while evidence context is task- conditioned Proposal and commitmentUp to three proposals compared within one inference, followed by an ordered action list Single proposal with one action per planner step Same as Full Execution channels GUI, Bash/Python, software API, and reserved control actions GUI, Bash/Python, and reserved con- trol actions Same as Full Step Abstraction Subgoal-conditioned evidence across execution channels Retained with task-level condition- ing Retained with task-level condition- ing State update and routingEvidence-conditioned continue, advance, error, stall, and finish routing Task-level error, stall, and finish han- dling without subgoal-based stage transitions Planner-intent state updates without S2L routing Recovery feedback State-conditioned error and stall feedback Retained at the task levelRemoved Context refinementEvidence-supported advance with maximum-interval fallback Periodic or forced fallback without subgoal advance Periodic or forced fallback without state routing Termination FinalVerificationafter TERMINATE, with rejection feed- back returned to planning Task-level Final Verification without subgoal context Termination without Final Verifica- tion highest NONE incidence at 42.3%, compared with an average ANY incidence of 59.4% across the four baselines. BGROUPED L2S–S2L ABLATION CONFIGURATIONS The grouped ablations isolate the two sides of the L2S–S2L execution boundary under the fixed Qwen3.5-9B backbone. w/o L2S removes subgoal-conditioned commitment formation and uses task-level single-action planning, while retaining task-level post-execution control and Step Ab- straction. w/o S2L retains the L2S commitment interface but replaces evidence-conditioned post- execution state updates with planner-intent updates. Table 8 specifies the corresponding configura- tions. CWINDOWSAGENTARENA RUNTIME DIAGNOSTICS Table 9 complements the OSWorld analysis in Table 2 with domain-level runtime statistics on Win- dowsAgentArena under the Qwen3.5-9B planner backbone. Across all 154 tasks, LocalLSTC averages 30.5 planner steps, 8.7 subgoals, and 2.8 context refine- ments per task. Final Verification rejects termination on 21.4% of tasks, and 51.5% of these tasks subsequently succeed after returning to execution, with an average of 5.2 additional planner steps. DIMPLEMENTATION DETAILS This section specifies how LocalLSTC instantiates the control flow in Section 3. Each planner call constructs a Short-Term Execution Commitment from the current observation and Long-Term Con- trol State. The runtime validates and executes the commitment, converts the resulting observations 16 Table 9:Domain-level runtime behavior of LocalLSTC with Qwen3.5-9B on WindowsAgentArena. Execution-channel columns report action shares. Pass, Fail, and Bypass denote accepted termination, rejected termination followed by recovery, and cutoff without verification. Recovery statistics are conditional on recov- ery entry. SummaryCommitmentExecution ChannelsEvidence Control Planning StructureChannel UseVerificationRecovery DomainTasks SR↑ (%) steps Subgoals / Task Actions / Subgoal Actions / step GUI (%) Bash (%) API (%) Context Refines Pass (%) Fail (%) Bypass (%) SR after↑ recovery (%) steps after recovery entry Chrome1764.117.84.55.01.3 92.90.50.01.270.623.55.975.02.7 Clock450.038.88.85.61.3 87.10.00.03.550.00.050.00.00.0 File Explorer1968.420.27.33.31.2 64.822.70.01.863.226.310.560.02.3 LibreOffice Calc240.041.310.97.31.9 95.11.30.04.216.78.375.00.00.0 LibreOffice Writer1910.543.513.25.01.5 90.52.80.04.65.315.878.933.333.0 Microsoft Paint366.736.74.012.31.3 93.60.00.02.033.333.333.3100.00.0 Microsoft Edge1330.824.26.64.31.2 93.11.50.02.253.815.430.80.00.0 Notepad2 100.07.53.03.31.3 65.411.50.00.00.0 100.00.0100.03.0 Settings5 100.04.82.61.60.9 72.40.00.00.4 100.00.00.00.00.0 VLC2143.732.110.23.81.2 79.311.40.03.028.628.642.933.34.5 VS Code2458.331.89.34.61.3 86.06.50.02.841.729.229.257.13.0 Windows Calculator366.745.36.713.11.9 87.37.60.02.366.733.30.0100.011.0 All15442.930.58.75.11.587.85.30.02.840.321.438.351.55.2 into subgoal-aligned evidence, and uses this evidence to update the persistent control state for the next interaction. D.1PROMPT CONSTRUCTION AND EXECUTION MODE Each planner call conditions on the task instruction, current screenshot, active subgoal, compact ev- idence history, runtime feedback, available action channels, and domain-specific operational guid- ance. Prompt construction adapts this context to the current execution mode. In GUI mode, the context emphasizes visual state, interface-operation guidance, and available software API schemas. In bashexecution mode, it emphasizes file-system state, textual execution evidence, and platform-specific operations. The runtime initializes the execution mode from task-level lexical cues and related-application pri- ors, then updates it according to the channels used during execution. GUI and software API actions select GUI for the next planner call, while Bash/Python actions select bash execution. For commitments spanning multiple channels, the final executed channel determines the subsequent mode. Planner context combines refined history with recent abstracted steps so that compressed cross- step evidence remains available together with recent execution detail. Runtime feedback records loop detections, no-change outcomes, tool failures, and rejected termination. Application-specific skills and software API schemas are selected from the foreground application and the task-level relatedapps set, allowing multi-application tasks to retain guidance for relevant domains. D.2PLANNER INTERFACE AND ACTION SPACE The planner returns a JSON object containing thought, subgoal, and actions.The subgoal field specifies the execution stage for the current commitment. The sentinel value continue retains the active subgoal, while any other value proposes a transition to a new sub- goal. The nonempty actions list contains the ordered actions to execute under that commitment and may combine GUI interaction, Bash/Python execution, and software API calls. The interface also provides the reserved actions WAIT, TERMINATE, and INFEASIBLE. WAIT retains the active subgoal while observing for an environment update or timeout. TERMINATE requests Final Verification after the preceding executable actions complete. INFEASIBLE records an infeasibility judgment. Terminal actions appear only at the end of a commitment, and validation rejects commitments containing both TERMINATE and INFEASIBLE. Table 10 summarizes the executable representation of each action type. The runtime identifies reserved control actions directly and distinguishes GUI interaction and soft- ware API calls from their registered prefixes. Remaining executable strings enter the Bash/Python channel, where the active platform skill determines whether they are interpreted as shell com- 17 Table 10: Action types exposed to the planner and their executable representations. Action typeExecutable representation GUI interactionExecutable pyautogui code string Software API callSchema-exposed Class.method(...) call string Bash executionA bash execution command or code string interpreted by the runtime as a Bash or Python snippet Event-driven waitReserved control action WAIT Completion requestReserved control action TERMINATE Infeasibility reportReserved control action INFEASIBLE mands or Python code. Candidate proposals remain internal to the planner inference, and only the actions list from the selected proposal is executed. D.3VALIDATION AND CANONICALIZATION Planner outputs are canonicalized before execution through list-level, channel-level, and schema- level validation. List validation enforces terminal-action placement and rejects commitments con- taining both TERMINATE and INFEASIBLE. Channel canonicalization removes wrapper artifacts from bash execution input, converts GUI scripts into executable pyautogui code, and con- structs stable fingerprints used by loop control. Software API validation restricts each action to a single callable expression defined by the exposed schema and implemented by a registered handler. Validated parameter literals are converted to runtime values before execution. Parsing or validation failures are returned to the planner together with the original response and the corresponding error message. The repair interface allows up to three attempts for each planner call before the configured runtime failure handling applies. D.4OBSERVATION AND STEP ABSTRACTION Step Abstraction evaluates each execution outcome against the intended effect of the current com- mitment and converts channel-specific observations into a common subgoal-aligned evidence repre- sentation. Each evidence record retains the executed action, observed result, task-relevant progress, execution exceptions, and relevant environment changes. For GUI interaction, planner-generated pyautogui code is divided into observation-aligned ex- ecution chunks. Actions requiring coordinate localization are grounded before execution unless executable coordinates are already available. After each chunk, the runtime waits for environment changes and passes the latest stable screenshot to subsequent execution. Screenshots acquired be- fore planning and after execution also undergo stability checks, with the latest available frame used when the timeout expires. To localize visual change, the runtime computes the pixel-wise absolute difference between pre- action and post-action screenshots and extracts the bounding box of changed pixels with a 50-pixel margin on each side. When no changed region is detected, Step Abstraction uses the full screenshot. The resulting visual transition is interpreted jointly with the task instruction, active subgoal, executed action, and available recovery feedback. For bash execution, the evidence record retains the executed command, execution status, exit code, textual output, and available readback evidence. Long outputs are compressed while preserv- ing task-relevant identifiers such as paths, filenames, URLs, IDs, table names, column names, and selected values. Software API evidence similarly retains the callable, execution status, return value, helper observation, and error message. GUI-mediated API execution additionally follows the visual observation policy above. All channels enter compact history through the same evidence interface. D.5STATE ASSIGNMENT AND EVIDENCE POLICY After Step Abstraction, the runtime assigns a transient routing state using a fixed precedence. A final TERMINATE request maps to finish. Execution failure maps to error, repeated no-progress evidence maps to stall, and a proposed subgoal different from the persistent subgoal maps to 18 advance. Remaining executions map to continue. State assignment occurs before a proposed subgoal transition is committed, keeping the resulting evidence aligned with the execution subgoal that produced it. No-change evidence is aggregated once per planner step, so a multi-action commitment contributes at most one no-change observation. Three consecutive no-change steps under the same subgoal produce stall. Steps containing bash execution are excluded from visual no-change aggre- gation because textual readback may establish progress without a corresponding interface change. Each stall increments the consecutive-stall counter, while any other route resets it. Three consec- utive stalls reach the runtime failure boundary. Loop detection compares normalized action and result fingerprints across interactions. GUI and software API fingerprints encode normalized executable requests, while bash execution fin- gerprints encode normalized commands or code. Result fingerprints represent GUI execution status, bashexecution status and textual output, or software API values and errors. Before execu- tion, the runtime suppresses repeated equivalent GUI commitments and repeated action–result pairs according to the thresholds in Table 11. The evidence record combines visual observations, execution logs, textual readback, software API returns, and abstracted steps. Textual readback may come from command output or subsequent file inspection. GUI context remains available across text-based actions, allowing confirmed textual evidence to support subsequent planning and Final Verification together with visual evidence. D.6CONTEXT REFINEMENT LocalLSTC extends the context-refinement mechanism of HiSA (Li et al., 2026b) with execution- state triggering. An evidence-supported advance makes completed subgoal evidence eligible for refinement once sufficient new evidence has accumulated. The first refinement summarizes the available abstracted history, while later refinements incorporate only steps not already represented in the existing summary. A maximum refinement interval provides a fallback for long trajectories without an earlier trigger. Subsequent planner calls and Final Verification receive the refined summary together with recent uncompressed steps. This preserves compact cross-step context while retaining local evidence from the most recent interactions. D.7FINAL VERIFICATION AND EVIDENCE REFRESH A TERMINATE request invokes Final Verification using the task instruction, active subgoal, com- pact execution history, initial screenshot, and latest screenshot. When refined context is available, the verifier combines the summary with abstracted steps not yet represented in it. Otherwise, veri- fication uses a recent sliding window. For office-style tasks modified through bash execution, the runtime may reopen the target file before verification so that the current interface provides re- freshed visual evidence. The verifier returns PASS or FAIL, with invalid responses mapped to FAIL. A PASS confirms termination. A FAIL records the rejected termination and returns execution to planning with feed- back about the remaining completion conditions. The latest screenshot and corresponding verifier feedback are retained as recovery evidence for the subsequent planner call. D.8RUNTIME PARAMETERS Table 11 reports the runtime parameters shared across LocalLSTC experiments. Planner sampling uses temperature, top p, and topk, while the remaining parameters govern proposal genera- tion, context refinement, execution timeouts, observation stability, repair, and runtime control. D.9MODEL RESPONSIBILITIES AND ACCOUNTING LocalLSTC uses two model roles during execution. The planner backbone produces candidate pro- posals and the selected execution commitment, and the same backbone performs Step Abstraction, 19 Table 11: Runtime parameters shared across LocalLSTC experiments. Rule or hyperparameterValue Model-inference temperature0 Model-inference topp0.95 Model-inference top k20 Maximum candidate proposals per inference step3 Minimum context-refinement interval5 steps Maximum context-refinement interval20 steps bashexecution timeout180 seconds Maximum planner parsing or repair attempts3 Post-action observation timeout10 seconds Explicit WAIT timeout20 seconds Screenshot acquisition timeout6 seconds Evaluation retry timeout25 seconds Initial screenshot stabilization5 low-change observa- tions Post-action screenshot stability check2 stable frames Step Abstraction ROI margin50 pixels Planner-side repeated GUI decision threshold3 consecutive planner calls Repeated GUI action–result threshold5 repetitions Repeated bash execution or software API action–result threshold 3 repetitions No-change threshold under the same subgoal3 consecutive abstracted steps Consecutive stall-to-fail threshold3 stalls context refinement, and Final Verification under dedicated prompts. The visual grounder localizes GUI actions when executable coordinates are unavailable. Output parsing, schema validation, action fingerprinting, timeout handling, state assignment, and routing are deterministic runtime operations. GPT-5.6-sol is used only for post-hoc control-failure annotation and does not participate in agent execution. Token accounting includes planner and visual-grounding calls when the corresponding model invocation is triggered. D.10EXECUTION BOUNDARY All benchmark actions execute inside the benchmark-defined virtual machine. Planner outputs are validated before execution, software API calls are restricted to schema-exposed handlers, and GUI and Bash/Python actions operate within the permissions of the benchmark environment. EPROMPT TEMPLATES This appendix reports the prompt templates used by LocalLSTC. The blocks use the paper-facing api action name for software API calls. E.1PLANNER SYSTEM PROMPT This prompt corresponds to GLOBAL PLANNERPROMPT. System Prompt: Global Planner You are an expert in GUI interaction, execution-side automation, and software-level API tools. Always keep the task instruction in mind. # General Instructions ... # Action Types 20 ## api Call one software-level API exposed for one of the current related apps. Use this when the task maps cleanly to an app-specific software operation. Represent it as a bare ‘Class.method(...)‘ action string. Use only methods explicitly listed in the current api tools section. Never invent method names. Use Python literals such as ‘True‘, ‘False ‘, and ‘None‘, not JSON/JavaScript literals like ‘true‘, ‘false‘, or ‘ null‘. ## gui_action Execute pyautogui code against the visible GUI. Use this when the task should be completed through visible interface interactions. Represent it as a bare ‘pyautogui...‘ action string. - Never use ‘pyautogui.hotkey(’alt’, ’f4’)‘ or any Alt+F4 variant to close a window. On Windows this can open the Shut Down Windows dialog and can terminate the VM session. Use a visible in-app close/cancel button, ‘pyautogui.press(’esc’)‘ for dialogs, or execution-side app- specific cleanup instead. ## wait Wait for async operations to complete and observe UI changes. Represent it as the exact action string ‘WAIT‘. ## bash_execution Execute execution-side automation through the ‘bash_execution‘ tool. This is a historical tool name: on Linux, output Bash commands or Python scripts; on Windows, output Python code snippets executed in the Windows guest. Use this when file-side editing, inspection, or automation is more reliable than GUI interaction. Follow the platform- specific skill section for paths, commands, and examples. Represent it as a bare command/code action string. ## infeasible Declare that the task is objectively impossible to complete. Represent it as the exact action string ‘INFEASIBLE‘. - Some tasks may be infeasible by design. If required capabilities, variables, or app features are unavailable, use ‘INFEASIBLE‘ instead of ‘TERMINATE‘ and explain the blocker in ‘thought‘. - Treat explicit app, method, source, target, format, variable, and final-state constraints as required. Do not satisfy a similar task through a workaround and claim completion. - Do not infer missing required parameters; verify them if possible, otherwise use ‘INFEASIBLE‘. E.2PLANNER RESPONSE FORMAT PROMPT This prompt corresponds to PLANNER RESPONSEFORMATPROMPT. System Prompt: Planner Response Format # Response Format You may think through up to 3 candidate strategies before the final answer, but the executable answer must be only the final JSON object. List only the strongest plausible candidates. Do not invent weak filler strategies just to reach 3 items. Use this exact structure: Candidates: 1. <one-sentence strategy> [optional] 2. <one-sentence strategy> [optional] 3. <one-sentence strategy> 21 Best: <candidate number> Why: <one-sentence selection reason> ‘json "thought": "Brief reasoning about the chosen strategy and current action. Check prerequisites and verify previous result.", "subgoal": "Meaningful phase-level objective for the current stage of work, or ’continue’ to keep the current one", "actions": [ "pyautogui.click(695, 514) # click the radio button next to Microsoft Bing", "pyautogui.click(663, 807) # click Set as default button" ] ‘ Field guide: - The final JSON object is the only executable output. Everything before it is optional planning text. - Output 1 to 3 candidates, not always 3. - Every candidate must be realistic and worth considering for this exact state. - If one approach is clearly dominant, output just 1 candidate. - If you list candidates, keep them short and plain text only. Do not put JSON, code blocks, or extra headings inside candidate text. - End your response with exactly one final JSON object. - The final JSON object must contain ‘thought‘, ‘subgoal‘, and ‘ actions‘. - ‘subgoal‘: a phase-level objective for a coherent stage, not an action title or method. Avoid labels like "Open menu", "Click link", " Scroll to inspect", or "Run script"; prefer "Navigate to the target page", "Update the spreadsheet", or "Verify the result". - Use ‘continue‘ while the next action still pursues, verifies, or strengthens evidence for the subgoal. Start a new subgoal only when the target state changes or the current stage is blocked. - ‘actions‘ must be a non-empty ordered list. The environment changes after each action, so plan them in execution order. - Prefer one meaningful interaction per action item. Even though some action strings can contain multiple low-level operations, you should usually split sequential interactions into separate ‘actions‘ items for clarity and better replanning. - Each action item should be one of: - a bare ‘pyautogui...‘ string for GUI actions - a bare ‘Class.method(...)‘ string for API calls - a bare command/code string for ‘bash_execution‘ - the string ‘WAIT‘ for ‘wait‘ - the string ‘TERMINATE‘ only when the task is already verified complete, with the blocker reason explained in ‘thought‘ - the string ‘INFEASIBLE‘ only when the task is objectively impossible, with the blocker reason explained in ‘thought‘ - Use exactly one API call per action item. - Bash example: ‘json "thought": "Locate the exact workbook before editing it.", "subgoal": "Locate the target workbook", "actions": [ 22 "find /home/user -type f -name ’ * .xlsx’ 2>/dev/null | head -20" ] ‘ - Good example: Candidates: 1. Use a short GUI action sequence to open the search engine menu and set Bing there. 2. Use one direct browser API call if an API exists. Best: 1 Why: The relevant setting is visible in the UI and there is no confirmed API call for this browser preference. ‘json "thought": "Open the visible browser setting and choose Bing through the interface.", "subgoal": "Set Bing as the default search engine", "actions": [ "pyautogui.click(695, 514) # click the radio button next to Microsoft Bing", "pyautogui.click(663, 807) # click Set as default button" ] ‘ - Termination example: ‘json "thought": "The required result is already visible and verified in the current screenshot.", "subgoal": "Verify task completion", "actions": [ "TERMINATE" ] ‘ - Infeasible example: ‘json "thought": "The requested result cannot be completed because the required app or capability is unavailable in the current environment .", "subgoal": "Confirm the blocker", "actions": [ "INFEASIBLE" ] ‘ E.3REPAIR PROMPT This prompt corresponds to FIXRESPONSEPROMPT. 23 System Prompt: Response Repair Repair the planner response below and preserve the intended plan. Repair rules: - Focus on fixing the concrete parse/schema/tool-syntax error shown above. - If the error points to a specific line, column, or token, fix that exact location. - Return exactly one final JSON object with ‘thought‘, ‘subgoal‘, and ‘actions‘. - Each ‘actions‘ item must be a bare string or number, not an object. - A GUI action must be a valid Python ‘pyautogui‘ code string. - A software API action must be exactly one Python call expression such as ‘GoogleDriveTools.upload_file(...)‘. - Return exactly one repaired response. - The final output must satisfy this response format exactly: response_format_prompt Original response: response Parse traceback: error_message E.4STEP ABSTRACTION PROMPT This prompt corresponds to STEPABSTRACTIONPROMPT. System Prompt: Step Abstraction Compare the latest observations and summarize what happened in 1-2 concise sentences. Task instruction: task_instruction Current subgoal: current_subgoal Action: action_description Recovery hint: recovery_hint Rules: - Describe what changed, or say no visible change. - Mention clear errors if shown. - Do not judge subgoal status or long-horizon task completion. - Keep the summary concise and concrete. - For bash/API output, preserve exact task-relevant identifiers from the output, especially paths, filenames, URLs, IDs, sheet/table/column names, window titles, counts, and selected values. - Use the task instruction and subgoal to decide which output details are key; do not replace needed paths or IDs with generic wording. E.5FINAL VERIFICATION PROMPT This prompt corresponds to FINALVERIFICATIONPROMPT. System Prompt: Final Verification Decide whether the task is fully completed after the observed workflow . 24 You will receive: - the original task - the subgoal - the full execution history - the initial screenshot from the start of the task - the latest screenshot from the end of the task Return exactly one word: PASS or FAIL Rules: - Return only ‘PASS‘ or ‘FAIL‘. - Use ‘PASS‘ only if the task requirements appear fully satisfied in the full execution history and the provided screenshots. - Some tasks may be infeasible by design. If required capabilities, variables, or app features were unavailable, return ‘FAIL‘ rather than accepting a workaround. - If there is uncertainty, return ‘FAIL‘. - The screenshots are primary evidence for visible state. Use the initial screenshot as baseline context and the latest screenshot as the final state to judge. - If the task outcome is only obvious by comparing before vs after, explicitly use that comparison before deciding. - If the latest screenshot does not directly show the requested result , return ‘FAIL‘. - Use the full execution history to confirm what was actually modified , saved, read back, or verified. If logs only show that a command or API call ran, but do not confirm the requested final result, return ‘ FAIL‘. - If a file was modified through ‘bash_execution‘ while the desktop app may still display stale content, prefer explicit readback/ verification evidence from the logs plus the latest screenshot of the reopened app state. - Fail if the logs reveal formatting mistakes, header corruption, partial coverage, or any mismatch with the task requirements. - Do not treat an explanation of impossibility, a command launch, or a transient status/toast as completion for a task that asked for an actual GUI, file, or configuration result. - Require exact evidence for explicit app, method, source, target, format, variable, and final-state constraints; return ‘FAIL‘ if the workflow guessed, changed, or bypassed them. - For multi-target tasks (‘all‘, ‘both‘, ‘each‘, ‘respectively‘), fail unless every requested target is explicitly covered by the logs. - For relative-date tasks, fail unless the exact resolved absolute date is explicitly covered. E.6CONTEXT REFINEMENT PROMPT This prompt corresponds to CONTEXTREFINEMENTPROMPT. System Prompt: Context Refinement Analyze task execution progress and provide guidance. You will receive: a task instruction, execution history range and execution history Instructions: 25 - Summarize the full execution history into one unified summary covering the entire range - List what was done in order (successes and failures) - ** IMPORTANT ** : Preserve coordinates in click actions (e.g., "click (500,300)") - these can be reused later - Identify if we’re stuck in loops, making progress, or stalled - Provide actionable suggestions for the next step if there are issues - If ‘bash_execution‘ or software API already verified the requested file/data state, do not suggest GUI Save, reopen, reload, or refresh just to sync a stale window; suggest termination or another text-level verification instead. Return a concise summary string in this format: Steps X ̃Y: [ordered list of what was done, keeping coordinates]. Suggestion: [actionable advice, or ’Continue’ if progressing well] Examples: - Steps 1 ̃5: Opened file, tried to edit (failed 3 times with permission error), attempted sudo (failed). Suggestion: Try a different approach - copy file to temp location first. - Steps 1 ̃5: Clicked Submit button at click(850,620), typed text, clicked Save at click(920,580). Suggestion: Continue - forms being filled correctly. - Steps 1 ̃10: Previously installed package and ran script (steps 1 ̃5). Then verified output, tested functionality (steps 6 ̃10). Suggestion: Continue - good progress. - Steps 1 ̃15: Clicked the same button 5 times with no response, tried alternative buttons (failed). Suggestion: This approach isn’t working - try an alternative method or termination as infeasible. FEXAMPLES OF RUNTIME CONTROL The runtime-event analysis in Table 4 shows how LocalLSTC behaves after repetition, recovery entry, and rejected termination. The six successful OSWorld trajectories below illustrate how persis- tent state and execution evidence guide subsequent commitments after these events. The examples are summarized from the corresponding benchmark task configurations, execution log.json, and modeltrace.txt. F.1RECOVERY AFTER REPEATED ACTIONS Task 01b269ae-2111-4a07-81fd-3fcd711993b0 requires filling blank cells in B1:E30 with the value from the cell above. Three equivalent file-editing commands produce no visible spreadsheet change. File readback then localizes the unresolved cells to column B while retain- ing the original range and the constraint against modifying unrelated cells. Under this evidence, L2S revises the execution command, reloads Calc, and verifies column B separately from columns C:E. Final Verification initially rejects termination because the visual evidence is insufficient. The runtime subsequently refreshes the spreadsheet, verifies the target range, and accepts completion. Task 4127319a-8b79-4410-b58a-7a151e15f3d7 requires recursively counting lines in all PHP files under the current directory and displaying the result in a terminal. The Bash channel repeatedly returns the same 54-line result without producing the required visible terminal evidence. The active subgoal retains both the counting scope and the display requirement. L2S then switches execution to the GUI, opens a terminal, and enters the command there. Final Verification accepts termination once the visible terminal establishes both the command and its result. F.2STAGE-PRESERVING RECOVERY ACROSS EXECUTION CHANNELS Task 3c8f201a-009d-4bbe-8b65-a6f8b35b57f requires downloading an image and exporting a compressed copy as compressed.jpeg on the desktop. The trajectory completes 26 the download through Bash and performs the export through the GUI, but repeated GUI attempts produce incorrect filenames. File readback identifies the candidate files and their sizes. The com- pression subgoal remains active while L2S switches back to Bash, retains the valid 115 KB file, and normalizes its filename. The resulting evidence establishes the required location, filename, and size without restarting the completed stages. Task c867c42d-a52d-4a24-8ae3-f75d256b5618 requires exporting the Personal Ad- dress Book from Thunderbird and converting the resulting CSV to XLSX in LibreOffice Calc. GUI export produces Personal Address Book.csv, and file readback reveals that the filename does not match the task requirement. S2L retains the completed export evidence while keeping the conversion stage active. L2S then uses Bash to rename the file to contacts.csv, completes the conversion in Calc, and confirms through Bash readback that contacts.xlsx exists on the desktop. F.3EVIDENCE-GROUNDED TERMINATION Task 1e8df695-bd1b-45b3-b557-e7d599cf7597 requires adding a Profit column beside Sales and COGS and computing the weekly difference. After initial numeric-type and file-state errors, a corrected script completes the edit, and readback confirms the column name and sample calculations. Final Verification rejects the first termination request because visual confirmation is still unavailable. L2S then resolves the Calc recovery dialog and verifies the Profit column and its values in the visible spreadsheet. Final Verification accepts the subsequent termination request once the required visual evidence is available. Task 6ada715d-3aae-4a32-a6a7-429b2e43fb93 requires inserting 1.png from the desktop at the current cursor position in Writer. The first termination request is rejected because the available evidence does not establish the insertion result. L2S retains the image-insertion subgoal and inspects both the inserted image and its document position. A second termination request is rejected after a context menu obscures the relevant content. After subsequent interaction exposes the required visual evidence, Final Verification accepts completion. 27

Instant research discovery