Paper deep dive
From Plan to Action: How Well Do Agents Follow the Plan?
Shuyang Liu, Saman Dehghan, Jatin Ganhotra, Martin Hirzel, Reyhaneh Jabbarvand
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 4/15/2026, 1:33:43 AM
Summary
This paper presents a systematic, large-scale analysis of plan compliance in programming agents, evaluating 16,991 trajectories across four LLMs (GPT-5 mini, DeepSeek-V3, DeepSeek-R1, Devstral-small) on SWE-bench Verified and Pro. The study introduces novel metrics—Plan Phase Compliance, Plan Order Compliance, and Plan Phase Fidelity—to measure how well agents adhere to instructed workflows. Findings indicate that while standard plans improve issue resolution, agents often deviate based on task difficulty and internal training biases. The research highlights that periodic reminders improve compliance, while subpar or misaligned plans can degrade performance, suggesting a need for fine-tuning paradigms that teach adaptive reasoning rather than static workflow memorization.
Entities (5)
Relation Signals (3)
SWE-Agent → evaluatedon → SWE-bench Verified
confidence 100% · examining 16,991 trajectories from SWE-agent across four LLMs on SWE-bench Verified
Periodic Plan Reminders → improves → Plan Compliance
confidence 95% · we observe that periodic plan reminders can mitigate plan violations and improve task success.
Plan Compliance → measures → Agent Trajectory
confidence 90% · Assessing whether an agent follows the instructed plan for a task requires process-centric analysis of trajectories.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Agents aspire to eliminate the need for task-specific prompt crafting through autonomous reason-act-observe loops. Still, they are commonly instructed to follow a task-specific plan for guidance, e.g., to resolve software issues following phases for navigation, reproduction, patch, and validation. Unfortunately, it is unknown to what extent agents actually follow such instructed plans. Without such an analysis, determining the extent agents comply with a given plan, it is impossible to assess whether a solution was reached through correct strategic reasoning or through other means, e.g., data contamination or overfitting to a benchmark. This paper presents the first extensive, systematic analysis of plan compliance in programming agents, examining 16,991 trajectories from SWE-agent across four LLMs on SWE-bench Verified and SWE-bench Pro under eight plan variations. Without an explicit plan, agents fall back on workflows internalized during training, which are often incomplete, overfit, or inconsistently applied. Providing the standard plan improves issue resolution, and we observe that periodic plan reminders can mitigate plan violations and improve task success. A subpar plan hurts performance even more than no plan at all. Surprisingly, augmenting a plan with additional task-relevant phases in the early stage can degrade performance, particularly when these phases do not align with the model's internal problem-solving strategy. These findings highlight a research gap: fine-tuning paradigms that teach models to follow instructed plans, rather than encoding task-specific plans in them. This requires teaching models to reason and act adaptively, rather than memorizing workflows.
Tags
Links
- Source: https://arxiv.org/abs/2604.12147v1
- Canonical: https://arxiv.org/abs/2604.12147v1
Trouble viewing inline? Open PDF directly →
Full Text
62,636 characters extracted from source content.
Expand or collapse full text
From Plan to Action: How Well Do Agents Follow the Plan? Shuyang Liu ∗ University of Illinois Urbana–Champaign USA sl225@illinois.edu Saman Dehghan ∗ University of Illinois Urbana–Champaign USA samand2@illinois.edu Jatin Ganhotra IBM USA jatinganhotra@us.ibm.com Martin Hirzel IBM USA hirzel@us.ibm.com Reyhaneh Jabbarvand University of Illinois Urbana–Champaign USA reyhaneh@illinois.edu Abstract Agents aspire to eliminate the need for task-specific prompt crafting through autonomous reason-act-observe loops. Still, they are com- monly instructed to follow a task-specific plan for guidance, e.g., to resolve software issues following phases for navigation, reproduc- tion, patch, and validation. Unfortunately, it is unknown to what extent agents actually follow such instructed plans. Without such an analysis—determining the extent agents comply with a given plan—it is impossible to assess whether a solution was reached through correct strategic reasoning or through other means, e.g., data contamination or overfitting to a benchmark. This paper presents the first extensive, systematic analysis of plan compliance in programming agents, examining 16,991 trajecto- ries from SWE-agent across four LLMs on SWE-bench Verified and SWE-bench Pro under eight plan variations. Without an explicit plan, agents fall back on workflows internalized during training, which are often incomplete, overfit, or inconsistently applied. Pro- viding the standard plan improves issue resolution, and we observe that periodic plan reminders can mitigate plan violations and im- prove task success. A subpar plan hurts performance even more than no plan at all. Surprisingly, augmenting a plan with additional task-relevant phases in the early stage can degrade performance, particularly when these phases do not align with the model’s inter- nal problem-solving strategy. These findings highlight a research gap: fine-tuning paradigms that teach models to follow instructed plans, rather than encoding task-specific plans in them. This re- quires teaching models to reason and act adaptively, rather than memorizing workflows. Keywords Programming Agents, Process-centric Analysis, Agent Planning ∗ Both authors contributed equally to this work. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference’17, July 2017, Washington, DC, USA © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-x-x-x/Y/M https://doi.org/10.1145/n.n 1 Introduction Large Language Model (LLM)-based agents have emerged as a promising paradigm for automating software engineering tasks, from code synthesis and translation to end-to-end issue resolu- tion [32,34,37]. Central to these systems is the use of structured instructions, a.k.a. a plan, which decomposes a high-level objective of a given task into an ordered sequence of steps that the agent can follow to accomplish the task successfully. In theory, a plan can help reduce cognitive load for reasoning about future steps at the local reason-act-observe loop [35]. As a result, planning has become a common design choice in agentic frameworks, usually encoded as step-by-step instructions in the system prompt [1,25,31]. For example, a plan for fixing GitHub issues will instruct the agent to navigate to a potential bug location (based on the issue description), reproduce the bug to ensure correct localization, patch the bug, and validate the patch’s correctness. In practice, the plan is only advisory: it is included in the system prompt, but the scaffold’s execution engine provides no mechanism to enforce it. At each trajectory step, the model performs local rea- soning over its current context, and its actions may or may not align with the plan. As the trajectory grows and the context fills with error messages, file contents, and prior reasoning, the plan’s influ- ence may diminish, consistent with the known limitations of LLMs in attending to earlier context [17]. Therefore, the true effectiveness of the instructed plans and whether the agent’s trajectory complies with the plan remain open questions. Evaluating plan compliance can also serve as an important process-centric [18] analysis; it can reveal whether the agent accomplishes a task through correct strate- gic reasoning or through overfitting to benchmark trajectories or data contamination. To shed light on the above-listed open questions, this paper presents a large-scale, systematic, and fully automated evaluation of plan compliance in programming agents. The analysis leverages a novel plan compliance metric, measured across three dimensions: Plan Phase Compliance, Plan Order Compliance, and Plan Phase Fidelity (§2). We evaluate 16,991 SWE-agent trajectories, generated to resolve instances of two popular benchmarks (SWE-bench Veri- fied [4] and SWE-bench Pro [6]), using four backbone LLMs (GPT-5 mini, DeepSeek-V3, DeepSeek-R1, and Devstral-small), under eight plan settings: the standard navigate-reproduce-patch-validation arXiv:2604.12147v1 [cs.SE] 13 Apr 2026 Conference’17, July 2017, Washington, DC, USAShuyang Liu, Saman Dehghan, Jatin Ganhotra, Martin Hirzel, and Reyhaneh Jabbarvand plan, no specified plan, and six variations of the standard plan, ob- tained by removing, adding, re-ordering, and repeating plan phases. Our study answers the following research questions: •RQ1: Standard Plan Compliance (§4.1). To what extent do agents follow the instructed plan? What factors impact plan compliance and violations? Does plan compliance help agents resolve issues? Findings. Agents follow the standard plan, although with vary- ing compliance rates. Some strictly follow the plan in the specified order, while others adaptively override the plan based on the tra- jectory, depending on the problem’s difficulty. Following the plan positively helps all agents resolve more GitHub issues. The fine- tuning paradigm, context window pressure, data contamination, overfitting, and optimizing for short-term reward are the most prevalent factors impacting plan compliance. •RQ2: Behavior of Agents in the Absence of Plan (§4.2). How do agents operate in the absence of a plan? To what extent does removing the plan impact overall performance? Findings. Without a plan, agents follow their internalized prob- lem-solving strategy, which overlaps with the standard plan to a varying degree. The success rate, however, drops in the absence of the standard plan. •RQ3–RQ5: Impact of Plan Variations (§5.1–5.3). Do agents heed removal, addition, and reordering of plan phases? To what extent does frequent reminding of the plan phases help compli- ance in the long-horizon task of program repair? Findings. Removing a standard plan phase, even if the agent usually ignores it under the standard plan setting, negatively impacts the agents’ performance, confirming the overall impact of a global plan on local reasoning steps. The negative impact of a bad plan is greater than no plan at all. Surprisingly, augmenting plans with task-relevant phases inspired by best practices also negatively affects agents’ performance when they are not aligned with the model’s internal strategy. Periodic plan reminders reduce plan violations and improve performance. •RQ6: Generalization to Other Benchmarks (§6.1). How much can the observations and conclusions about plan compliance generalize to another benchmark, i.e., SWE-bench Pro [6]? Findings. The plan compliance rate of the agents across all settings on SWE-bench Pro drops by 13%, on average, compared to SWE-bench Verified. The agents exhibit different phase flow patterns, e.g., they give up on generating reproduction tests early and validate patches using existing regression tests rather than generating tests. This is likely because SWE-bench Pro instances are more challenging and less contaminated, and the high-level standard plan is no longer effective at guiding the agents. •RQ7: Impact of Nondeterminism (§6.2). To what extent is plan compliance of agents under different plan settings attributed to nondeterminism? Findings. Nondeterminism exists but does not impact our find- ings. We account for nondeterminism by repeating experiments and comparing persistent behaviors across plan settings. We are the first to (1) conduct a large-scale, systematic analysis of plan compliance by agents, (2) introduce novel plan compliance metrics, (3) speculate the root causes of plan violations, and (4) study how plan compliance and task success relate to each other. Our findings suggest that the effectiveness of the plan is tightly coupled to its alignment with the model’s internalized workflow and the task’s complexity. Therefore, future research should focus on fine- tuning paradigms that teach models to follow plans more effectively, rather than encoding task-specific plans into them. 2 Experimental Design We aim to analyze whether and to what extent programming agents follow the specified, task-appropriate software engineering work- flows. Given the popularity of programming agents for fixing real- world GitHub issues, this study will focus on program repair. The default practical workflow for this task involves localizing the bug, patching the code, and then validating whether the patch resolved the bug. Many existing scaffolds, e.g., SWE-agent/mini-SWE-agent, Trae agent, and OpenHands, explicitly instruct the agent to follow a similar plan in their system prompt 1 [1, 25, 31]: •Navigation (N). The agent searches for, opens, and reads files relevant to the issue description, building an understanding of the codebase and localizing the relevant components. •Reproduction (R). The agent generates new tests to reproduce the bug, i.e., tests that fail on the buggy code. • Patch (P). The agent edits the application code to fix the bug. •Validation (V). The agent runs reproduction tests and generates new tests to validate patch correctness. Assessing whether an agent follows the instructed plan for a task requires process-centric analysis of trajectories. We build our process-centric analysis on top of Graphectory and Langutory [18]. Graphectory represents linear raw trajecto- ries as enriched graph structures, where nodes are the agent’s distinct actions and edges denote the chronological execution order. Langutory is an abstract representation of the trajectory in the form of language. That is, by mapping the agent’s action through the sequence of푛trajectory steps푇= (푠 1 , . . .,푠 푛 )to an alphabet Φ= 푝 1 , . . .,푝 푚 of푚letters 2 , LangutoryL(푇,Φ)explains the agent’s problem-solving strategy as a sequence of letters. When letters denote plan phases, Langutory translates trajec- tory actions into a sequence of plan phases. Considering a plan phase alphabetΦand expected plan phase sequenceL ★ (Φ), we determine a Langutory complies with the instructed plan if it includes all and only specified plan phases in the specified order. We propose a novel process-centric metric, plan compliance (푃퐶), measured across three dimensions: Plan Phase Compliance (푃퐶), Plan Order Compliance (푃푂퐶), and Plan Phase Fidelity (푃퐹 ). To illustrate the concept, Figure 1 shows three trajectories gener- ated by SWE-agent along with their corresponding Graphectory and Langutory. Figure 1a shows a compliant and successful ex- ecution. SWE-agent DSK-R1 starts by navigating to the buggy file nanops.py(step 1), creates and executes a reproducing test (steps 2–3), edits the buggy file (step 4), validates the patch by creating and executing a more comprehensive test (steps 5–7), edits the file again to handle corner cases (step 8), and re-executes the test (step 9) be- fore submitting the patch. This yields a Langutory of푁푅 2 푃푉 3 푃푉, which is compliant with the instructed planL ★ (Φ)=NRPV. 1 The specified plan for some agents could be more verbose. Regardless, all existing agents follow similar high-level plans. 2 푚 ≪ 푛 to show an overall strategy rather than detailed actions. Conference’17, July 2017, Washington, DC, USA 1 view nanops.py create test_min_count.py 2 str_replace nanops.py str_1 str_2 observation: edit succeeds 4 8 10 submit (a)(DSK-R1 · xarray-4356 · Resolved) Navigation (N) Patch (P) Validation (V) General Reproduction (R) python test_min_count.py 3 python test_min_count.py 5 create test_min_count_comprehensive.py 6 python test_min_count_comprehensive.py 7 str_replace nanops.py str_3 str_4 observation: edit succeeds python test_min_count_comprehensive.py 9 nanops.py view test_min_count.py create nanops.py <old_str1, new_str2> submit 1 3 5 str_replace ✔ python 2 4 test_min_count.py create test_min_count_comprehensive.py python 6 nanops.py <old_str3, new_str4> str_replace ✔ 7 8 9 Langutory: NRRPVVVPV (NR2PV3PV) PPC=1, POC=1, PPF=1 PC=1 5 view dates.py create reproduce_bug.py 1 str_replace dates.py str_3 str_4 observation: edit fails 10 create test_fix.py python test_fix.py python reproduce_bug.py 2 str_replace reproduce_bug.py str_1 str_2 observation: edit succeeds 3 python reproduce_bug.py 4 6 view dates.py --view_range 607 638 7 view dates.py --view_range 594 602 8 str_replace dates.py str_5 str_6 observation: edit succeeds 9 11 12 submit create 1 reproduce_bug.py python reproduce_bug.py <old_str1, new_str2> str_replace ✔ 2 3 dates.py view 4 dates.py [607, 638] view dates.py [594, 602] view dates.py <old_str3, new_str4> str_replace ❌ dates.py <old_str5, new_str6> str_replace ✔ 5 create test_fix.py python test_fix.py 6 7 8 9 10 11 submit 1 view /testbed 2 view /testbed/django/contrib/auth 3 view /testbed/django/contrib/auth/token.py str_replace token.py str_1 str_2 observation: Edit succeeds 4 submit 5 /testbed view /testbed/django/ contrib/auth view /testbed/django/co ntrib/auth/token.py view 1 2 token.py <old_str1, new_str2> str_replace ✔ 3 submit 4 Langutory: RRRRNNNPPVV(R4N3P2V2) Langutory: NNNP(N3P) PPC=1, POC=0.75, PPF=1 PC=0.91 PPC=0.5, POC=0.5, PPF=1 PC=0.63 (c) (DSK-V3 · django-13551 · Unresolved) (b) (DSK-R1 · matplotlib-21568 · Unresolved) test_min_count_comprehensive.py reproduce_bug.py Figure 1: Illustrative examples of agent trajectories and their corresponding Graphectory and Langutory representations. Plan:L ★ (Φ)=NRPV . The execution in Figure 1b covers all plan phases in its trajec- tory, but violates the intended order, with excessive reproduc- tion(steps 1–4)preceding navigation (steps 5–7) and leading to an unresolved patch. The execution in Figure 1c skips key phases, tran- sitioning directly from navigation (steps 1–3) to patching (step 4) before submission, violating the plan. The consequence of plan violation is a low-quality patch that does not resolve the issue. We will explain our novel process-centric plan compliance metrics using this illustrative example.푃퐶measures whether LangutoryL(푇,Φ) covers the phases specified in the plan: 푃퐶= | Φ∩L(푇,Φ) 푡 | 1≤ 푡 ≤ 푛 | |Φ| (1) 푃퐶=1 if every phase inΦappears at least once in the Langutory. In practice, an agent may skip some plan phases, e.g., directly jump- ing into patching after navigation without reproduction test gener- ation. Therefore,푃퐶 ∈ [0,1]. In Figure 1c, the agent skips repro- duction and validation, resulting in푃퐶=0.5. The executions in Figure 1a and 1b cover all plan phases inΦ and achieve 푃퐶= 1. Not only is covering all phases important, but also following the proper order through trajectory execution.푃푂퐶measures the fraction of phases inL ★ (Φ)that appear in the correct relative order: 푃푂퐶= LIS(푖 1 , . . .,푖 푚 ) 푚 (2) whereLIS(·)denotes the length of the longest increasing subse- quence and푖 푘 denotes the first occurrence index of phase푝 푘 in L(푇,Φ)(if present). Phase revisits are allowed;푃푂퐶evaluates the order of first occurrences. In Figure 1b, the first occurrence indices ofNRPVare[5,1,8,10]. The longest increasing subsequence is[1,8,10]with length 3, yielding푃푂퐶= 3 4 . Failing to follow the expected order can cause inefficient trajectories or task failure. In Figure 1b, the agent begins with reproduction before properly navigating the codebase, leading to repeated modifications to the reproduction script (steps 3–4), and a failed edit at step 8. Agents operate through iterative reasoning–action–observation cycles [35], in which decisions are locally conditioned on the current context rather than the initial instructed plan. Moreover, training strategies can overfit the LLMs to certain actions outside of the instructed plans for specific tasks. Consequently, some actions may not be mapped to plan phases in the Langutory. For example, an agent may decide to open a pull request after patch validation, which is not part of the instructed plan in existing programming agents [1,25,31]. In such a case, Langutory may contain unknown letters that are considered gibberish with respect to the specified plan phases. Including additional actions beyond those in the rec- ommended plan is not necessarily negative, but it can be distracting. Therefore,푃퐹penalizes the appearance of phases outside the spec- ified plan alphabet: 푃퐹= |Φ| | Φ∪L(푇,Φ) 푡 | 1≤ 푡 ≤ 푛 | (3) 푃퐹 ∈ (0,1], and푃퐹=1 if every phase appearing in the Langutory belongs toΦ. The overall compliance score is the geo- metric mean of its three component metrics: 푃퐶=(푃퐶 . 푃푂퐶 . 푃퐹) 1/3 (4) 푃퐶 ∈ [ 0,1], where score푃퐶=1 indicates perfect plan compliance. Geometric mean aggregates sub-metrics multiplicatively, ensuring equal weighting and preventing compensation across dimensions. Low compliance in any dimension proportionally reduces the over- all score. Lower푃퐶scores reflect deviations in missing phases, spurious phases, or violations of the logical phase ordering. 3 Empirical Setup Models and Scaffold. To capture a multi-dimensional analy- sis of plan compliance, we evaluate the SWE-agent scaffold [34] across four diverse LLMs: GPT-5 mini [23] (closed-source fron- tier reasoning model), DeepSeek-R1 [5] (open-source reasoning model), DeepSeek-V3 [16] (open-source general-purpose model), and Devstral-small [24GB] [22] (distilled model specialized in coding). We use the default settings of the models and agent 3 . SWE-agent provides a standardized execution environment, supports multiple LLMs, and includes a default planning mechanism embedded in its system prompt. These properties make it a natural testbed for studying the role of planning in programming agents. 3 The details about the versions and settings of each model are available on the artifact website for reproducibility. Conference’17, July 2017, Washington, DC, USAShuyang Liu, Saman Dehghan, Jatin Ganhotra, Martin Hirzel, and Reyhaneh Jabbarvand Table 1: Summary of studied plan settings, their corresponding formulation, and type of plan variation. Plan SettingPlan Formulation Plan Variation Plan Description Standard (Default) Plan⟨푁 , 푅, 푃 ,푉⟩BaselineStandard Navigation-Reproduction-Patch-Validation plan No Plan—ReductionPlan removed entirely from the system prompt Default Plan - Reproduction⟨푁 ,¬R, 푃 ,푉⟩ReductionReproduction phase removed Default Plan - Validation⟨푁 , 푅, 푃 ,¬V⟩ReductionValidation (after patching) phase removed Default Plan + Regression Test Execution ⟨푅 퐺 , 푁 , 푅 푃 ,푉 ,푉 퐺 ⟩AugmentationRegression test execution phases added Default Plan + Summary of Changes ⟨푁 , 푅, 푃 ,푉 , 푆⟩AugmentationSummarizing changes before submission added Reordered Default Plan⟨푁 , 푃 , 푅,푉⟩ReorderingPatching moved before Reproduction Periodic Plan Reminder⟨푁 , 푅, 푃 ,푉⟩RepeatingDefault plan re-injected every five trajectory steps (a) All(b) Easy (c) Medium (d) Hard 56.3 38.4 39.4 64.8 Success Rate (%) GPT5-mini DeepSeek-V3 DeepSeek-R1 Devstral-small Resolved Unresolved Resolved Unresolved Resolved Unresolved Resolved Unresolved Figure 2: Standard plan compliance metrics and success rate for studied trajectories across all models. Plan Settings. We evaluate performance of agents on a given dataset problem under eight plan settings: (1) default plan (RQ1), (2) no-plan, i.e., removing the entire plan from the system prompt (RQ2), (3) removing the reproduction phase (RQ3), (4) re- moving the validation phase (RQ3), (5) adding a regression test exe- cution phases before navigation,푅 퐺 , and after validation,푉 퐺 (RQ4), (6) adding a change summarization phase푆before submission (RQ4), (7) step reordering, i.e., reproduction test generation after patch- ing (RQ4), and (8) plan reminder, i.e., periodically re-injecting the default plan into the agent’s prompt (RQ5). Table 1 lists plan settings and their corresponding plan-compliant phase sequence, which our pipeline checks trajectories against. We will explain the rationale for these plan mutations in the corresponding RQs. Dataset. We evaluate the mentioned LLMs and plan settings for resolving real-world GitHub issues from SWE-bench Verified [4,13] and SWE-bench Pro [6]. Specifically, our primary evaluation (RQ1– RQ6) uses all 500 instances of SWE-bench Verified, covering three difficulty levels (Easy, Medium, and Hard), providing a realistic setting for agent behavior. To study the generalizability of findings, we repeat the RQ1–RQ5 for Python instances of SWE-bench Pro. Analysis and Metrics. Along with the plan compliance metrics (Equations 1–4), we report the success rate [13] and Graphec- tory metrics (the number of nodes푁퐶, temporal edges푇퐸퐶, and loops퐿퐶in the Graphectory) [18]. Success rate determines the overall impact of plans on the agent’s ability to resolve the issue, and Graphectory metrics provide insights into how plans affect the overall trajectory toward resolution. In addition to metrics, we leverage process-centric Phase Flow Analysis by Liu et al. [18] to provide an in-depth analysis of plan-phase changes in trajec- tories exhibiting plan violations. Phase Flow Analysis can reveal consistent trends in agent trajectories across different problems. The outcome of this analysis is a Sankey diagram illustrating the evolution of trajectories from one plan phase to the next. 4 Standard Plan Compliance and Violation This section first investigates to what extent SWE-agent with differ- ent choices of LLMs follows the standard program repair workflow, i.e.,Navigation,Reproduction,Patch, andValidation(§4.1). As an extreme alternative, we remove the entire plan and evaluate how the trajectories change (§4.2). We analyze 4,000 trajectories, generated by SWE-agent using four different LLMs to repair 500 SWE-bench Verified instances under two plan settings. 4.1 RQ1. Standard (Default) Plan Setting The standard program repair plan that has been used for years by software developers is localizing the bug (navigating through files, classes, methods, and lines to pinpoint the bug location) and attempting to reproduce it, patch it, and validate the patch through test execution. Existing scaffolds instruct agents to follow a similar plan with the given order in their default system prompt (Φ= 푁,푅,푃,푉andL ★ (Φ)=NRPV). Figure 2a presents the average푃퐶,푃푂퐶,푃퐹, and푃퐶values (Equations 1–4) cal- culated for 2,000 trajectories under this plan. Figures 2b–2d show breakdown per problem difficulty levels (Easy, Medium, and Hard). Beyond quantitative metrics, Figure 3 shows the plan phase flow of the agents for all the trajectories. Finding 1. Standard plan compliance varies across models. Devstral-small consistently follows the plan phases in the given order, demonstrated by high푃퐶and푃푂퐶values. However, it exhibits out-of-plan phases to a notable degree in its trajectories (gray flows in Figure 3), with low overall푃퐶. GPT-5 mini, on the other hand, may adapt its strategy depending on the difficulty of the problem. Its trajectories show out-of-plan phases (lower푃퐹), and it usually skipsReproduction(lower푃퐶and푃푂퐶). DeepSeek-V3 exhibits a near-perfect푃퐹=0.99 but substantially lower푃퐶and 푃푂퐶, i.e., restricts itself to plan phases but frequently omits some or executes them out of order. DeepSeek-R1 consistently demonstrates lower plan compliance than others, both in following the instructed plan phases and in doing so in the correct order. Conference’17, July 2017, Washington, DC, USA All Easy Medium Hard 012345678910012345678910012345678910012345678910 Devstral-smallGPT5-miniDeepSeek-V3DeepSeek-R1 Figure 3: Phase flow analysis under Standard plan (L ★ (Φ)= NRPV ). Flow thickness notes the proportion of trajectories going from one phase to another. The black bar indicates trajectory termination, and gray flows represent out-of-plan phases. Finding 2. Standard plan compliance is overall higher on resolved instances. Intuitively, following the instructed standard plan that reflects decades of best practices should lead to successful bug repair. The Mann–Whitney U test [20] confirms the significance of this observation for Devstral-small and DeepSeek-R1, where resolved instances consistently exhibit higher plan compliance (푝= 1푒− 5 and 푝= 0.032, respectively). The correlation is positive for DeepSeek-V3, but with less statis- tical significance (푝=0.60). The exception is GPT-5 mini, where unresolved trajectories are usually more compliant with the plan, demonstrating negative correlation but with low statistical sig- nificance (푝=0.285). Phase flow analysis (Figure 3) demystifies this observation: GPT-5 mini adapts its strategy based on problem difficulty. For easier problems, where the issue description also likely contains all the information to localize the bug, it often skips Reproductionand transitions from navigation to patch. For harder problem, it follows the instructed plan more closely, with thicker Navigation -to-Reproduction flows in earlier phase changes. Finding 3. Necessity for process-centric plan compliance metrics. Graphectory metrics <node count, edge count, loop count> are in general higher for Devstral-small (<86,179,64>) and GPT-5 mini (<38,49,18>) compared to DeepSeek-V3 (<15,27,4>) and DeepSeek-R1 (<14,21,4>). Pearson correlation [26] shows a very weak positive correlation (0< 푟 ≤0.2) between plan compliance 푃퐶and Graphectory metrics. This confirms the need for a new process-centric metric to specifically target plan compliance, as an orthogonal factor to trajectory complexity. Finding 4. The standard plan, in its current form, is incom- plete. We observe that GPT-5 mini and Devstral-small, in addition to creating and executing new tests as instructed by the plan, fre- quently run existing tests in the repository (lower푃퐹compared to other models). The practice is, in fact, useful for better reproduction test generation and patch validation [3]. This finding motivates augmenting existing plans with additional, relevant phases, and assessing the impact of this plan on trajectories (§5.2). 4.1.1 Contributing Factors to Plan Compliance/Violation. Fine- tuning paradigm. Depending on the LLM, agents may skip spe- cific plan phases, perform them in a different order, or exhibit out- of-plan actions. Except for Devstral-small, SWE-agent with other LLMs tends to skipReproduction(illustrated by lower presence of Reproductionphase in Figure 3). SWE-agent with DeepSeek mod- els often prioritizesReproductiontoNavigation. We investigate this under a controlled setting in RQ2 (§4.2). This also motivates modifying the plan by removing some steps (§5.1) to further inves- tigate plan compliance across models. Context window pressure. As trajectories grow, the initial plan must compete with an increasingly long history of thoughts, tool calls, file contents, and error messages, which can make the plan less salient later in execution. Deviation is further encouraged by the agent’s locally-conditioned decision process, in which each action is chosen primarily based on the current context and re- cent tool feedback rather than explicit adherence to the original global workflow. We further investigate this speculation in RQ5 by frequent plan reminders (§5.3.2). Data contamination and overfitting. Backbone LLMs may over- fit to the workflow defined by the standard plan. There is also a risk of data contamination when a successful trajectory for solving individual problems in SWE-bench Verified is used to fine-tune them [24,29]. Therefore, plan compliance may not be rooted in their ability to follow plan instructions [27] or the plan positively impacting their reasoning to accomplish the task [15]. We will eval- uate the impact of this factor by repeating the experiments on a less contaminated SWE-bench Pro dataset (§6.1). 4.2 RQ2. No Plan Setting The previous research question shows a notable variance in compli- ance with the Standard plan across agents. Given that the scaffold is Conference’17, July 2017, Washington, DC, USAShuyang Liu, Saman Dehghan, Jatin Ganhotra, Martin Hirzel, and Reyhaneh Jabbarvand 012345678910012345678910 0123456789 012345678910 Devstral-smallGPT5-miniDeepSeek-V3DeepSeek-R1 Figure 4: Phase flow analysis under No Plan setting. Agents still exhibit traces of Standard plan phases (Φ=N ,R ,P ,V ). Figure 5: Impact of No Plan setting on the success rate. identical in all agents, two important factors influencing the obser- vations are (1) the ability of specific LLMs to follow the instructed plan, or (2) a conflict between following plan-prescribed phases and training-prescribed workflows. We investigate the magnitude of the former in §5. For the latter, we repeated the experiments under a No Plan setting to observe how agents perform without any specific plan. We completely remove the default plan from the system prompt of SWE-agent. Thereby, the agent only receives a high-level guideline to fix the issue: given the issue description, make changes to satisfy the issue description requirements 4 . Figure 5 compares the success rate in repairing SWE-bench Ver- ified problems with and without the Standard plan. The left bars represent the set size, i.e., the number of issues that are resolved under each plan setting. The grid dots demonstrate whether the plan setting applies to the instances on the top bars. When a grid dot is black, it demonstrates that the instances on the top bars are resolved under that setting. Although the agent receives no plan instruction under this setting, we further investigate if it exhibits any trace of Standard plan in its trajectory. The rationale here is that the backbone LLM of the agent may already have seen instructions related to this task during training/fine-tuning. Figure 4 shows phase flow analysis of trajectories under the No Plan setting. Finding 5. Even when not explicitly instructed, agents follow the Standard plan to a notable degree. The phase flow analysis in Figure 4 shows that Devstral-small starts withNavigation, and most trajectories still follow the Standard workflowNRPV, with some phases out of the Standard plan in between. Similarly, GPT-5 mini trajectories also follow a subset of Standard plan, often withoutReproduction. In contrast, DeepSeek-V3 and DeepSeek- R1 largely reduce their trajectories toNPpatterns, skipping ReproductionorValidation. This suggests that different models internalize problem-solving processes differently, depending on their training. In the absence of global plans, the encoded strategy takes over the reasoning to solve the problem. 4 The system prompt for this experiment is available on the artifact website under artifacts/plan-settings/no_plan/default.yaml Finding 6. The success rate drops in the absence of the stan- dard plan, although to different degrees across models and difficulty levels. Figure 5 shows that removing the plan consis- tently reduces performance across all models. The majority of the instances that SWE-agent resolved only under the Standard plan setting are of Medium difficulty. Devstral-small and GPT-5 mini, which exhibit problem-solving strategies similar to the Standard plan, show only minor drops when the plan is removed. In contrast, DeepSeek models, particularly DeepSeek-R1, experience a substan- tial performance drop, despite showing lower compliance when the plan is present. This indicates that the plan, even if not properly followed, can positively impact the local reasoning of the agents, guiding them towards their goal. Without it, reasoning becomes less focused, often resulting in premature convergence: these mod- els demonstrate smaller Graphectory metric values under the No Plan setting, compared to the Standard plan. Finding 7. Agents can fix previously unresolved issues under no-plan setting. DeepSeek-V3, DeepSeek-R1, Devstral-small, and GPT-5 mini each resolve additional instances that are not solved un- der the default plan: 23, 11, 28, and 34, respectively. As we will show later (§6.2), this is largely affected by the inherent nondeterminism of LLM-based agents, with 4, 7, 16, and 4 instances deterministi- cally only resolved under the no-plan setting. Manual inspection of these instances reveals a consistent trend across all models: The Standard plan instructs the model to reproduce the bug before the patch. However, test generation for reproduction is a complex, non- trivial problem [2]. In the instances studied, the models generated incorrect reproduction tests, leading to repeated patch-test failure cycles without success. Under the No Plan setting, the same model skipped the reproduction phase and generated the correct patch. This is alarming but interesting: the solution under No Plan can be due to data contamination [24]. The instructed plan, however, overrides it, and the agent attempts to generate a reproduction test, which fails due to an inability to produce a good test. 5 Plan Variations RQ2 shows that agents, even if not explicitly instructed to follow the standard plan, still incorporate it in their problem-solving strategy. As discussed, this is likely due to the training objectives of backbone LLMs. For a more controlled plan compliance analysis, we create variations of the Standard plan with small changes (removing one plan phase in §5.1 or adding one phase outside of the Standard plan in §5.2). We then investigate compliance with the mutated plan and the impact of isolated plan changes on success rate. 5.1 RQ3. Reduced Plan Settings We study two reduced plan variations by removing either the Reproductionor theValidationphase. Removing Navigation and Patching is unlikely to reveal notable observations, as these Conference’17, July 2017, Washington, DC, USA No Reproduce 012345678910012345678910012345678910012345678910 No Validation 012345678910012345678910012345678910 012345678 Devstral-smallGPT5-miniDeepSeek-V3DeepSeek-R1 Figure 6: Phase flow analysis under No Reproduction (L ★ (Φ)=NPV) and No Validation plan setting (L ★ (Φ)=NRP). Trajectories still show traces of removed phases from the Standard plan. (a) (b) Figure 7: Impact of No Reproduction plan (a) and No Validation plan (b) on the success rate. GPT5-mini DeepSeek-V3 DeepSeek-R1 Devstral-small Resolved Unresolved Resolved Unresolved Resolved Unresolved Resolved Unresolved Figure 8: No Reproduction plan compliance metrics. two phases are essential and consistently appear in agent trajecto- ries from observations in the No Plan setting. 5.1.1 RQ3.1. No Reproduction Setting. Figures 6–8 illustrate the results underL ★ (Φ)=NPV. Devstral-small and GPT-5 mini achieve lower scores on푃퐹, sinceReproductionis encoded in their internal problem-solving strategy. They also achieve near- perfect compliance for푃퐶and푃푂퐶, i.e., follow all other plan phases in proper order. Phase flow analysis (Figure 6) shows an inter- esting observation about the impact of the plan on their trajectory: in the absence of plan, these two models includeReproduction more consistently in their trajectories compared to when the plan excludes only that phase (compare Figure 6 with Figure 4). This, consequently, impacts the ability of these models to accomplish the task. As shown in Figure 7a, their success rate decreases notably. In contrast, DeepSeek models achieve nearly-perfect푃퐹val- ues, indicating that their trajectories do not observeReproduction. However, they suffer the most from lower푃퐶and푃푂퐶, confirmed by phase flow analysis. Similarly, these two models suffer from an GPT5-mini DeepSeek-V3 DeepSeek-R1 Devstral-small Resolved Unresolved Resolved Unresolved Resolved Unresolved Resolved Unresolved Figure 9: No Validation plan compliance metrics. incomplete plan, reflected in the drop in success rate. DeepSeek- R1’s performance drop is more substantial. A deeper analysis of its trajectories reveals that, in many cases (349 instances), the model produces malformed tool calls, emitting function calls as plain text rather than in the expected format. This leads to repeated execution errors and eventual termination 5 . The observation suggests that re- movingReproductiondestabilizes DeepSeek-R1 interaction with the scaffold itself, beyond its effect on problem-solving. 5.1.2 RQ3.2. No Validation Setting. Figures 6, 7b, and 9 illustrate the results underL ★ (Φ)=NRP. GPT-5 mini suffers greatly withoutValidation, indicating a strong dependence on this phase, as it enables the model to identify incorrect patches and iteratively refine them, rather than prematurely submitting them. In contrast, DeepSeek-V3 shows only slight performance degra- dation and high푃퐹, consistent with its tendency to skip Validationin the No Plan setting. This further reinforces the im- portance of either alignment of the model’s internalized strategy 5 Exit message: “Exit due to repeated format/blocklist/bash syntax errors” Conference’17, July 2017, Washington, DC, USAShuyang Liu, Saman Dehghan, Jatin Ganhotra, Martin Hirzel, and Reyhaneh Jabbarvand Regression Summary 012345678910012345678910012345678910012345678910 Devstral-smallGPT5-miniDeepSeek-V3DeepSeek-R1 Figure 10: Phase flow analysis under Regression Testing (L ★ (Φ)=푅 퐺 NRPV푉 퐺 ) and Summary plan setting (L ★ (Φ)= NRPVS ). (a) (b) Figure 11: Impact of Regression Testing (a) and Summary plan (b) on the success rate. GPT5-mini DeepSeek-V3 DeepSeek-R1 Devstral-small Resolved Unresolved Resolved Unresolved Resolved Unresolved Resolved Unresolved Figure 12: Regression Testing plan compliance metrics. with the instructed plan or enabling true reasoning and adaptive planning in models. Devstral-small is less affected than in the No Re- production setting, suggesting that early-stage grounding through reproduction (e.g., for better bug localization) is more critical to its performance, while validation plays a less central role in shaping its trajectory. We believe this is yet another signal about the data con- tamination in this model, as it can generate correct patches without validation. DeepSeek-R1 again exhibits the lowest success rate and significantly low푃퐶and푃푂퐶, with the majority of trajectories terminating early with repeated tool-calling failures (§5.1.1). Finding 8. The negative impact of a bad, incomplete plan is greater on trajectories than the impact of no plan at all. Overall, removing a specific plan phase can strongly affect models that (1) do not have it in their internal workflow to compensate or (2) are trained to incorporate the plan in their reasoning, as in its absence, their reasoning is incomplete. Finding 9. Agents can fix previously unresolved issues un- der reduced plan settings. Similar to Finding 7, the primary reason for exclusive resolution under the No Reproduction set- ting is the agents’ inability to generate a good reproduction test GPT5-mini DeepSeek-V3 DeepSeek-R1 Devstral-small Resolved Unresolved Resolved Unresolved Resolved Unresolved Resolved Unresolved Figure 13: Summary plan compliance metrics. when instructed by the Standard plan. In contrast, under the No Validation setting, specifically when we eliminate the impact of non- determinism (§6.2), agents can rarely resolve previously unsolved issues: GPT-5 mini (7), Devstral-small (19), DeepSeek-V3 (5), and DeepSeek-R1 (1). Analysis of those instances shows that agents still incorporated the validation phase, suggesting remaining impact of nondeterminism. 5.2 RQ4. Augmented Plan Settings We investigate compliance with plans that include new phases to further demonstrate overfitting to known plans. Arbitrary, task- irrelevant phases may bias the findings; it will be unclear whether an agent struggles with plan compliance or reacts negatively to in- coherent instructions. To account for this threat, we only introduce relevant phases to the issue repair task, namely, (1) executing re- gression tests at the beginning and end (§5.2.1) and (2) summarizing changes in a PR-style format before submission (§5.2.2). 5.2.1 RQ4.1. Plan with Regression Test Execution. Figures 10–12 show the results of this experiment. Devstral-small and GPT-5 mini already perform regression testing, even when not explicitly Conference’17, July 2017, Washington, DC, USA Reordered 012345678910012345678910012345678910012345678910 Reminded 012345678910012345678910012345678910 0123456 Devstral-smallGPT5-miniDeepSeek-V3DeepSeek-R1 Figure 14: Phase flow analysis under Reordered (L ★ (Φ)= NPRV ) and Reminded setting (L ★ (Φ)=NRPV ). (a) (b) Figure 15: Impact of Reordered plan (a) and Reminded plan (b) on the success rate. GPT5-mini DeepSeek-V3 DeepSeek-R1 Devstral-small Resolved Unresolved Resolved Unresolved Resolved Unresolved Resolved Unresolved Figure 16: Reordered plan compliance metrics. instructed to do so (Finding 4). The drop in their success rate is minimal and is likely due to non-determinism (§6.2). Devstral-small has a high푃퐶, including all phases, while GPT-5 mini adaptively skips or reorders some phases as discussed (lower푃퐶). Concern- ing regression testing, Devstral-small and GPT-5 mini show a no- table difference: as shown through phase flow analysis (Figure 10), Devstral-small performs regression testing afterNavigationand afterValidation, while GPT-5 mini consistently runs regression tests as the first step and afterValidation using new tests. DeepSeek-V3 experiences a higher performance drop, accom- panied by low푃퐶, suggesting difficulty in incorporating the re- gression testing phases. Running existing tests early in the trajec- tory and their lengthy feedback likely override the impact of other plan phases (lower푃푂퐶), shifting focus toward test environment setup and irrelevant execution results rather than effective bug localization and patching. DeepSeek-R1 continues to exhibit severe performance issues in this setting. The persistent tool-calling er- rors result in a very low success rate, preventing any conclusion regarding plan compliance. We speculate that this behavior is due to optimization for short-term reward [9], which is a known issue in DeepSeek-R1 and specifically in reinforcement learning [8, 28]. GPT5-mini DeepSeek-V3 DeepSeek-R1 Devstral-small Resolved Unresolved Resolved Unresolved Resolved Unresolved Resolved Unresolved Figure 17: Reminded plan compliance metrics. 5.2.2RQ4.2. Plan with Change Summary. Adding a summary phase, which is independent of the core repair process, yields minimal behavioral changes across most models. The overall plan compli- ance (푃퐶) for Devstral-small, GPT-5 mini, and DeepSeek-V3 remains nearly unchanged from the Standard setting (Figure 13). The suc- cess rates are also largely unaffected (Figure 11b), as the summary phase typically occurs at the end of the trajectory and does not impact intermediate reasoning. DeepSeek-R1, however, exhibits a substantial performance drop, with pervasive tool-calling fail- ures observed in 413 instances. This suggests that even orthogonal additions to the plan can destabilize this model’s reasoning. Finding 10. Plan augmentation highlights plan overfitting, and is effective only when aligned with a model’s internal strategy. Introducing additional phases provides limited benefit and can degrade performance if the model does not naturally em- ploy those steps. Adding early phases can introduce unnecessary overhead or distract the model if phases are not well internalized. 5.3RQ5. Reordered and Reminded Plan Settings 5.3.1 RQ5.1. Reordered Plan Setting. Previous settings challenge agents to achieve high plan phase compliance (푃퐶) and Plan Phase Fidelity (푃퐹). Challenging plan order compliance (푃푂퐶) in- volves reordering phases and investigating plan compliance. As Conference’17, July 2017, Washington, DC, USAShuyang Liu, Saman Dehghan, Jatin Ganhotra, Martin Hirzel, and Reyhaneh Jabbarvand 012345678910012345678910012345678910 0123456789 Devstral-smallGPT5-miniDeepSeek-V3DeepSeek-R1 Figure 18: Phase flow analysis under Standard plan for SWE-bench Pro (L ★ (Φ)=NRPV ). Figure 19: Compliance metrics on SWE-bench Verified deter- ministic (a) and SWE-bench Pro (b) in augmented plan settings, to avoid bias or noise, we reorder the phases so that the new plan remains relevant to the task. Specifically, we instruct the agent to patch the bug immediately after navigation and to postpone generating the reproduction test after patching, primarily for patch validation rather than bug localization. The reordered plan (L ★ (Φ)=NPRV) slightly impacts agents’ behavior. Phase flow analysis (Figure 14) shows that DeepSeek-V3 often proceeds directly from navigation to patch- ing, consistent with the modified plan. As shown in Figure 15a, this slightly improves its success rate (from 191 resolved under the default plan to 196). In contrast, Devstral-small performs reproduc- tion before patching despite the reordered instruction, resulting in a lower푃푂퐶as reflected in Figure 16. For models that rely on repro- duction, executing it before patching remains preferred, as it helps confirm the bug and improve the overall process. The reordering leads to moderately reduced success rates for Devstral-small, which consistently relies on reproduction, and GPT-5 mini, which adap- tively incorporates reproduction as problem difficulty increases. Finding 11. Agents prioritize effective workflows over pre- scribed phase ordering. Agents do not rigidly follow suboptimal ordering constraints; instead, they override them in favor of execu- tion orders that better support their problem-solving process. Finding 12. Delaying weakly internalized phases can reduce interference, but it does not consistently improve perfor- mance. For models that do not naturally rely on a phase (e.g., Reproductionin DeepSeek-V3), postponing it can reduce interfer- ence with early steps. However, this benefit is not consistent: after accounting for nondeterminism (§6.2), the reordered plan yields only a negligible change in success rate (38.3% to 38%). 5.3.2 RQ5.2. Reminded Plan Setting. To mitigate the context win- dow pressure (§4.1.1), where the initial plan becomes less influential as the trajectory length increases, we introduce a Reminded plan setting. In this variant, the Standard plan is periodically re-inserted into the context every five steps. Figure 17 shows that periodic plan reminders improve plan compliance for DeepSeek-V3 and maintain similar푃퐶for the other models. This leads to consistent improve- ments in success rates across models, as shown in Figure 15b. The reminders prevent drifting into irrelevant sub-goals (e.g., exploring unrelated directories) and return focus to the repair task. 6 Factors Beyond the Scaffold and LLM 6.1 RQ6. Generalization to Other Benchmarks In previous RQs, we observed that plan variants affect model per- formance and plan compliance differently, largely due to their in- ternalized problem-solving strategies. One potential explanation is data contamination, where models may overfit to the trajectories present in the SWE-bench Verified dataset. To assess generaliza- tion of the findings, we repeated RQ1–RQ5 on SWE-bench Pro [6], a contamination-resistant benchmark designed to minimize over- lap with LLM training corpora through licensing constraints. To minimize moving factors, we focused on 266 Python instances of SWE-bench Pro. This benchmark is more challenging and less con- taminated, and the studied models achieve near-zero success rates in many instances. To obtain more meaningful comparisons, we selected instances that at least one of the LLMs could resolve under the Standard plan settings, leaving us with 31 instances. Figure 19b shows the plan compliance (푃퐶) values of studied agents under all plan settings. We observe that plan compliance drops by 13% on average across all agents. The phase flow analysis in Figure 18 demonstrates a different trend under the Standard plan setting in SWE-bench Pro compared to SWE-bench Verified (Fig- ure 3), specifically for DeepSeek-V3 and DeepSeek-R1. This is likely because SWE-bench Pro instances are more challenging and less contaminated, and most of the problem solving effort goes into repetitiveNavigationandPatching, without reaching to the Validation. Similar to SWE-bench Verified, the majority of the agents still skipReproduction, achieving relatively high푃퐶scores on the No Reproduction setting. 6.2 RQ7. Impact of Nondeterminism To mitigate any bias due to agents’ inherent nondeterminism, we repeated the experiments under the Standard plan three times. Pair- wise McNemar test [21] shows statistically significant differences across runs, confirming the non-determinism. We then identify in- stances that are consistently resolved or unresolved across Standard plan runs and evaluate the remaining plan variants on this repro- ducible subset (GPT-5 mini: 401, Devstral-small: 308, DeepSeek-V3: 323, and DeepSeek-R1: 287). Figure 19a reports the plan compliance 푃퐶, remaining nearly identical to that observed on the full bench- mark. The results are consistent with our earlier findings: plan Conference’17, July 2017, Washington, DC, USA reduction, augmentation, and reordering affect models differently depending on their underlying problem-solving strategies, as re- flected in the No Plan setting. Periodic plan reminders consistently improve performance by maintaining focus on the core task. 7 Related Work Planning has become a central mechanism for improving the relia- bility of agents, especially for long-horizon and tool-using tasks. A recent survey [11] explores existing research on Agent planning, including task decomposition, plan selection, external planning modules, reflection, and memory. Building on these ideas, several studies focus on constructing and refining plans in single- and multi-agent systems. Agent-Oriented Planning [14] and PMC [36] decompose complex tasks into structured subtasks and coordinate multiple agents to satisfy constraints, while EAGLET [30] and Plan- and-Act [7] separate planning and execution into distinct LLMs to improve long-horizon reasoning. ReWoo took an extreme stance, planning all actions up-front [33]. Liu et al. [18] introduce process- centric metrics, but as we saw in Finding 3 (§4.1), those alone are insufficient to understand plan compliance. SAGE [10] shows that abstract plans distilled from prior exe- cutions can guide future behavior and improve performance on software engineering tasks. While these approaches demonstrate the benefits of planning, they primarily evaluate success at the task level and implicitly assume that agents will follow the generated or provided plans during execution. Jia et al. [12]assesses whether a web agent’s actions align with its stated plan using LLM-based judges. However, this approach relies on costly and potentially unstable LLM scoring, limiting its scalability. In contrast, our work introduces mathematically defined plan compliance metrics that enable systematic analysis of agent behavior under controlled plan variations. 8 Threats to Validity External Validity. We evaluate four models spanning diverse capabilities, and conduct experiments on two different benchmarks, SWE-bench Verified and SWE-bench Pro, which consist of real- world GitHub issues and differ in data contamination and task composition, providing complementary evaluation settings. We adopt SWE-agent as the scaffold due to its widespread use. We use a structured plan common among several programming agents (e.g., mini-SWE-agent, Trae agent, OpenHands) as the standard plan to avoid bias and to be representative of common practice. Internal Validity. Agent behavior is inherently non-deterministic due to the probabilistic nature of LLMs and their step-by-step decision-making process, where each action depends on local rea- soning and previous execution results. To minimize the impact of inherent nondeterminism of agents, we use a consistent default con- figuration across all experiments and repeat each experiment three times, focusing on stable behavioral patterns rather than random artifacts. Construct Validity. Our analysis relies on a phase-level abstrac- tion of trajectories, which may omit fine-grained or project-specific actions (e.g., environment setup). However, it captures the core problem-solving stages that determine plan compliance: naviga- tion, reproduction, patching, and validation. To minimize such threats, our pipeline is built on top of peer-reviewed artifacts and is validated with well-vetted tools. We distinguish between key variations, such as newly generated tests and regression tests. Re- maining low-level differences are treated as general actions and do not affect the main conclusions. 9 Conclusion This paper analyzes 16,991 trajectories to assess plan compliance in programming agents. It introduces novel plan compliance metrics and runs agents under a variety of plans. It also evaluates the impact of different system-prompt plans on the agent’s success in the issue- resolution task. We find that while plans clearly matter for task success, agents often struggle to comply with them. This highlights the potential to further boost agent performance in future work via better plans and/or improved plan compliance. 10 Data Availability Statement The artifacts of this paper are publicly available at [19]. 11 Acknowledgment This work is supported by the IBM-Illinois Discovery Accelerator Institute (IIDAI) and NSF CCF-2238045 grants. References [1]Trae Agent. 2026. Trae Agent System Prompt with Default Plan. https://github. com/bytedance/trae-agent/blob/main/trae_agent/prompt/agent_prompt.py. [2] Toufique Ahmed, Martin Hirzel, Rangeet Pan, Avraham Shinnar, and Saurabh Sinha. 2024. TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved? https://arxiv.org/abs/2412.02883 [3]Yang Chen, Toufique Ahmed, Reyhaneh Jabbarvand, and Martin Hirzel. 2026. Can Old Tests do New Tricks for Resolving SWE Issues?. In Symposium on the Foundations of Software Engineering (FSE). [4]Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Kevin Liu, and Aleksander Madry. 2024. Introducing SWE- bench Verified. https://openai.com/index/introducing-swe-bench-verified/ [5] DeepSeek-AI. 2025. DeepSeek-R1-0528. https://huggingface.co/deepseek-ai/ DeepSeek-R1-0528. [6] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al.2025. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? arXiv preprint arXiv:2509.16941 (2025). [7]Lutfi Eren Erdogan, Hiroki Furuta, Sehoon Kim, Nicholas Lee, Suhong Moon, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. 2025. Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks. In Forty-second Inter- national Conference on Machine Learning. https://openreview.net/forum?id= ybA4EcMmUZ [8]Sebastian Farquhar, Vikrant Varma, David Lindner, David Elson, Caleb Bid- dulph, Ian Goodfellow, and Rohin Shah. 2025. Mona: Myopic optimization with non-myopic approval can mitigate multi-step reward hacking. arXiv preprint arXiv:2501.13011 (2025). [9]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. Deepseek-r1: Incen- tivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025). [10]Hiroaki Hayashi, Bo Pang, Wenting Zhao, Ye Liu, Akash Gokul, Srijan Bansal, Caiming Xiong, Semih Yavuz, and Yingbo Zhou. 2025. Self-Abstraction from Grounded Experience for Plan-Guided Policy Refinement. arXiv preprint arXiv:2511.05931 (2025). [11]Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024. Understanding the planning of LLM agents: A survey. arXiv preprint arXiv:2402.02716 (2024). [12]Allison Sihan Jia, Daniel Huang, Nikhil Vytla, Nirvika Choudhury, Shayak Sen, John C Mitchell, and Anupam Datta. 2025. What Is Your Agent’s GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignment. arXiv preprint arXiv:2510.08847 (2025). Conference’17, July 2017, Washington, DC, USAShuyang Liu, Saman Dehghan, Jatin Ganhotra, Martin Hirzel, and Reyhaneh Jabbarvand [13]Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real- World GitHub Issues?. In International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=VTF8yNQM66 [14]Ao Li, Yuexiang Xie, Songze Li, Fugee Tsung, Bolin Ding, and Yaliang Li. 2025. Agent-Oriented Planning in Multi-Agent Systems. In The Thirteenth Interna- tional Conference on Learning Representations. https://openreview.net/forum?id= EqcLAU6gyU [15] Shanchao Liang, Spandan Garg, and Roshanak Zilouchian Moghaddam. 2025. The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason. arXiv preprint arXiv:2506.12286 (2025). [16]Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024). [17]Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics 12 (2024), 157–173. [18]Shuyang Liu, Yang Chen, Rahul Krishna, Saurabh Sinha, Jatin Ganhotra, and Reyhan Jabbarvand. 2025. Process-Centric Analysis of Agentic Software Systems. arXiv preprint arXiv:2512.02393 (2025). [19] Shuyang Liu, Saman Dehghan, Jatin Ganhotra, Martin Hirzel, and Reyhaneh Jabbarvand. 2026. "From Plan to Action: How Well Do Agents Follow the Plan?" artifact website. https://github.com/Intelligent-CAT-Lab/Planning-Analysis. [20]Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50–60. [21] Quinn McNemar. 1947. Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages. Psychometrika 12, 2 (1947), 153–157. https://doi.org/10.1007/BF02295996 [22]Mistral AI. 2025. Devstral-Small-2512. https://openrouter.ai/mistralai/devstral- 2512. [23]OpenAI. 2025. GPT5-Mini. https://developers.openai.com/api/docs/models/gpt- 5-mini. [24] OpenAI. 2026. Why SWE-bench Verified no longer measures frontier coding capabilities. https://openai.com/index/why-we-no-longer-evaluate-swe-bench- verified. [25]OpenHands.2025.OpenHandsSystemPromptwithDe- faultPlan.https://github.com/OpenHands/OpenHands/blob/ 08118d742b564add3e970921ac8910c265ece975/evaluation/benchmarks/swe_ bench/prompts/swe_default.j2. [26]Karl Pearson. 1895.VII. Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London 58, 347-352(121895),240–242.https://doi.org/10.1098/rspl.1895.0041 arXiv:https://royalsocietypublishing.org/rspl/article-pdf/58/347- 352/240/263745/rspl.1895.0041.pdf [27]Thanosan Prathifkumar, Noble Saji Mathews, and Meiyappan Nagappan. 2025. Does SWE-Bench-Verified Test Agent Ability or Model Memory? arXiv preprint arXiv:2512.10218 (2025). [28] Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani- Tür, Gokhan Tur, and Heng Ji. 2025. ToolRL: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958 (2025). [29]IBM Research. 2026. From 73% to 11%: Revealing True SWE-Agent Capabilities with Discriminative Subsets. https://jatinganhotra.dev/blog/swe-agents/2025/06/ 05/swe-bench-verified-discriminative-subsets.html. [30]Shuzheng Si, Haozhe Zhao, Kangyang Luo, Gang Chen, Fanchao Qi, Minjia Zhang, Baobao Chang, and Maosong Sun. 2025. A Goal Without a Plan Is Just a Wish: Efficient and Effective Global Planner Training for Long-Horizon Agent Tasks. https://arxiv.org/abs/2510.05608 [31]SWE-Agent. 2026. SWE-agent System Prompt with Default Plan. https://github. com/SWE-agent/SWE-agent/blob/main/config/default.yaml. [32]Scott Wu. 2024. Introducing Devin, the first AI software engineer. Cognition Labs Blog (2024). [33]Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, and Dongkuan Xu. 2023. ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models. https://arxiv.org/abs/2305.18323 [34] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: agent-computer interfaces enable automated software engineering. In Proceedings of the 38th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS ’24). Curran Associates Inc., Red Hook, NY, USA, Article 1601, 125 pages. [35]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR). https: //openreview.net/forum?id=WE_vluYUL-X [36] Cong Zhang, Xin Deik Goh, Dexun Li, Hao Zhang, and Yong Liu. 2025. Planning with multi-constraints via collaborative language agents. In Proceedings of the 31st International Conference on Computational Linguistics. 10054–10082. [37]Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Au- toCodeRover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592–1604.