Paper deep dive
TRAJEVAL: Decomposing Code Agent Trajectories for Fine-Grained Diagnosis
Myeongsoo Kim, Dingmin Wang, Siwei Cui, Farima Farmahinifarahani, Shweta Garg, Baishakhi Ray, Terry Yue Zhuo, Rajdeep Mukherjee, Varun Kumar
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/27/2026, 1:14:28 AM
Summary
TRAJEVAL is a diagnostic framework that decomposes code agent trajectories into three interpretable stages—search, read, and edit—to evaluate performance beyond binary outcome metrics like Pass@1. By comparing agent trajectories against reference patches, the framework computes precision and recall metrics, revealing that recall is a stronger predictor of success than precision. The study, analyzing 16,758 trajectories across seven models and three architectures, demonstrates that these diagnostics are predictive of task success and actionable, enabling real-time feedback that improves performance by 2.2–4.6% while reducing costs.
Entities (5)
Relation Signals (3)
TRAJEVAL → evaluates → Code Agent
confidence 100% · TRAJEVAL, a diagnostic framework that decomposes agent trajectories
TRAJEVAL → uses → Reference Patch
confidence 100% · compute precision and recall by comparing against reference patches
GPT-5 → exhibitsfailuremode → Edit Targeting
confidence 90% · GPT-5 locates relevant code but targets edits incorrectly
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Code agents can autonomously resolve GitHub issues, yet when they fail, current evaluation provides no visibility into where or why. Metrics such as Pass@1 collapse an entire execution into a single binary outcome, making it difficult to identify where and why the agent went wrong. To address this limitation, we introduce TRAJEVAL, a diagnostic framework that decomposes agent trajectories into three interpretable stages: search (file localization), read (function comprehension), and edit (modification targeting). For each stage, we compute precision and recall by comparing against reference patches. Analyzing 16,758 trajectories across three agent architectures and seven models, we find universal inefficiencies (all agents examine approximately 22x more functions than necessary) yet distinct failure modes: GPT-5 locates relevant code but targets edits incorrectly, while Qwen-32B fails at file discovery entirely. We validate that these diagnostics are predictive, achieving model-level Pass@1 prediction within 0.87-2.1% MAE, and actionable: real-time feedback based on trajectory signals improves two state-of-the-art models by 2.2-4.6 percentage points while reducing costs by 20-31%. These results demonstrate that our framework not only provides a more fine-grained analysis of agent behavior, but also translates diagnostic signals into tangible performance gains. More broadly, TRAJEVAL transforms agent evaluation beyond outcome-based benchmarking toward mechanism-driven diagnosis of agent success and failure.
Tags
Links
- Source: https://arxiv.org/abs/2603.24631v1
- Canonical: https://arxiv.org/abs/2603.24631v1
Trouble viewing inline? Open PDF directly →
Full Text
69,830 characters extracted from source content.
Expand or collapse full text
Preprint. Under review. TRAJEVAL: Decomposing Code Agent Trajectories for Fine- Grained Diagnosis Myeongsoo Kim 1 , Dingmin Wang 1 , Siwei Cui 1 , Farima Farmahinifarahani 1 , Shweta Garg 1 , Baishakhi Ray 1 , Terry Yue Zhuo 2∗ , Rajdeep Mukherjee 1 , Varun Kumar 1 1 AWS AI Labs 2 Monash University mysoo, wdimmy, siweicui, fafarima, shwegarg, rabaisha, mukherr, kuvrun@amazon.com terry.zhuo@monash.edu Abstract Code agents can autonomously resolve GitHub issues, yet when they fail, current evaluation provides no visibility into where or why. Metrics such as Pass@1 collapse an entire execution into a single binary outcome, making it difficult to identify where and why the agent went wrong. To address this limitation, we introduce TRAJEVAL, a diagnostic framework that de- composes agent trajectories into three interpretable stages: search (file localization), read (function comprehension), and edit (modification tar- geting). For each stage, we compute precision and recall by comparing against reference patches. Analyzing 16,758 trajectories across three agent architectures and seven models, we find universal inefficiencies (all agents examine 22× more functions than necessary) yet distinct failure modes: GPT-5 locates relevant code but targets edits incorrectly, while Qwen-32B fails at file discovery entirely. We validate that these diagnostics are pre- dictive, achieving model-level Pass@1 prediction within 0.87–2.1% MAE, and actionable: real-time feedback based on trajectory signals improves two state-of-the-art models by 2.2–4.6 percentage points while reducing costs by 20–31%. These results demonstrate that our framework not only provides a more fine-grained analysis of agent behavior, but also translates diagnostic signals into tangible performance gains. More broadly, TRAJE- VAL transforms the agent evaluation beyond outcome-based benchmarking towards mechanism-driven diagnosis of agent success and failure. 1 Introduction Large language model (LLM)-based code agents can now autonomously resolve GitHub issues, generating patches that pass unit tests on real-world repositories. Systems such as SWE-Agent, OpenHands, and Live-SWE-Agent achieve resolution rates of roughly 65–70% on SWE-bench Verified, and commercial variants are increasingly deployed in production development workflows (Yang et al., 2024; Wang et al., 2024; Xia et al., 2025b). Most existing evaluations rely on outcome-based metrics such as Pass@k. While useful for comparing aggregate performance or controlling cost, they cannot reveal where or why an agent fails, nor explain why one successful agent is more effective than another. We argue that evaluating code agents should move beyond outcome-based metrics toward fine-grained, stage-level diagnostics. Reference patches provide a natural basis for this shift: a correct patch implicitly identifies which files must be found, which functions must be examined, and which code regions must be modified. By comparing an agent’s trajectory against this reference, we convert execution traces from unstructured logs into quantitative, stage-specific signals. This enables diagnosis of where failures arise and how even successful agents differ in their execution strategies. ∗ Work done during an internship at AWS AI Labs. 1 arXiv:2603.24631v1 [cs.SE] 25 Mar 2026 Preprint. Under review. Figure 1: Overview of TRAJEVAL. Agent execution traces are decomposed into three stages: search, read, and edit, and then compared against reference patches to compute stage-wise precision and recall, enabling task-success prediction and actionable intervention Building on this perspective, we introduce TRAJEVAL, a trajectory-based diagnostic frame- work that decomposes agent execution into three interpretable stages (Figure 1): (i) Search, capturing file-level localization; (i) Read, reflecting function-level code understanding; and (i) Edit, measuring whether modifications target the correct locations. For each stage, we compute precision and recall by comparing the agent’s trajectory against the reference patch. Precision captures efficiency (how much of the agent’s exploration was necessary), while recall captures effectiveness (how much of the required context was successfully identified). We empirically validate TRAJEVAL on SWE-bench Verified and PolyBench, ana- lyzing 16,758 trajectories across three agent architectures (SWE-Agent, OpenHands, LiveSWEAgent) and seven language models spanning 8B to 480B parameters. Across settings, simple trajectory features derived from search, read, and edit precision and recall are highly predictive of task success. Using a lightweight logistic regression model, we predict model-level Pass@1 within 0.87–2.1% mean absolute error under in-distribution eval- uation, while preserving strong ranking correlation (ρ≥0.886) under repository-held-out and language-held-out distribution shifts. Beyond prediction, trajectory-level diagnostics reveal systematic behavioral differences that are invisible to outcome-only metrics. Across all agents and model scales, we observe substantial over-exploration: precision remains low at every stage, with read stage pre- cision around 4–5%. However, inefficiency alone does not determine success. Instead, recall strongly correlates with task completion, particularly at the edit stage, where higher- performing models consistently achieve broader coverage of the necessary code regions. These diagnostics distinguish not only failed executions but also successful ones, revealing differences in how agents localize files, identify functions, and target edits even when they ultimately solve the same task. Crucially, these diagnostics are not merely descriptive. Because trajectory metrics identify when an agent has interacted with relevant code, they enable actionable intervention during execution. We implement a lightweight feedback mechanism that provides real-time signals when an agent encounters golden context (e.g., “✓You are looking at a relevant file”). This guidance improves Pass@1 by 2.2–4.6 percentage points across two state-of-the-art models, while reducing token usage by up to 29% and cost by up to 31%. These results demonstrate that trajectory-level diagnostics enable simultaneous improvements in both accuracy and efficiency, as agents converge more quickly when guided toward productive exploration. In summary, our contributions are: •We introduce TRAJEVAL, a diagnostic framework that decomposes agent trajecto- ries into search, read, and edit stages, computing precision and recall at each stage by comparison against reference patches. • We analyze 16,758 trajectories across three agent architectures and seven models, revealing that recall (not precision) predicts success and that different models exhibit distinct, diagnosable failure modes. 2 Preprint. Under review. •We demonstrate that trajectory features predict model-level Pass@1 within 0.87–2.1% MAE, with ranking correlation ρ≥ 0.886 preserved under distribution shift. •We validate that these diagnostics are actionable: trajectory-derived feedback im- proves Pass@1 by 2.2–4.6 percentage points while reducing cost by up to 31%. 2 Our Methodology We formalize agent trajectories as stage-specific sets over files, functions, and edit targets, then compare them against a reference patch to compute diagnostic precision and recall metrics. 2.1 Problem Formulation Consider a code agent tasked with resolving an issue in repositoryRgiven an issue descriptiond. The agent produces an execution trajectoryT = (a t ,o t ) n t=1 , wherea t denotes the action at step t and o t the corresponding observation, and outputs a candidate patch ˆ P. Given a ground-truth reference patchP ∗ , our goal is to extract interpretable features fromTthat (i) diagnose where agent reasoning succeeds or fails, and (i) predict task success without requiring access to P ∗ at inference time. The key insight is thatP ∗ implicitly defines the minimal context required to solve the task: which files must be located, which functions must be examined, and which code regions must be modified. We formalize this as the golden contextG = (F ∗ ,H ∗ ), whereF ∗ denotes the set of files modified by P ∗ , andH ∗ denotes the set of functions modified by P ∗ . 2.2 Three-Stage Decomposition We decompose agent behavior into three sequential stages, each operating at a distinct granularity. LetF T denote the set of files viewed during trajectoryT,H T denote the set of functions observed in agent observations, andH ˆ P denote the set of functions modified in the candidate patch. Stage 1: Search (File-Level).Search quantifies the agent’s ability to localize relevant files within the repository. Let P s and R s denote precision and recall for search: P s = |F ∗ ∩F T | |F T | , R s = |F ∗ ∩F T | |F ∗ | (1) High recall indicates successful file discovery, while high precision reflects focused explo- ration without redundant navigation. Stage 2: Read (Function-Level). File localization alone is insufficient: an agent may open a file without examining the relevant code region. Read quantifies function-level comprehension. Let P r and R r denote precision and recall for read: P r = |H ∗ ∩H T | |H T | , R r = |H ∗ ∩H T | |H ∗ | (2) This finer granularity distinguishes agents that achieve semantic understanding from those that perform superficial file scanning. Stage 3: Edit (Function-Level). Edit quantifies the localization accuracy of the agent’s code modifications. Let P e and R e denote precision and recall for edit: P e = |H ∗ ∩H ˆ P | |H ˆ P | , R e = |H ∗ ∩H ˆ P | |H ∗ | (3) Edit metrics capture localization accuracy: an agent may achieve perfect edit recall while producing an incorrect patch. Semantic correctness is evaluated separately via Pass@k. 3 Preprint. Under review. 2.3 Golden Context Hierarchy We define golden context at two levels: 1) Tier 0 (Core). Files and functions explicitly modified inP ∗ , representing the minimal modification footprint. Tier 0 comprises 1.4 files and 2.1 functions on average; 2) Tier 1 (Extended). Tier 0 augmented with one-hop structural dependencies extracted via static analysis, including imports referenced in modified lines, test files, and parent classes via inheritance. Tier 1 expands to 4.2 files on average. We leverage both tiers in complementary ways. Recall metrics are computed against Tier 0, providing a strict measure of whether the agent located essential code. Precision metrics benefit from Tier 1 context, as exploration of structurally related elements constitutes reasonable behavior rather than wasteful navigation. This hierarchical design enables a nuanced distinction between meaningful exploration and truly redundant reading. 2.4 Feature Extraction Algorithm 1 describes the trajectory feature extraction procedure. The extraction is architecture-agnostic: given an agent-specific tool mappingφ : A → view,edit,other, the comparison logic remains invariant across implementations. Algorithm 1 Trajectory Feature Extraction Require: Trajectory T =(a t , o t ) n t=1 , Reference patch P ∗ , Tool mapping φ Ensure: Feature vector m∈R 6 1: F ∗ ,H ∗ ← PARSEPATCH(P ∗ ) 2: F T ,H T ,H ˆ P ←∅,∅,∅ 3: for (a t , o t )∈ T do 4:if φ(a t ) = view then 5: F T ←F T ∪ EXTRACTFILES(a t ) 6: H T ←H T ∪ EXTRACTFUNCTIONS(o t ) 7:else if φ(a t ) = edit then 8: H ˆ P ←H ˆ P ∪ EXTRACTEDITTARGETS(a t ) 9:end if 10: end for 11: m← [P s , R s , P r , R r , P e , R e ] using Eqs. 1–3 12: Return m Function extraction employs a hybrid approach: tree-sitter parsing for accurate AST-based extraction when possible, with regex fallback for partial code snippets and edge cases. The procedure supports Python, Java, JavaScript, and TypeScript via modular specifications. Feature extraction runs in O(|T|) time, enabling real-time analysis during agent execution. 2.5 Success Prediction Model To validate that trajectory features capture meaningful predictive signal, we train per-model logistic regression classifiers: P(success| m) = σ β 0 +β ⊤ m ,(4) wherem ∈R 6 andσ(·)denotes the sigmoid function. Features are standardized to zero mean and unit variance prior to training. We apply isotonic calibration via 3-fold cross- validation to ensure predicted probabilities reflect empirical success rates. We evaluate generalization under three distribution shifts: (i) instance-held-out (60/40 stratified split), (i) repository-held-out (5-fold cross-validation across repositories), and (i) language-held-out (train on Python/JavaScript, test on TypeScript/Java). We report mean absolute error (MAE) between predicted and actual Pass@1 rates and Spearman correlation (ρ) for model ranking preservation. 4 Preprint. Under review. 3 Experiments 3.1 Experimental Setup We evaluate seven language models: GPT-5, GPT-5-mini , Qwen3-8B, Qwen3-32B, Qwen3- 235B, Qwen3-Coder-30B, and Qwen3-Coder-480B. Our evaluation is conducted on SWE- bench Verified (Yang et al., 2024), containing 500 real-world GitHub issues from 12 Python repositories with human-verified patches, and PolyBench Verified, containing 382 multilin- gual instances across Python, Java, JavaScript, and TypeScript from 20 repositories. We analyze three agent architectures that differ in their tool primitives: SWE-Agent (Yang et al., 2024), which provides specialized commands for viewing (view), editing (strreplace), and bash execution; OpenHands (Wang et al., 2024), which usesfsRead,fsWrite, and executeBash; and LiveSWEAgent (Xia et al., 2025b), which operates exclusively through shell commands without specialized file manipulation tools. This architectural diversity validates that our framework captures agent behavior rather than implementation artifacts. All experiments use random seed 42. Trajectory analysis runs in minutes on CPU; the primary cost is agent execution, performed prior to this work. In total, we analyze 16,758 trajectories: 9,500 from SWE-bench and 7,258 from PolyBench. Code and evaluation scripts are provided in supplementary materials. 3.2 Evaluation Protocol We evaluate three generalization settings with increasing distribution shift. Instance-Held- Out uses a 60/40 stratified split, preserving the resolved/unresolved ratio per model to test in-distribution prediction. Repository-Held-Out trains on 60% of repositories and tests on entirely unseen ones to evaluate cross-codebase generalization. Language-Held-Out trains on Python and JavaScript, testing on TypeScript and Java for cross-language transfer. For each model, we train a logistic regression classifier with L2 regularization (C =1.0) on the six trajectory features. The predicted Pass@1 is the mean predicted probability across test instances. We report Mean Absolute Error (MAE) between predicted and actual Pass@1, and Spearman’s ρ for ranking correlation across models. 4 Results We present three main findings: (1) trajectory features accurately predict Pass@1 across models and agents, (2) the framework generalizes under distribution shift across repositories and languages, and (3) real-time feedback based on trajectory diagnostics yields statistically significant improvements in agent performance. 4.1 Pass@1 Prediction Accuracy Table 1a presents Pass@1 prediction accuracy across all agent-benchmark configurations. Our framework achieves 1.20% mean absolute error (MAE) on SWE-bench using SWE- Agent, with perfect ranking preservation (ρ =1.00). This level of accuracy enables reliable model comparison without requiring ground-truth evaluation on held-out instances. Prediction accuracy is consistent across agent architectures despite substantial differences in tool primitives. SWE-Agent uses specialized commands (view,strreplace), OpenHands operates through browser-based file operations (fsRead,fsWrite), and LiveSWEAgent relies exclusively on bash commands. This pattern is consistent across all three agents. 4.2 Generalization Under Distribution Shift A key question is whether trajectory-based prediction generalizes beyond the training distribution. We evaluate three settings: instance-held-out (random split), repository-held- out (unseen codebases), and language-held-out (unseen programming languages). 5 Preprint. Under review. SWE-benchPolyBench AgentMAEρMAEρ SWE-Agent1.20% 1.001.48% 1.00 OpenHands1.98% 0.961.98% 1.00 LiveSWEAgent2.06% 1.000.87% 1.00 (a) Pass@1 prediction SearchReadEdit ModelPRPRPR GPT-5.12.87-.23.45.33 2.13 Qwen3-Coder-480B .21 1.03.08.89.25 1.45 Qwen3-8B.15.45-.12 1.61 .42.53 Average.16.90.02.78.24 1.22 (b) Logistic coefficients Table 1: Pass@1 prediction accuracy and logistic regression coefficients. Table 2: SWE-Agent Pass@1 prediction across evaluation settings. Instance split uses 60/40 stratified sampling; Repo split uses 5-fold cross-validation across repositories; Language split trains on Python/JavaScript and tests on TypeScript/Java. Values in parentheses show prediction bias (Predicted− Actual). SWE-bench VerifiedPolyBench Verified Model Actual † InstanceRepoActual † InstanceRepoLanguage GPT-567.269.2 (+2.0) 67.4 (+5.3)39.141.3 (+2.2) 40.5 (+2.6) 48.6 (+18.4) GPT-5-mini65.165.9 (+0.8) 66.1 (+4.6)25.622.6 (-3.0) 26.2 (+0.6) 26.2 (+6.2) Qwen3-Coder-480B62.163.3 (+1.2) 61.7 (+2.5)26.126.7 (+0.6) 29.2 (+1.3) 34.8 (+14.5) Qwen3-Coder-30B53.056.6 (+3.6) 54.4 (+0.5)19.020.8 (+1.8) 19.6 (-2.1) 24.4 (+10.3) Qwen3-235B 47.447.1 (-0.3) 49.5 (+2.7)16.215.7 (-0.5) 19.9 (+1.2) 14.7 (+2.3) Qwen3-8B20.319.7 (-0.6) 21.3 (+2.9)— Qwen3-32B17.117.8 (+0.7) 20.3 (+3.2)4.23.4 (-0.8)4.6 (+1.6)3.7 (+1.3) MAE—1.20%2.73%—1.48%1.56%8.83% Rank ρ —1.001.00—1.000.941.00 † Actual Pass@1 on 40% held-out test instances (200 for SWE-bench, 153 for PolyBench). Full dataset Pass@1 differs slightly (e.g., GPT-5: 66.0% full vs 67.2% test split). Table 2 presents complete results for SWE-Agent across all evaluation settings. Repository- held-out evaluation degrades MAE from 1.20% to 2.73% on SWE-bench—a modest increase that reflects the challenge of generalizing to unseen codebases with different coding con- ventions and project structures. Critically, ranking correlation remains perfect (ρ =1.00), indicating that the framework reliably identifies which models perform better even when absolute prediction accuracy degrades. Language-held-out evaluation on PolyBench (training on Python and JavaScript, testing on TypeScript and Java) presents the greatest challenge, with MAE increasing to 8.83%. This degradation reflects fundamental differences in how agents navigate statically-typed versus dynamically-typed codebases. Nevertheless, ranking correlation remains perfect (ρ =1.00), demonstrating that trajectory features capture transferable patterns even across programming paradigms. The largest cross-language errors occur for GPT-5 (+18.4% bias) and Qwen3-Coder-480B (+14.5% bias), suggesting these models’ behaviors transfer poorly across language paradigms, likely due to Python-centric training. Examining per-model bias patterns in Table 2 reveals systematic trends. The framework produces smaller errors for mid-tier models (Qwen3-235B:<1% bias on instance splits) and larger errors under distribution shift (language split). This pattern suggests that trajectory features are most discriminative within the same language family, where search and edit patterns vary across instances. Overall, predictions on instance-held-out achieve sub-2% MAE for all models, with error consistently below the gap between adjacent models. 4.3 Feature Importance Analysis Logistic regression coefficients reveal which trajectory features drive prediction. Table 1b presents the learned coefficients for selected models, demonstrating both universal patterns and model-specific variations. 6 Preprint. Under review. Across all models, recall metrics dominate: edit recall shows the strongest average coefficient ( ̄ β = 1.22), followed by search recall ( ̄ β = 0.90) and read recall ( ̄ β = 0.78). Precision coefficients are weaker and less stable ( ̄ β<0.3 across all stages), reflecting the universal over- exploration pattern observed in agent behavior. This asymmetry shows that interventions should prioritize recall (ensuring agents find relevant code) rather than precision (reducing unnecessary exploration). An agent that examines every file in a repository but ultimately edits the correct function succeeds, whereas an efficient agent that navigates directly to the wrong location fails. Model-specific coefficients reveal capability differences. GPT-5 relies heavily on edit recall (β =2.13), suggesting its failures occur primarily at the localization-to-modification transi- tion. Qwen3-8B shows elevated dependence on read recall (β =1.61), indicating frequent failures to comprehend relevant code even when files are correctly identified. These patterns enable targeted diagnosis: improvements for GPT-5 should focus on edit targeting, while Qwen3-8B requires better semantic comprehension. 4.4 Intervention Study Results To demonstrate that trajectory diagnostics are actionable beyond post-hoc analysis, we implement a real-time feedback mechanism that provides signals during agent execution. Implementation details. We modify the SWE-Agent harness to intercept tool observations and append feedback messages before returning to LLM: (1) after eachviewcommand, we check whether the viewed file or any functions visible in the output belong to the golden context; (2) if so, we append a single feedback line: “[SYSTEM]✓ You are examining relevant code for this issue.” We inject feedback at most once per golden context to avoid flooding the context. When all golden files have been viewed at least once, we inject: “[SYSTEM]✓ All relevant context identified---proceed to your fix.” No feedback is provided for incorrect exploration (to avoid revealing what not to look at). The feedback is confirmatory and does not reveal file names, function names, or solution content. In particular, we conduct experiments on GPT-5 and Qwen3-Coder-480B to evaluate the intervention, observing statistically significant improvements for both models: with GPT-5 gaining 4.6 percentage points (p<0.001) and Qwen3-Coder-480B gaining 2.2 percentage points (p<0.01). In relative terms, GPT-5 achieves a 7.0% gain over its baseline, while Qwen3-Coder-480B achieves 3.6%. GPT-5’s larger improvement suggests it benefits more from confirmation that exploration is progressing correctly, allowing it to transition from reading to editing with greater confidence. Qwen3-Coder-480B’s smaller gain may reflect that its failures occur earlier in the pipeline—at stages where knowing the destination is relevant provides less benefit because the model struggles to navigate there regardless. This interpretation aligns with the feature importance analysis, where GPT-5 shows stronger dependence on edit-stage features while Qwen3-Coder-480B relies more uniformly across stages. 4.5 Cross-Agent Behavioral Patterns Despite architectural differences, all three agents exhibit remarkably consistent behavioral patterns. Table 3a summarizes key metrics across agents on SWE-bench Verified. Search recall averages 0.72–0.78 across agents, indicating that locating the correct files is a shared challenge with similar difficulty regardless of tool implementation. Read recall shows more variance (0.58–0.71), with OpenHands achieving highest values, potentially due to its structured file reading interface. Edit recall ranges from 0.45–0.68, with SWE- Agent performing best, likely due to its specializedstrreplacecommand that provides explicit feedback on edit success. Complete per-model results for OpenHands (Table 4) and LiveSWEAgent (Table 5) in Appendix A.2 confirm these consistent patterns. Precision is uniformly low across all stages and agents, indicating substantial over- exploration: agents inspect 8–12×more files and 17–25×more functions than necessary. This suggests that current agents trade efficiency for coverage. 7 Preprint. Under review. MetricSWE-AgentOpenHandsLiveSWE Search Recall0.780.740.72 Read Recall0.650.710.58 Edit Recall0.680.520.45 Search Precision0.120.080.10 Read Precision0.050.060.04 Edit Precision0.250.180.15 Avg. Tool Calls30.724.213.5 Files Viewed8.29.57.8 (a) Cross-Agent Behavioral Summary. ModelPass@1Tools/TaskSubmit %View % GPT-566.0%30.799.9%99.7% GPT-5-mini60.4%32.299.9%99.9% Qwen3-Coder-480B61.2%50.399.9%99.7% Qwen3-Coder-30B52.6%50.5100%99.8% Qwen3-235B45.0%42.526.5%99.9% Qwen3-8B13.2%48.693.2%99.2% Qwen3-32B14.8%43.899.3%100% (b) Efficiency Metrics. Table 3: Behavioral patterns and efficiency metrics on SWE-bench Verified. 4.6 Failure Mode Analysis Search failures (low search recall<0.5): The agent fails to locate the relevant files within the repository. This mode is common for Qwen3-8B and Qwen3-32B on complex multi- file issues, where limited model capacity prevents effective navigation of large codebases. Remediation: improved file discovery prompts, repository structure summaries, or retrieval- augmented approaches. Read failures (high search recall>0.8, low read recall<0.5): The agent finds the correct files but examines the wrong functions within them. This mode appears when issues span multiple functions within a file and the agent fails to identify which are relevant. Edit failures (high read recall>0.8, low edit recall<0.5): The agent comprehends the relevant context but targets incorrect locations for modification. This mode is common for GPT-5 on issues requiring coordinated changes across multiple sites. Remediation: enhanced localization guidance, multi-site edit planning, or explicit dependency tracking. We present a case study of failure modes from GPT-5 on instancedjangodjango-11400, requiring modification of four functions across three files. GPT-5’s trajectory metrics were: search recall 1.00, read recall 1.00, but edit recall only 0.33. The agent understood the problem but failed to propagate changes to all necessary sites—a classic edit localization failure. In contrast, Qwen3-32B on the same instance achieved search recall 0.00, never finding the relevant files at all—a search failure requiring entirely different intervention. This model-specific diagnosis is invisible to Pass@1, which shows both as simply “failed.”. 4.7 Robustness to Alternative Correct Solutions To determine whether the reference patch represents the unique correct solution or whether alternative valid fixes may modify different code locations, we analyze 334 instances that were successfully resolved by two or more models. At the function level, 100% of multiply- resolved instances show overlapping modifications—all models edit at least one common function. 93.3% of instances show exact function-level convergence, where all successful models modify precisely the same functions. At the file level, 99.4% of instances show at least partial overlap, with only 0.6% (2 of 334) modifying completely disjoint file sets. This convergence suggests that SWE-bench issues have well-defined canonical fix locations. We observe 106 resolved instances (8.6% of successful resolutions) with low edit recall (<0.8). Manual inspection of 10 cases with edit recall=0 reveals two patterns: (1) partial completions where agents modify a subset of required functions (e.g., editing 2 of 4 golden locations), and (2) alternative solutions where agents fix the bug via completely different code paths (e.g., editingcontenttypes/fields.pywhen the reference patch modifiesfields/init.py). Both patterns successfully resolve the issue by passing all tests. 4.8 Operational Efficiency Insights Beyond prediction features, our extracted operational metrics reveal model-specific effi- ciency patterns. Table 3b presents selected efficiency metrics for SWE-Agent on SWE-bench. 8 Preprint. Under review. The most striking finding is Qwen3-235B’s anomalously low submit success rate of 26.5%, compared to>99% for most other models. This indicates systematic patch formatting issues that cause submissions to fail validation, independent of whether the patch content is correct. Such operational failures are invisible to trajectory features but critical for deployment. The view operation maintains near-perfect reliability (>99%) across all models, confirming file inspection as the most robust tool in the SWE-agent toolkit. 5 Related Work Code Generation Agents. LLM-based code agents advance from simple code comple- tion to autonomous issue resolution. SWE-Agent (Yang et al., 2024) introduced a tool- augmented interface enabling agents to navigate repositories, view files, and apply edits. OpenHands (Wang et al., 2024) extended this with browser-based interactions and structured tool primitives. LiveSWEAgent (Xia et al., 2025b) demonstrated effective issue resolution using only bash commands, without specialized file manipulation tools. Agentless (Xia et al., 2025a) showed that hierarchical localization without agent loops can achieve competitive results. Recent work has explored multi-agent systems (Hong et al., 2024), reinforcement learning from execution feedback (Le et al., 2022), and retrieval-augmented generation for code (Zhang et al., 2023). While these systems achieved impressive performance, it remains challenging to understand why they succeed or fail. Our work complements agent develop- ment by providing diagnostic tools revealing behavioral patterns in agent architectures. LLM Evaluation Metrics. Traditional Pass@k measures the probability of generating a correct solution withinkattempts (Chen, 2021). Efficiency metrics such as token consump- tion, latency, and cost per solution have been proposed for practical deployment (Yang et al., 2024). CodeBLEU (Ren et al., 2020) and similar metrics capture syntactic and semantic similarity to references. However, these metrics operate on outputs, not processes. Our trajectory-based metrics fill this gap by decomposing how agents arrive at solutions, enabling diagnosis of failure modes invisible to outcome-only evaluation. Agent Trajectory Analysis. Analysis of agent behavior has received increasing attention. AgentBench (Liu et al., 2025) evaluates agents across diverse tasks but focuses on task success rather than trajectory patterns. WebArena (Zhou et al., 2024) and MiniWoB (Shi et al., 2017) provide trajectory data for web agents. More recent work analyzed LLM reasoning traces (Wei et al., 2022), tool use patterns (Schick et al., 2023), and planning behav- iors (Huang et al., 2022). The most related work is DevBench (Golnari et al., 2026), which provides development environment metrics but lacks the stage-wise decomposition and golden context comparison. Our framework leverages reference patches to define “correct” behavior, enabling precision/recall computation without manual trajectory annotation. 6 Conclusion We introduced a trajectory-based diagnostic framework that decomposes code agent exe- cution into search, read, and edit stages. Evaluating 16,758 trajectories across three agents, seven models, and two benchmarks, we find that six trajectory features predict Pass@1 within 1.20% MAE with perfect ranking correlation—revealing that agents universally over-explore (4–8% precision) while recall determines success. These diagnostics are ac- tionable: real-time golden context feedback improves GPT-5’s Pass@1 by 4.6% (p<0.001), demonstrating that trajectory analysis identifies exploitable bottlenecks rather than merely correlational patterns. By transforming opaque agent behavior into interpretable metrics, our framework enables mechanism-driven optimization: low search recall indicates file dis- covery limitations, low edit recall suggests localization failures, and the universal precision deficit reveals systematic inefficiency across all current agent architectures. 9 Preprint. Under review. References Mark Chen.Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Pareesa Ameneh Golnari, Adarsh Kumarappan, Wen Wen, Xiaoyu Liu, Gabriel Ryan, Yuting Sun, Shengyu Fu, and Elsie Nallipogu. Devbench: A realistic, developer-informed benchmark for code generation models, 2026. URLhttps://arxiv.org/abs/2601.11895. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and J ̈ urgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VtmBAGCN7o. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Proceedings of the 39th International Conference on Machine Learning (ICML), 2022. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C.H. Hoi. Coderl: mastering code generation through pretrained models and deep reinforcement learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as agents, 2025. URLhttps: //arxiv.org/abs/2308.03688. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis, 2020. Timo Schick, Jane Dwivedi-Yu, Roberto Dess ́ ı, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: language models can teach themselves to use tools. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc. Tianlin Tim Shi, Andrej Karpathy, Linxi Jim Fan, Jonathan Hernandez, and Percy Liang. World of bits: an open-domain platform for web-based agents. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, p. 3135–3144. JMLR.org, 2017. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741, 2024. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Demystifying llm- based software engineering agents. Proc. ACM Softw. Eng., 2(FSE), June 2025a. doi: 10.1145/3715754. URL https://doi.org/10.1145/3715754. Chunqiu Steven Xia, Zhe Wang, Yan Yang, Yuxiang Wei, and Lingming Zhang. Live- swe-agent: Can software engineering agents self-evolve on the fly? arXiv preprint arXiv:2511.13646, 2025b. 10 Preprint. Under review. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024. Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian- Guang Lou, and Weizhu Chen. RepoCoder: Repository-level code completion through iterative retrieval and generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 2471–2484, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.151. URLhttps://aclanthology.org/2023.emnlp-main. 151/. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024. URLhttps://arxiv. org/abs/2307.13854. 11 Preprint. Under review. A Appendix A.1 Appendix Overview This appendix provides supplementary results supporting the main paper findings: • Section A.1: Full Pass@1 prediction tables for OpenHands and LiveSWEAgent • Section A.2–A.3: Complete efficiency metrics, token consumption, and tool success rates for all models on SWE-bench and PolyBench • Section A.4–A.5: OpenHands and LiveSWEAgent operational metrics • Section A.6: Trajectory feature extraction methodology • Section A.7: Per-model classifier coefficients and feature importance • Section A.8: Cross-benchmark and cross-language transfer analysis • Section A.9: Assumptions and limitations A.2 Full Pass@1 Prediction Tables This section provides the complete per-model Pass@1 prediction results for OpenHands and LiveSWEAgent, complementing the SWE-Agent results in the main paper (Table 2). A.2.1 OpenHands Results Table 4 presents Pass@1 prediction accuracy for OpenHands across SWE-bench and Poly- Bench. Despite different agent architecture (browser-based interface with distinct tool primitives), our framework achieves comparable prediction accuracy to SWE-Agent. Table 4: OpenHands Pass@1 prediction accuracy. The framework generalizes across agent architectures, achieving 1.98% MAE on SWE-bench and maintainingρ ≥0.93 ranking correlation across all settings. Values in parentheses show prediction bias. SWE-benchPolyBench ModelActual † InstanceRepoActual † InstanceRepoLanguage GPT-568.570.7 (+2.2)67.4 (+2.9)44.549.1 (+4.5)44.8 (+0.7)46.4 (+7.0) GPT-5-mini58.462.7 (+4.3)58.7 (+3.6)38.041.5 (+3.5)39.0 (+0.4)32.7 (+1.3) Qwen-480B 61.057.3 (-3.7)59.4 (+2.4)35.937.4 (+1.4)36.5 (+3.6)39.8 (+12.8) Qwen-30B 44.542.1 (-2.4)42.4 (-0.5)23.523.2 (-0.4)24.0 (+2.9)23.9 (+10.0) Qwen-235B 45.045.3 (+0.3)44.1 (+2.3)20.922.8 (+1.9)22.7 (+3.2)22.6 (+10.1) Qwen-8B11.011.6 (+0.6)12.1 (-1.7)— Qwen-32B11.011.4 (+0.4)12.3 (-0.4)9.29.0 (-0.2)9.9 (-0.6)9.1 (+4.9) MAE—1.98%1.97%—1.98%1.89%7.67% Rank ρ—0.9550.929—1.0001.0000.943 † Actual Pass@1 on 40% held-out test instances. A.2.2 LiveSWEAgent Results Table 5 presents results for LiveSWEAgent, which operates exclusively through bash com- mands without specialized file manipulation tools. Remarkably, our framework achieves the lowest prediction error on PolyBench (0.54% MAE for repo split), demonstrating robustness to agent implementation details. A.3 Complete Model Results: SWE-bench Verified A.3.1 Efficiency Metrics Table 6 presents complete efficiency metrics for all seven models evaluated on SWE-bench Verified. GPT-5 family demonstrates a clear efficiency gradient: GPT-5 achieves the lowest 12 Preprint. Under review. Table 5: LiveSWEAgent Pass@1 prediction accuracy. Despite operating through bash-only interactions, trajectory features achieve sub-1% MAE on PolyBench, demonstrating that our framework captures agent behavior independent of tool implementation. Values in parentheses show prediction bias (Predicted− Actual). SWE-benchPolyBench ModelActual † InstanceRepoActual † InstanceRepoLanguage GPT-565.568.2 (+2.7)65.0 (-0.5)37.438.7 (+1.3)37.8 (+0.4)37.8 (+0.4) GPT-5-mini60.559.7 (-0.8)62.9 (+2.4)27.029.6 (+2.6)27.7 (+0.7)24.6 (-2.4) Qwen-480B53.555.5 (+2.0)54.8 (+1.3)22.322.2 (-0.1)21.7 (-0.6)35.5 (+13.2) Qwen-235B29.025.1 (-3.9)30.0 (+1.0)13.914.0 (+0.1)13.7 (-0.2)20.6 (+6.7) Qwen-30B 45.045.8 (+0.8)46.9 (+1.9)8.99.1 (+0.2)9.7 (+0.8)13.0 (+4.1) MAE—2.06%1.42%—0.87%0.54%5.65% Rank ρ—1.0001.000—1.0001.0000.900 † Actual Pass@1 on 40% held-out test instances. average tool usage at 30.71 calls per task, while GPT-5 nano requires 43% more invocations (43.81 calls). This efficiency gap correlates with exploration behavior—GPT-5 nano dedicates 54.1% of bash commands to localization operations compared to 33.3% for GPT-5, suggesting smaller models compensate for reduced reasoning capability through exhaustive search. The Qwen family exhibits different patterns: Qwen-235B achieves comparable efficiency to GPT-5 mini despite lower bash usage (15.25 vs 16.84), while Qwen-30B and Qwen-480B show elevated tool counts (50.45 and 50.30) with moderate localization percentages (20.5% and 23.4%), indicating more deliberate but verbose problem-solving strategies. Table 6: Complete Efficiency Metrics (SWE-bench Verified) Agentμ tool μ LLM μ bash P pipe Loc.Loc.% GPT-530.7130.7115.360.22255833.3 GPT-5 mini32.1632.1616.840.24349341.5 GPT-5 nano43.8143.8139.190.421059554.1 Qwen-235B42.5342.5315.250.20168222.1 Qwen-30B50.4550.4527.440.42281520.5 Qwen-32B48.5948.5913.620.23246936.2 Qwen-480B50.3050.3024.800.36290023.4 A.3.2 Token Consumption by Tool Table 7 presents complete token consumption statistics for all tools across all models on SWE-bench Verified. Bash output tokens exhibit the highest variance across models, ranging from 315 tokens (Qwen-30B) to 8248 tokens (GPT-5 nano). This 26×difference reflects divergent command strategies: Qwen-30B issues precise, targeted commands while GPT- 5 nano generates verbose outputs through exploratory commands with extensive error messages. The view operation shows consistent input requirements (25.7–29.0 tokens) but substantial output variance (448–1682 tokens), indicating models inspect similar file regions but receive different context lengths based on file structure. The strreplace operation reveals model-specific editing patterns: Qwen models require higher input tokens (190–616) compared to GPT models (232–327), suggesting more verbose edit specifications, while output tokens remain consistent (96–457), reflecting similar confirmation message lengths. A.3.3 Tool Success Rates Table 8 presents complete tool success rates for all models on SWE-bench Verified. The most striking finding is Qwen-235B’s anomalously low submit success rate of 26.5% across 3557 attempts, compared to 99.9% for GPT-5 and 100% for Qwen-30B. This suggests systematic issues with patch formatting or validation logic specific to this model configuration. Qwen- 13 Preprint. Under review. Table 7: Complete Token Consumption (SWE-bench Verified) ToolGPT-5mininanoQ-235BQ-30BQ-32BQ-480B bash In76.096.8199.441.887.044.754.2 Out5210149782489063151301593 view In25.727.128.327.426.629.027.2 Out168214061223902776448715 strreplace In322327232560616190555 Out31637996418457132432 create In253228290258721235570 Out15161918152016 insert In1491119713261103— Out231244265261182234— undoedit In17.017.318.817.8—20.417.1 Out14.315.016.115.0—15.714.4 submit In5.35.35.35.35.35.35.3 Out1864351441240725836141318 32B exhibits degraded performance across multiple operations: create success drops to 78.9% (vs 98.8% for GPT-5), strreplace falls to 92.6% (vs 98.8%), and undoedit reaches only 71.7% (vs 100%). These patterns indicate Qwen-32B struggles with file manipulation operations, potentially due to training data gaps or instruction-following limitations. The view operation maintains near-perfect reliability (>99%) across all models, confirming file inspection as the most robust tool in the SWE-agent toolkit. Table 8: Complete Tool Success Rates (SWE-bench Verified) ToolGPT-5mininanoQ-235BQ-30BQ-32BQ-480B bash96.2%96.4%98.1%96.3%97.3%94.5%97.7% (7213)(7953)(19615)(7149)(13225)(6417)(11905) view99.7%99.9%100%99.9%99.8%99.2%99.7% (4549)(4200)(642)(4843)(5970)(7521)(6165) str replace98.8%98.9%97.6%97.2%99.5%92.6%98.4% (1467)(1656)(1090)(4665)(2086)(8137)(2748) create98.8%98.9%91.1%92.7%99.3%78.9%97.2% (659)(264)(45)(1035)(2901)(1816)(2493) insert100%100%100%100%100%92.6%— (94)(23)(79)(12)(1)(122) undo edit100%100%100%100%—71.7%100% (67)(15)(4)(1)(46)(2) submit99.9%99.9%99.3%26.5%100%93.2%99.9% (1305)(1897)(423)(3557)(1042)(236)(1838) A.4 Complete Model Results: PolyBench Verified A.4.1 Efficiency Metrics Table 9 presents complete efficiency metrics for all eight models evaluated on PolyBench Verified. Cross-benchmark comparison reveals that multilingual tasks increase complexity: GPT-5 requires 25% more tool invocations on PolyBench (38.38) compared to SWE-bench (30.71). The Qwen-8B anomaly stands out with unusually low tool usage (24.89) and minimal localization (4.1%), suggesting early termination or limited exploration capability rather than efficient problem-solving. Qwen-32B exhibits the highest localization percentage 14 Preprint. Under review. (71.6%), indicating extensive code navigation before editing—a pattern that correlates with its degraded tool success rates. Pipe command usage increases substantially on PolyBench, with Qwen-32B reaching 64% and GPT-5 nano at 60%, reflecting the need for complex command chaining in multilingual codebases with diverse build systems and directory structures. Table 9: Complete Efficiency Metrics (PolyBench Verified) Agentμ tool μ LLM μ bash P pipe Loc.Loc.% GPT-538.3838.3818.060.34220435.6 GPT-5 mini34.6234.6218.890.37244744.4 GPT-5 nano45.7045.7042.260.60850557.0 Qwen-235B43.1243.1218.260.41208132.3 Qwen-32B48.4548.4530.260.64764471.6 Qwen-8B24.8924.8924.890.023624.1 Qwen-30B46.0546.0523.660.57214925.7 Qwen-480B50.2150.2124.530.53243528.1 A.4.2 Token Consumption by Tool Table 10 presents complete token consumption statistics for all tools across all models on PolyBench Verified. Bash output tokens increase dramatically on PolyBench: GPT- 5 mini reaches 21460 tokens (vs 1497 on SWE-bench), a 14×increase reflecting verbose compilation errors and test output in Java/TypeScript projects. The strreplace output tokens for GPT-5 reach 1396 on PolyBench versus 316 on SWE-bench, a 4.4×increase indicating more complex edit contexts in multilingual codebases. File creation (create) requires substantially more input tokens on PolyBench (283–647 tokens) compared to SWE-bench (228–721 tokens), reflecting boilerplate requirements and import statements in statically-typed languages. The insert operation shows extreme variance for GPT-5 mini (2537 output tokens), suggesting verbose feedback when inserting code into Java/TypeScript files with complex type annotations. Table 10: Complete Token Consumption (PolyBench Verified) ToolGPT-5mininano Q-235B Q-32B Q-8B Q-30B Q-480B bash In88.7102.0188.245.064.051.753.549.2 Out12297 21460 117063983434496824721539 view In26.830.129.032.130.8—28.329.0 Out1800150116921012711—12441068 strreplace In306314209438190—597565 Out13961313105611130—544555 create In336341347293283—647625 Out6297183816—143189 insert In1612201388467—352— Out5102537278161197—518— undoedit In19.317.0—19.7— Out16.714.2—13.9— submit In5.35.35.35.35.3—5.35.3 Out36992882137564118258—496112060 A.4.3 Tool Success Rates Table 11 presents complete tool success rates for all models on PolyBench Verified. Cross- benchmark degradation is evident: Qwen-32B’s create success drops from 78.9% (SWE- 15 Preprint. Under review. bench) to 58.7% (PolyBench), and strreplace falls from 92.6% to 83.5%. The undoedit operation shows severe degradation for Qwen-32B, dropping from 71.7% to 43.8%, indicat- ing that edit state management becomes increasingly problematic in multilingual contexts. Qwen-235B’s submit success improves from 26.5% (SWE-bench) to 65.3% (PolyBench), though this remains substantially below other models, suggesting partial but incomplete resolution of patch formatting issues. Qwen-8B shows complete absence of editor tool usage (all marked as —), confirming that this model fails to engage with file manipula- tion operations and relies exclusively on bash commands. The GPT-5 family maintains robust performance across benchmarks, with view operations achieving 100% success and strreplace remaining above 98.5%. Table 11: Complete Tool Success Rates (PolyBench Verified) ToolGPT-5mininanoQ-235BQ-32BQ-8BQ-30BQ-480B bash96.2%95.9%97.6%94.5%92.9%82.8%95.9%96.3% (5921)(5266)(14886)(6152)(10514)(8786)(8043)(8368) view100%99.9%100%99.8%96.6%—99.9%99.9% (4239)(2846)(405)(4177)(2993)(4676)(5391) strreplace99.4%99.8%98.5%98.2%83.5%—99.9%99.2% (1757)(1177)(536)(2946)(3032)(1381)(1957) create99.1%99.6%100%92.4%58.7%—99.8%98.2% (450)(233)(16)(1106)(387)(1545)(1448) insert100%100%100%100%96.9%—100%— (109)(19)(36)(38)(64)(1) undoedit100%100%—43.8%— (44)(20)(16) submit99.8%98.1%99.6%65.3%94.7%—100%100% (646)(521)(245)(801)(95)(610)(561) A.5 OpenHands Results A.5.1 SWE-bench Verified Table 12 presents efficiency metrics for OpenHands on SWE-bench Verified. OpenHands demonstrates distinct behavioral patterns compared to SWE-agent: lower average tool usage (13.98–45.71 vs 30.71–50.45) but higher localization command percentages, with Qwen-32B reaching 97.6% localization. The Qwen-480B configuration requires the most tool invocations (45.71) while maintaining moderate localization (61.7%), suggesting a more balanced exploration-exploitation strategy. Pipe command usage varies substantially, with Qwen-235B showing anomalously high usage (110.2%) indicating command chaining patterns that exceed single-command baselines. Table 12: OpenHands Efficiency Metrics (SWE-bench Verified) Agentμ tool μ LLM μ bash P pipe Loc.Loc.% GPT-524.1824.1818.840.56704874.8 GPT-5 mini20.0620.0615.150.49562774.3 GPT-5 nano24.0724.0721.310.49266947.3 Qwen-235B26.2626.269.651.10365075.7 Qwen-32B13.9813.984.490.42219297.6 Qwen-8B26.5526.558.970.45418893.4 Qwen-480B45.7145.7120.610.74635461.7 Qwen-30B37.4837.4818.920.76557959.0 Table 13 presents token consumption for OpenHands tools on SWE-bench. The fsWrite operation shows substantial variance in input tokens (230–694), reflecting different code 16 Preprint. Under review. generation verbosity across models. Bash output tokens range from 195 (Qwen-480B) to 4503 (Qwen-32B), a 23×difference indicating divergent command strategies. The think tool, unique to OpenHands, shows input tokens ranging from 235 (GPT-5 nano) to 634 (Qwen-30B), capturing model-specific reasoning verbosity. Table 13: OpenHands Token Consumption (SWE-bench Verified) ToolGPT-5mininanoQ-235BQ-32BQ-8BQ-480BQ-30B executeBash In31.634.333.328.133.022.931.641.0 Out10461555370054945031962195208 fsRead In18.717.418.918.418.718.718.218.5 Out1694161020091305510500695753 fsWrite In277417246324230374593694 Out332250164267141131206244 system In31043104310435283528352835283528 Out5.85.85.85.85.85.85.85.8 think In481296235252537391479634 Out7.37.37.37.37.37.37.37.3 Table 14 presents tool success rates for OpenHands on SWE-bench. The fsWrite operation exhibits the most variance, ranging from 29.0% (Qwen-8B) to 97.7% (Qwen-480B). This 3.4×difference highlights model-specific challenges with file creation and modification. The fsRead operation shows degraded performance for GPT-5 mini (63.2%) and Qwen-32B (59.8%), suggesting file path resolution issues. System and think tools maintain 100% success across all models, confirming their reliability as non-destructive operations. Table 14: OpenHands Tool Success Rates (SWE-bench Verified) ToolGPT-5mininanoQ-235BQ-32BQ-8BQ-480BQ-30B executeBash95.4%96.3%98.5%91.5%98.0%97.0%94.9%95.1% (9418)(7575)(5647)(4823)(2246)(4483)(10304)(9462) fsRead87.4%63.2%90.0%98.2%59.8%95.5%97.8%98.2% (952)(960)(231)(2953)(3260)(4159)(7930)(5345) fsWrite89.0%74.1%45.7%76.1%44.5%29.0%97.7%95.6% (1168)(953)(232)(927)(483)(1143)(2803)(1817) system100%100%100%100%100%100%100%100% (498)(492)(263)(500)(500)(500)(500)(500) think100%100%100%100%100%100%100%100% (55)(20)(5)(3929)(501)(1964)(1317)(1618) A.5.2 PolyBench Verified Table 15 presents efficiency metrics for OpenHands on PolyBench Verified. Cross-benchmark comparison reveals increased complexity: GPT-5 requires 20% more tool invocations on PolyBench (29.03) compared to SWE-bench (24.18). Qwen-480B maintains the highest tool usage (47.45) with moderate localization (66.2%). The Qwen-30B configuration shows elevated pipe command usage (87.9%), reflecting complex command chaining requirements in multilingual codebases. Table 16 presents token consumption for OpenHands on PolyBench. Bash output tokens increase substantially: GPT-5 reaches 2329 tokens (vs 1046 on SWE-bench), while Qwen-8B shows extreme output (6628 tokens), indicating verbose error messages in multilingual com- pilation. The fsWrite input tokens remain relatively stable across benchmarks, suggesting consistent code generation patterns regardless of target language. 17 Preprint. Under review. Table 15: OpenHands Efficiency Metrics (PolyBench Verified) Agentμ tool μ LLM μ bash P pipe Loc.Loc.% GPT-529.0329.0320.180.56588476.3 GPT-5 mini25.4525.4519.590.51544972.8 GPT-5 nano37.1337.1334.680.63622260.8 Qwen-235B30.4030.4014.680.84451180.5 Qwen-32B18.0318.038.830.93278682.6 Qwen-8B25.4325.4310.980.30346382.6 Qwen-480B47.4547.4519.580.66495666.2 Qwen-30B37.8037.8017.380.88442166.6 Table 16: OpenHands Token Consumption (PolyBench Verified) ToolGPT-5 mini nano Q-235B Q-32B Q-8B Q-480B Q-30B executeBash In40.338.651.330.841.526.526.035.3 Out232930862119101612906628524623 fsRead In23.524.423.224.223.925.023.023.1 Out1516112213731348575498690823 fsWrite In311383268363279472572606 Out308253207240154124270285 system In31043104310435283528352835283528 Out5.85.85.85.85.85.85.85.8 think In688349122281138455523701 Out7.37.37.37.37.47.37.37.3 Table 17 presents tool success rates for OpenHands on PolyBench. The fsWrite operation shows degraded performance across most models: GPT-5 drops from 89.0% (SWE-bench) to 66.7% (PolyBench), and Qwen-8B falls from 29.0% to 20.6%. This cross-benchmark degradation reflects increased complexity in multilingual file manipulation. The fsRead operation also degrades, with GPT-5 mini dropping from 63.2% to 61.0%. Table 17: OpenHands Tool Success Rates (PolyBench Verified) ToolGPT-5mininanoQ-235BQ-32BQ-8BQ-480BQ-30B executeBash98.2%97.8%98.7%96.2%98.6%98.1%95.3%96.0% (7710)(7483)(10232)(5607)(3374)(4195)(7481)(6641) fsRead89.2%61.0%74.3%97.3%66.3%90.0%93.2%97.7% (1217)(677)(257)(2883)(2401)(2465)(7016)(4621) fsWrite66.7%55.4%52.5%54.5%39.8%20.6%90.5%77.8% (1722)(1140)(177)(1264)(530)(880)(2087)(1540) system100%100%100%100%100%100%100%100% (365)(375)(287)(382)(382)(380)(382)(382) think100%100%100%100%100%100%100%100% (75)(47)(1)(1478)(200)(824)(1158)(1255) A.6 Live SWE Agent Results A.6.1 SWE-bench Verified Table 18 presents efficiency metrics for Live SWE Agent on SWE-bench Verified. Live SWE Agent operates exclusively through bash commands, with tool usage equal to bash usage across all models. The GPT-5 family shows efficient operation (13.45–14.67 calls per task), 18 Preprint. Under review. while Qwen models require substantially more invocations (34.63–50.56). Localization percentages range from 50.9% (Qwen-30B) to 66.4% (Qwen-480B), indicating moderate exploration overhead. Pipe command usage is notably higher for GPT-5 models (74.2–74.8%) compared to Qwen-235B (26.5%), suggesting different command composition strategies. Table 18: Live SWE Agent Efficiency Metrics (SWE-bench Verified) Agentμ tool μ LLM μ bash P pipe Loc.Loc.% GPT-513.4513.4513.450.75422462.8 GPT-5 mini14.6714.6714.670.74454561.9 Qwen-235B43.7643.7643.760.261444766.0 Qwen-30B42.3842.3842.380.441078250.9 Qwen-32B34.6334.6334.630.21896851.8 Qwen-480B50.5650.5650.560.211679366.4 Table 19 presents token consumption for Live SWE Agent on SWE-bench. Bash input tokens range from 79.8 (Qwen-30B) to 158.8 (GPT-5), reflecting different command verbosity. Output tokens show extreme variance: Qwen-235B generates 14688 tokens per command on average, compared to 128 tokens for Qwen-32B—a 115×difference indicating fundamentally different execution patterns, likely due to verbose error handling or extensive test output. Table 19: Live SWE Agent Token Consumption (SWE-bench Verified) ToolGPT-5miniQ-235BQ-30BQ-32BQ-480B executeBash In158.896.7122.379.888.097.5 Out52843014688817128152 Table 20 presents tool success rates for Live SWE Agent on SWE-bench. Bash success rates range from 92.9% (Qwen-30B) to 98.2% (Qwen-480B). The GPT-5 family maintains consistent performance (94.3–95.8%), while Qwen models show more variance. The high success rates across all models confirm bash command execution as a reliable operation, though the variance suggests model-specific command formatting issues. Table 20: Live SWE Agent Tool Success Rates (SWE-bench Verified) ToolGPT-5miniQ-235BQ-30BQ-32BQ-480B executeBash94.3%95.8%97.8%92.9%97.9%98.2% (5434)(6010)(20880)(17701)(16418)(24474) A.6.2 PolyBench Verified Table 21 presents efficiency metrics for Live SWE Agent on PolyBench Verified. Cross- benchmark comparison shows mixed patterns: GPT-5 requires slightly more invocations on PolyBench (15.23 vs 13.45), while Qwen-32B shows reduced usage (16.45 vs 34.63). Localiza- tion percentages increase for most models on PolyBench, with Qwen-480B reaching 68.9%, reflecting additional navigation requirements in multilingual codebases. Pipe command us- age increases substantially for GPT-5 (88.5% vs 74.8%), indicating more complex command chaining for multilingual tasks. Table 22 presents token consumption for Live SWE Agent on PolyBench. Bash input tokens remain relatively stable across benchmarks, with GPT-5 at 148.8 tokens (vs 158.8 on SWE- bench). Output tokens decrease substantially for Qwen-235B (230 vs 14688), suggesting more efficient command execution in multilingual contexts, possibly due to different error handling patterns or reduced test verbosity. Table 23 presents tool success rates for Live SWE Agent on PolyBench. Success rates remain high across all models (93.3–97.9%), with Qwen-30B achieving the highest rate (97.9%) 19 Preprint. Under review. Table 21: Live SWE Agent Efficiency Metrics (PolyBench Verified) Agentμ tool μ LLM μ bash P pipe Loc.Loc.% GPT-515.2315.2315.230.89390667.1 GPT-5 mini14.2814.2814.280.86367467.3 Qwen-235B25.1025.1025.100.31629965.7 Qwen-30B42.3342.3342.330.46770347.6 Qwen-32B16.4516.4516.450.29264249.0 Qwen-480B46.7846.7846.780.261231168.9 Table 22: Live SWE Agent Token Consumption (PolyBench Verified) ToolGPT-5miniQ-235BQ-30BQ-32BQ-480B executeBash In148.895.5143.689.380.4101.1 Out501466230131167191 despite moderate efficiency. GPT-5 mini shows slight degradation (93.3% vs 95.8% on SWE-bench), while Qwen-32B maintains consistent performance (97.5% vs 97.9%). These results confirm bash-only operation as a robust strategy across benchmark types. Table 23: Live SWE Agent Tool Success Rates (PolyBench Verified) ToolGPT-5miniQ-235BQ-30BQ-32BQ-480B executeBash94.5%93.3%95.3%97.9%97.5%97.6% (4653)(4235)(8657)(14358)(4716)(17184) A.7 Trajectory Feature Extraction Methodology This section details the feature extraction pipeline used to compute the six trajectory metrics. A.7.1 Golden Context Definition We define golden context as the minimal code elements required to generate the correct patch: 1.Golden Files: Files modified in the ground-truth patch, extracted from diff headers (diff --git a/... b/...) 2. Golden Functions: Functions containing modified lines, extracted from hunk headers (@@ ... @@ functionname) 3. Golden Lines: Line ranges modified in each file, extracted from hunk markers A.7.2 Agent Action Extraction For each agent framework, we extract actions from trajectory logs: SWE-Agent Actions: • view: File opened for inspection→ files viewed • strreplace: Edit operation→ filesedited • bash: Commands like cat, grep→ files viewed 20 Preprint. Under review. OpenHands Actions: • fsRead: File read operation→ filesviewed • fsWrite: File write operation→ filesedited • executeBash: Shell commands parsed for file references LiveSWEAgent Actions: • executeBash: All operations via bash commands • Parse cat, head, vim, sed for filesviewed • Parse patch, sed -i, file redirects for filesedited A.7.3 Metric Computation Given golden context G and agent actions A: Search Precision = |G files ∩ A viewed | |A viewed | (5) Search Recall = |G files ∩ A viewed | |G files | (6) Read Precision = |G funcs ∩ A funcs read | |A funcsread | (7) Read Recall = |G funcs ∩ A funcs read | |G funcs | (8) Edit Precision = |G funcs ∩ A funcsedited | |A funcsedited | (9) Edit Recall = |G funcs ∩ A funcsedited | |G funcs | (10) A.8 Per-Model Classifier Coefficients Table 24 presents the learned logistic regression coefficients for each model. The coefficient β indicates the log-odds contribution of each feature to task success. Table 24: Per-Model Logistic Regression Coefficients (β) for SWE-Agent on SWE-bench. Positive values indicate positive correlation with task success. Edit recall is universally positive across all models. Modelβ searchP β searchR β readP β readR β editP β editR GPT-5+0.12+0.87-0.23+0.45+0.33+2.13 GPT-5-mini+0.08+0.92+0.15+0.67-0.11+1.87 Qwen-480B+0.21+1.03+0.08+0.89+0.25+1.45 Qwen-30B-0.05+1.61+0.22+0.53+0.18+0.98 Qwen-235B+0.33+0.78+0.61+0.42-0.08+0.87 Qwen-8B+0.15+0.45-0.12+1.61+0.42+0.53 Qwen-32B+0.28+0.63-0.57+0.88+0.67+0.71 Average+0.16+0.90+0.02+0.78+0.24+1.22 Key Observations: • Edit recall universally positive: All models showβ editR >0 (range: +0.53 to +2.13), confirming that successfully targeting correct code locations is essential 21 Preprint. Under review. •Recall dominates precision: Average recall coefficients (+0.90, +0.78, +1.22) exceed precision coefficients (+0.16, +0.02, +0.24) • Read precision unstable: Ranges from -0.57 (Qwen-32B) to +0.61 (Qwen-235B), suggesting model-specific value of contextual reading •Model-specific patterns: GPT-5 relies heavily on edit recall (β =2.13), while Qwen-8B depends more on read recall (β = 1.61) A.9 Cross-Benchmark and Cross-Language Transfer Analysis A.9.1 Benchmark Transfer Table 25 compares prediction accuracy when training on one benchmark and testing on another. Table 25: Cross-Benchmark Transfer: Training on SWE-bench, Testing on PolyBench AgentSame-Bench MAECross-Bench MAEDegradation SWE-Agent1.48%3.82%+2.34% OpenHands1.98%4.21%+2.23% LiveSWEAgent2.06%3.95%+1.89% The 1.9–2.3% degradation when transferring from SWE-bench (Python-only) to PolyBench (multilingual) indicates that while trajectory features capture fundamental patterns, some benchmark-specific adaptation improves accuracy. A.9.2 Language Transfer Analysis Table 26 shows prediction accuracy across different train/test language splits. Table 26: Language Transfer Matrix: MAE (%) for Different Train/Test Splits Train→ TestPythonJavaTypeScriptJavaScript Python1.69%5.42%6.18%5.87% Java4.93%2.14%4.51%4.78% TypeScript5.67%4.23%1.92%3.45% JavaScript5.21%4.89%3.12%1.85% Key Findings: •In-language prediction is best: Diagonal entries (1.7–2.1%) substantially outper- form off-diagonal (3.1–6.2%) • TypeScript/JavaScript transfer well: Due to syntactic similarity (3.1–3.5% MAE) •Python most isolated: Python→TypeScript shows largest gap (6.2%), reflecting different programming paradigms •Java intermediate: Transfers moderately well to TypeScript (4.5%) due to shared static typing A.9.3 Feature Importance by Language Table 27 shows which features are most predictive for each programming language. Java’s reliance on searchrecall reflects the importance of navigating complex package structures, while TypeScript’s dependence on readrecall suggests type annotation compre- hension is critical. 22 Preprint. Under review. Table 27: Top Feature by Language (Highest|β|) LanguageTop Featureβ Pythoneditrecall+1.87 Javasearchrecall+1.45 TypeScriptread recall+1.23 JavaScripteditrecall+1.56 Table 28: Complete Dataset Statistics MetricSWE-benchPolyBench Total instances500382 Languages1 (Python)4 (Py/Java/TS/JS) Unique repositories1220 Avg files per patch1.41.6 Avg functions per patch2.12.4 Avg lines changed4238 Models evaluated77 Agents evaluated33 Total trajectories9,5007,258 A.10 Dataset Statistics A.11 Qualitative Case Studies To illustrate how our framework diagnoses agent failures, we present three concrete case studies from GPT-5 on SWE-bench Verified. These examples demonstrate that different failure modes are distinguishable through trajectory metrics. A.11.1 Case 1: Edit Localization Failure Instance: django django-11400 Metrics: • Search: Precision 0.38, Recall 1.00 (found all 3 golden files) • Read: Precision 0.04, Recall 1.00 (read all 4 golden functions) • Edit: Precision 1.00, Recall 0.33 (edited only 2 of 4 needed functions) Analysis: The agent successfully located and read all relevant code but failed at edit targeting. It found the filesfilters.py,init.py, andreverserelated.py, read all four golden functions, but only editedfilters.py:choicesandfilters.py:hasoutput, missing the getchoices functions in the fields modules. Diagnosis: This is an edit localization failure—the agent understood the problem context but failed to identify all locations requiring modification. Low edit recall (<0.5) with high read recall (>0.9) is the signature pattern for this failure mode. A.11.2 Case 2: Over-Exploration Success Instance: django django-10880 Metrics: • Search: Precision 0.17, Recall 1.00 (viewed 6 files for 1 golden file) • Read: Precision 0.01, Recall 1.00 (viewed 90 functions for 1 golden function) • Edit: Precision 0.50, Recall 1.00 (task resolved) 23 Preprint. Under review. Analysis: The agent successfully resolved the task despite extremely low precision—it explored 6×more files and 90×more functions than necessary. The single golden file was django/db/models/aggregates.py. Diagnosis: This exemplifies the universal over-exploration pattern we observe. Success does not require efficiency; agents can compensate for low precision with thorough exploration. However, this comes at significant cost (token usage, latency). Our framework quantifies this waste: 0.01 read precision means 99% of function reads were unnecessary. A.11.3 Case 3: Read Comprehension Failure Instance: django django-10554 Metrics: • Search: Precision 0.33, Recall 1.00 (found all golden files) • Read: Precision 0.00, Recall 0.50 (read only 1 of 2 golden functions) • Edit: Precision 0.00, Recall 0.00 (no edits attempted) Analysis: The agent found the relevant files (compiler.py,query.py) and read the func- tiongetorderby, but missedclearselectfields. Without understanding the complete context, it made no edit attempts. Diagnosis: This is a read comprehension failure—the agent found the right files but didn’t examine all necessary functions. Low read recall (<0.8) with high search recall indicates the agent needs better context selection within files. A.11.4 Case 4: Cross-Model Comparison Instance: djangodjango-11400 (same as Case 1) Model Comparison: ModelSearch RRead REdit RResolved GPT-51.001.000.33X Qwen-32B0.000.000.00X Analysis: On the same instance, GPT-5 and Qwen-32B fail for completely different reasons. GPT-5 found all relevant code but edited incompletely (edit localization failure). Qwen-32B failed at the earliest stage—it never found the relevant files (search failure). Diagnosis: Different models require different interventions. GPT-5 would benefit from edit targeting guidance, while Qwen-32B needs improved file discovery. This model-specific diagnosis is invisible to Pass@k, which shows both as “failed.” A.12 Assumptions and Limitations Reference Patch Assumption. We assume the reference patch represents the canonical solution. Our convergence analysis (93.3% exact function-level match, 100% partial overlap) covers 334 instances solved by multiple models. A key question is whether harder, unsolved instances exhibit greater solution diversity. We cannot directly measure this since unsolved instances lack successful agent patches to compare. However, SWE-bench instances were designed to have well-defined solutions verified by test suites, suggesting canonical fix locations exist even for hard instances. Future work could incorporate alternative reference patches where available in the dataset, enabling multi-reference evaluation that computes recall against the union of valid fix locations. Function Extraction. Function extraction uses tree-sitter parsing with regex fallback for robustness. While tree-sitter achieves high accuracy on well-formed code, the regex fallback may miss complex constructs such as nested functions, decorators, or metaprogramming 24 Preprint. Under review. patterns. The hybrid approach trades some accuracy for broad language support and handling of partial code snippets in trajectory observations. Intervention Study Design. Our intervention study has several design limitations: (1) Golden context oracle: Feedback requires knowing the reference patch, unavailable at deployment—practical systems would require learned predictors or retrieval-based approx- imations; (2) Coarse feedback: Our binary signal is intentionally minimal—richer feedback could provide stronger guidance; (3) Limited model coverage: We evaluated two models (GPT-5, Qwen3-480B) due to computational constraints. Despite these limitations, the study demonstrates proof of concept that trajectory diagnostics identify actionable bottlenecks. Benchmark Scope. Our evaluation focuses on SWE-bench and PolyBench, emphasiz- ing bug fixes in well-maintained open-source repositories. Generalization to other task types (security vulnerabilities, performance optimization) or codebases (proprietary, poorly documented) requires further validation. A.13 Reproducibility All trajectory analysis code, feature extraction pipelines, and evaluation scripts are available in the supplementary materials. Key parameters: • Train/test split: 60/40 for instance and repo splits; 50/50 for language split • Classifier: Logistic regression with L2 regularization (C = 1.0) • Metrics: Mean Absolute Error (MAE) and Spearman’s ρ for ranking correlation • Random seed: Fixed at 42 for reproducibility 25