Paper deep dive
Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering
Jingyue Li, AndrĂŠ Storhaug
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%
Last extracted: 4/2/2026, 11:50:47 PM
Summary
This paper analyzes 18 recent software engineering papers to evaluate the current state of Agentic AI research. It identifies significant challenges in reproducibility, explainability, and cost-effectiveness due to the black-box nature of LLMs and inconsistent reporting. The authors propose a framework centered on the public release of Thought-Action-Result (TAR) trajectories and demonstrate a proof-of-concept using automated LLM-based summarization to compare agent performance across different models.
Entities (5)
Relation Signals (2)
Jingyue Li â authored â Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering
confidence 100% ¡ Jingyue Li and AndrÊ Storhaug. 2026. Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering.
Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering â proposes â Thought-Action-Result (TAR) trajectories
confidence 95% ¡ this position paper proposes a set of guidelines and recommendations... we recommend that Agentic AI researchers make their Thought-Action-Result (TAR) trajectories... publicly accessible.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:With the advancement of Agentic AI, researchers are increasingly leveraging autonomous agents to address challenges in software engineering (SE). However, the large language models (LLMs) that underpin these agents often function as black boxes, making it difficult to justify the superiority of Agentic AI approaches over baselines. Furthermore, missing information in the evaluation design description frequently renders the reproduction of results infeasible. To synthesize current evaluation practices for Agentic AI in SE, this study analyzes 18 papers on the topic, published or accepted by ICSE 2026, ICSE 2025, FSE 2025, ASE 2025, and ISSTA 2025. The analysis identifies prevailing approaches and their limitations in evaluating Agentic AI for SE, both in current research and potential future studies. To address these shortcomings, this position paper proposes a set of guidelines and recommendations designed to empower reproducible, explainable, and effective evaluations of Agentic AI in software engineering. In particular, we recommend that Agentic AI researchers make their Thought-Action-Result (TAR) trajectories and LLM interaction data, or summarized versions of these artifacts, publicly accessible. Doing so will enable subsequent studies to more effectively analyze the strengths and weaknesses of different Agentic AI approaches. To demonstrate the feasibility of such comparisons, we present a proof-of-concept case study that illustrates how TAR trajectories can support systematic analysis across approaches.
Tags
Links
- Source: https://arxiv.org/abs/2604.01437v1
- Canonical: https://arxiv.org/abs/2604.01437v1
Trouble viewing inline? Open PDF directly â
Full Text
35,041 characters extracted from source content.
Expand or collapse full text
Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering Jingyue Li jingyue.li@ntnu.no Norwegian University of Science and Technology Trondheim, Norway AndrĂŠ Storhaug andre.storhaug@ntnu.no Norwegian University of Science and Technology Trondheim, Norway Abstract With the advancement of Agentic AI, researchers are increasingly leveraging autonomous agents to address challenges in software engineering (SE). However, the large language models (LLMs) that underpin these agents often function as black boxes, making it dif- ficult to justify the superiority of Agentic AI approaches over base- lines. Furthermore, missing information in the evaluation design description frequently renders the reproduction of results infeasible. To synthesize current evaluation practices for Agentic AI in SE, this study analyzes 18 papers on the topic, published or accepted by ICSE 2026, ICSE 2025, FSE 2025, ASE 2025, and ISSTA 2025. The analysis identifies prevailing approaches and their limitations in evaluating Agentic AI for SE, both in current research and poten- tial future studies. To address these shortcomings, this position paper proposes a set of guidelines and recommendations designed to empower reproducible, explainable, and effective evaluations of Agentic AI in software engineering. In particular, we recommend that Agentic AI researchers make their ThoughtâActionâResult (TAR) trajectories and LLM interaction data, or summarized ver- sions of these artifacts, publicly accessible. Doing so will enable subsequent studies to more effectively analyze the strengths and weaknesses of different Agentic AI approaches. To demonstrate the feasibility of such comparisons, we present a proof-of-concept case study that illustrates how TAR trajectories can support systematic analysis across approaches. CCS Concepts ⢠General and referenceâEmpirical studies;⢠Software and its engineeringâReusability; Empirical software validation;⢠Computing methodologiesâ Artificial intelligence. Keywords Agentic AI, Empirical software engineering, Responsible AI, Re- search methodology ACM Reference Format: Jingyue Li and AndrĂŠ Storhaug. 2026. Reproducible, Explainable, and Effec- tive Evaluations of Agentic AI for Software Engineering. In 34th ACM Joint European Software Engineering Conference and Symposium on the Founda- tions of Software Engineering (FSE Companion â26), July 05â09, 2026, Montreal, FSE Companion â26, Montreal, QC, Canada Š 2026 Copyright held by the owner/author(s). This is the authorâs version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE Companion â26), July 05â09, 2026, Montreal, QC, Canada, https://doi.org/10.1145/3803437.3805548. QC, Canada. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/ 3803437.3805548 1 Introduction Agentic AI technologies are increasingly being adopted by software engineering (SE) researchers. At ICSE 2025, only seven research track papers included the keyword âagentâ in their titles. In contrast, the number of accepted papers for ICSE 2026 featuring âagentâ in their titles has risen to 30 1 . Although Agentic AI technologies have demonstrated superior performance in addressing SE challenges compared to classical approaches, such as static and dynamic code analysis, and machine learning techniques, including deep neural networks, the underlying reasons for this superiority are often difficult to justify and reproduce. This challenge stems from the black-box nature of large language models (LLMs) and the inher- ent randomness in their outputs. Furthermore, querying LLMs for comprehensive evaluations can be computationally expensive. As Agentic AI approaches gain popularity, they are likely to become standard baselines for future studies. Consequently, it is imperative to establish methods for evaluating Agentic AI approaches in a reproducible, explainable, and efficient manner. To address these challenges, we first analyzed a set of papers [1â7,9â16,20,21,23] published or accepted in recent prestigious software engineering conferences, including ICSE 2025, ICSE 2026, FSE 2025, ISSTA 2025, and ASE 2025. The selection of papers is based on their relevance and availability at the time of this paperâs submission. For conferences with published proceedings by the time of submitting this paper, such as ICSE 2025, FSE 2025, and ISSTA 2025, we selected papers containing the keyword âagentâ in their titles. For ICSE 2026 and ASE 2025, whose proceedings have not yet been published, we selected papers from the accepted paper lists and included only those with preprints available as of 12 January 2026. This paper selection is not intended to be exhaustive or fully representative. Rather, the goal is to obtain a preliminary understanding of the current state of empirical evaluations in SE studies employing Agentic AI approaches, identify potential chal- lenges, and propose guidelines for future improvements. Results of the analysis show that: ⢠Most current evaluations of Agentic AI approaches compare these methods against classical techniques, deep learning models, or naive LLMs as baselines. Only one out of the 18 investigated papers included comparisons with relevant state-of-the-art Agentic AI baselines. 1 https://conf.researchr.org/track/icse-2026/icse-2026-research-track (accessed 12 Jan- uary 2026) arXiv:2604.01437v1 [cs.SE] 1 Apr 2026 FSE Companion â26, July 05â09, 2026, Montreal, QC, CanadaJingyue Li and AndrĂŠ Storhaug ⢠Many studies employing Agentic AI approaches conducted ablation experiments to assess the contribution of individual components to overall performance. â˘Several studies performed failure analyses and case studies to gain deeper insights into the factors influencing the success or failure of their Agentic AI approaches. ⢠A subset of studies incorporated cost analyses to provide evidence regarding the cost-effectiveness of their proposed Agentic AI approaches. The analysis also indicates that several parameters can signif- icantly influence evaluation outcomes, including the versions of the LLMs, temperature configurations, prompt templates, and the inherent randomness of LLM outputs. Although some studies have conducted experiments to examine the impact of these parameters, the majority have not. Accordingly, this study proposes guidelines, strategies, and methodologies to enhance future evaluation of Agen- tic AI in software engineering: ⢠publishing the prompts used, temperature configurations, and LLM versions for improving reproducibility. ⢠making the TAR trajectories and LLM interaction data, or summarized versions of these artifacts, openly available and automatically analyzed to enable explainable, cost-effective comparisons across different Agentic AI approaches. To illustrate the practicality of using TAR data for comparing approaches, we presented an example method that analyzes agentsâ TAR trajectories per case and then performs cross-case comparisons as meta-analysis. In contrast to the classical approach, where studies evaluate agents on specific benchmarks independently and publish only the aggregate results, our proposed method enables more fine-grained, explainable, and cost-effective comparisons. The remainder of this paper is organized as follows. Section 2 describes the process used to identify and analyze the chosen papers and results. Section 3 presents our proposed approach and provides examples. Section 4 discusses our ideas. Finally, Section 5 concludes the paper and outlines future work directions. 2 Data Collection and Analysis As explained, the purpose of our analysis of existing papers apply- ing Agentic AI for software engineering is to gain a preliminary understanding of the current state of practice in evaluating pro- posed approaches. Based on the title containing the keyword âagentâ and the availability of papers in the proceedings or preprints by January 12, 2026, we analyzed the following papers from recent top software engineering conferences. ⢠ICSE 2025 [1, 2, 9, 13â15]. â˘ICSE 2026 (the accepted papers with preprints available) [6, 11, 23]. ⢠FSE 2025 [22]. ⢠ISSTA 2025 [4, 7, 21]. â˘ASE 2025 (the accepted papers with preprints available) [5, 10, 20]. For each paper, the study identified and summarized its research focus, the Agentic AI approaches employed, the evaluation design, the baselines used, and the analyses conducted to enhance the reproducibility, explainability, and effectiveness of the evaluations. The results are categorized and synthesized as follows. Research focuses. These papers aim at addressing different software engineering challenges, including: testing or auditing [4,7,9,11â13,15,23], software design [1], program repair [2,21], code generation [14], error analysis [6], bug reproducing [16,20], automated building [22], automated specification generation [10], and effort estimation [5]. The broad scope of software engineering topics covered highlights the relevance and growing interest in applying Agentic AI approaches, underscoring the need for com- prehensive studies to evaluate these methods. Agentic AI approaches applied. Eleven of the eighteen studies investigated [1,5â7,9â11,13,15,20,23] employed multi-agent approaches, while the remaining studies focused on developing and applying a single agent to address software engineering problems. Baselines. Six of the investigated studies [2,12,13,20,21,23] em- ployed classical approaches as baselines and assessed performance differences using well-established benchmark datasets. One study [5] compared an Agentic AI approach with deep learning methods. To evaluate Agentic AI approaches, several studies compared their performance against naĂŻve prompt-based LLMs [1,7,14,16,22] or fine-tuned models [15]. In [11], which focuses on agent devel- opment, the evaluation centered on comparing the performance of multiple LLMs underlying the agents. For studies addressing novel challenges without prior work [4,9,10], baselines included human-curated datasets, system execution logs, or expert judg- ments. Among all the investigated studies, only one [6] adopted an existing Agentic AI approach as a baseline, indicating that com- parative evaluations of such methods are still in their infancy and require further research to establish guidelines and best practices. Extra studies performed in the evaluation. Beyond perfor- mance comparisons between Agentic AI approaches and their base- lines, the most common evaluation technique was ablation analysis. Thirteen of the eighteen studies conducted ablation analyses to examine the contributions of individual agents or tools within the Agentic AI pipeline. To enhance the explainability of the results, several studies [6,7,10,16,20] have analyzed the failures of the proposed approaches to better understand the agentsâ challenges and limitations. Other works [2,7,13,20] present case studies that illustrate why Agentic AI approaches succeed in practice. To im- prove reproducibility, most studies provide prompt examples or raw data, although not all do so. Because the temperature setting of LLMs can influence their outputs, Batole et al. [1]and Lin et al. [14]conducted LLM temperature sensitivity analyses, demonstrat- ing that different temperature configurations yield varying results. Furthermore, Ke[12], Kim et al. [13], and Shah et al. [16]reported the specific temperature settings used in their experiments. To miti- gate the randomness in LLM outputs during evaluations, studies such as [10,13] perform multiple repetitions and report averaged results. While all investigated studies disclose the names of the LLMs employed, only a few [10,14,21] provide precise version iden- tifiers, such as GPT-3.5-0125 in [21]. To demonstrate the efficiency of Agentic AI approaches, analyses of time complexity and API costs were conducted in [1], while LLM token costs were examined in [4, 21, 23]. Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software EngineeringFSE Companion â26, July 05â09, 2026, Montreal, QC, Canada 3 Evaluation Guideline and Strategy Proposals The data analysis results underscore several weaknesses in current practices for evaluating Agentic AI approaches. To enhance the reproducibility, explainability, and efficiency of evaluations in the era of Agentic AI for software engineering, this study proposes the following guidelines, strategies, and methodologies for discussion. Reproducibility Several existing studies have attempted to mitigate the limitations of LLMsâ black-box nature and output ran- domness by publishing the prompts used, temperature config- urations, and LLM versions. However, not all papers adhere to these practices. Collecting and publishing such information intro- duces minimal overhead and can significantly enhance the repro- ducibility of the evaluations. Thus, reporting the values of these parameters should be established as a standard requirement within the software engineering community for Agentic AI approaches. Explainability. While ablation studies, failure analyses, and examples of success offer some insight into the strengths and weak- nesses of proposed Agentic AI approaches, they are insufficient to provide a systematic and comprehensive explanation of their superiority over baseline methods. Because Agentic AI approaches are perceived as novel, researchers often emphasize their positive aspects. However, Xia et al. [19]presented evidence suggesting that agent-based approaches may not outperform agentless alternatives. Given the early stage of Agentic AI research, there are currently few Agentic AI approaches that serve as state-of-the-art baselines. As the popularity of Agentic AI continues to grow, it is essential to develop innovative methodologies that systematically assess the strengths and weaknesses of these approaches in comparative evaluations. Inspired by [3], which compared three Agentic AI approaches using agentsâ Thought-Action-Result (TAR) trajectories and LLM interactions, we propose to enforce open access to agent TAR trajectories and develop methods for automatically analyz- ing these trajectories to effectively improve the explainability of Agentic AI evaluations. Since TAR trajectories can be automatically generated as execution logs, creating them introduces minimal over- head. Publishing these trajectories as open-access data alongside research papers would enable future researchers to better under- stand why a proposed approach achieves its observed performance. Moreover, when multiple Agentic AI approaches aim to address the same problem, automated analysis and comparison of their trajectories could provide highly interpretable insights into the rel- ative strengths and weaknesses of the approaches under evaluation. Currently, trajectory analysis remains a manual process [3]. Thus, developing automated techniques for such analyses represents a valuable direction for future research in evaluating Agentic AI ap- proaches for software engineering. The study by [8] demonstrated that LLMs can accurately summarize the intent of malicious code. As a result, leveraging summarization models 2 to automatically analyze TAR trajectories may represent a promising solution. Efficiency Evaluating Agentic AI approaches can be costly, par- ticularly when interaction with commercial LLM APIs is required. Although running evaluations multiple times to report averaged 2 https://huggingface.co/docs/transformers/en/tasks/summarization results is considered good practice, such repetition imposes high ad- ditional costs for researchers attempting to reproduce the findings. Instead of relying solely on replicating baselines and comparing performance metrics (such as accuracy, false positives, and false negatives, etc.), which may encourage cherry-picking and remain susceptible to LLM output randomness, a more robust and efficient alternative is for both baseline and subsequent approaches to publish their TAR trajectory data. This approach would en- able qualitative comparison and justification of the strengths and weaknesses of the methods under evaluation. If such comparisons can be automated, the cost of assessing baselines against newly proposed approaches could be significantly reduced. This, in turn, would encourage researchers to focus on developing novel and valid ideas rather than spending excessive time reproducing the results of baseline approaches, which, in practice, may not always be fully reproducible. To demonstrate the idea, we provide an example of such an au- tomated comparison. We use the openly available TAR trajectories 3 from [17] that uses an agent to identify CVE (Common Vulnera- bilities and Exposures) vulnerability fixes. They run their experi- ments using different models, including Qwen3-235B, Llama-3.3- 70B-Instruct, and Gemma-3-27B. Each model represents an agent. To illustrate how TAR trajectories can be leveraged to analyze dif- ferences in agent performance, we compare a subset of agentsâ TAR trajectories across the three models. We select TAR trajectories from 10 randomly sampled failed runs of vulnerability-fix detection using the Qwen3-235B model and compare them with the corre- sponding TAR trajectories from agents using the Gemma-3-27B and Llama-3.3-70B-Instruct models on the same 10 runs. Although the agent using the Qwen3-235B model failed to detect the cor- rect vulnerability fixes in these 10 runs, the agents powered by the Gemma-3-27B and Llama-3.3-70B-Instruct models succeeded in detecting some of them. This highlights the importance of un- derstanding the strengths and limitations of different agents when they are applied to the same task. To automatically analyze and compare the TAR trajectories, we use Kimi K2.5 Instant [18]. One challenge in analyzing TAR trajec- tories is that they can be very long. To address this challenge, we applied multi-step summarization to manage their size and com- plexity. As shown in Figure 1, starting from raw TAR trajectories, we automatically performed three-step analyses: ⢠Step 1: summarize individual runs ⢠Step 2: generate comparative analyses between the agents across the same runs â˘Step 3: aggregate these comparisons to extract recurring strengths and weaknesses The prompts we used to analyze the TAR trajectories are shown in Figure 2 (for Step 1), Figure 3 (for Step 2), and Figure 4 (for Step 3). The results of the analysis are shown in Figure 5. The results show that automated trajectory analysis can surface structured, system-specific insights, including: 1) Agentsâ strengths, e.g., âLlama-3.3-70B consistently outperforms others by checking foun- dational facts, first dates, versions, CVE metadata, before diving into code analysis.â 2) The weaknesses of the agents, e.g., âPattern match- ing without validation is the dominant failure mode.â 3) Their key 3 https://huggingface.co/datasets/andstor/favia_trajectories FSE Companion â26, July 05â09, 2026, Montreal, QC, CanadaJingyue Li and AndrĂŠ Storhaug Agent using Qwen3-235BAgent using Gemma-3-27B Agent using Llama-3.3-70B-Instruct Llama-3.3-7 0B-Instruct Gemma-3 -27B Qwen3-2 35B TAR data comparison results of each run across agents Summarized TAR data of the agentsRaw TAR data of the agents Step 1: summarize the TAR data Step 2: compare across the same runs Step 3: summarize the compari- son results Final results Legend The TAR data of the run that failed to detect the vulnerability x Summarized TAR data of the run that failed to detect the vulnerability x Comparison results of TAR data of the run across agents Summarized TAR data of the run that succeeded to detect the vulnerability x The TAR daya of the run that succeeded to detect the vulnerability x Figure 1: The process to analyze TAR trajectories Prompt Template Analyze the following trace of a security analysis session and provide a small summary of why the agent 'succeeded' if resolved else'failed' the task. == role at iteration 0 == trace_0 == role at iteration n == trace_n Figure 2: Prompt template for summarizing individual run from iteration 0 to n. differentiator, e.g., âLlamaâs 70B parameters outperform Qwenâs 235B not through deeper reasoning chains, but through stricter adherence to verification protocols: check dates, check versions, check affected files, check tool outputs.â The results illustrate the feasibility of using LLM-based summarization to analyze TAR trajectories to produce interpretable, qualitative evaluations of agent behavior at scale. Prompt Template An expert analyst have written summaries below. They are summaries of three agentic traces for classifying patch commits for CVEs. They classify the same commit for the same CVE, but use different LLMs as the engine for the agent. Your task is to compare the models performance. How did the type of model affect the classification outcome? It is not how the summaries are written, but the actual agent performance that is important. What makes one better than the other? Summarize the differences in a short paragraph. == Qwen3-235B-A22B-Instruct-2507 == summary_qwen == gemma-3-27b-it == summary_gemma == Llama-3.3-70B-Instruct == summary_llama Figure 3: Prompt template for comparative analyses between the agents on the same runs. Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software EngineeringFSE Companion â26, July 05â09, 2026, Montreal, QC, Canada Prompt Template An expert analyst has written the summaries below comparing agentic traces for CVE patch commit classification using different LLMs as the underlying engine. Your task is to identify strengths and weaknesses across the models, cross-cutting patterns, etc. Focus on what consistently works well, what consistently fails, and any recurring behavioral or reasoning patterns. Provide short, to-the-point paragraphs summarizing your findings. == summary_1 == == summary_n === Figure 4: Prompt template for aggregate analysis of compar- isons 1 to n. 4 Discussions When comparing Agentic AI approaches for software engineering tasks, researchers often evaluate different agents using the same benchmark datasets in baseline studies. Our example suggests that comparing TAR trajectories directly could serve as an alternative strategy that reduces the cost and effort of rerunning baseline ap- proaches. However, this requires baseline studies to make their TAR trajectories openly availableâan expensive requirement given that TAR trajectories can be extremely large. A more practical alter- native is for baseline studies to release only the summaries of their TAR trajectories for each run. In this setup, researchers execute Step 1 of Section 2 locally to generate TAR trajectories summaries, which are then shared as open-access artifacts. Comparative evalu- ations of Agentic AI approaches can then focus on Steps 2 and 3 in Section 2, enabling efficient, scalable analysis without requiring full TAR trajectories. One potential threat is the possibility of deceptive or biased summarized TAR outputs. Given the studiesâ prompts, temperature settings, LLM version, and summarization prompts, any biases introduced during summarization can be examined. If a study aims to compare multiple Agentic AI approaches us- ing benchmark datasets that differ from those used in the baseline papers, relying solely on TAR trajectories or their summaries may yield misleading conclusions, as agent behavior can vary signifi- cantly across datasets. In such cases, baseline studies should provide sufficient detail to support reproducibility of agent behavior, in- cluding prompts, temperature configurations, and the specific LLM versions employed. Nevertheless, making the TAR trajectories, or their summarized forms, open access can still help researchers ef- fectively compare agentsâ behavior across different datasets and better understand their dataset-dependent characteristics. One po- tential threat to validity arises when using agents to analyze agent trajectories, as this may introduce bias into the results. To mitigate this risk, employing multiple agents to cross-validate the findings can help improve the overall reliability of the analysis. 5 Conclusion and Future Work As software engineering researchers increasingly adopt Agentic AI approaches to address complex challenges, ensuring that evalua- tions of these approaches are reproducible, explainable, and efficient is essential. This paper first analyzes recent high-quality studies investigating Agentic AI in software engineering and summarizes their evaluation designs and practices. The analysis identifies sev- eral issues related to current evaluation methods. Based on these findings, this study proposes guidelines and strategies, particu- larly leveraging TAR trajectories, to enhance the reproducibility, explainability, and efficiency of evaluations. To demonstrate the feasibility of LLM-based summarization for automatically analyzing TAR trajectories, we present a case study. As future work, we plan to conduct more extensive empirical studies to compare Agentic AI approaches in software engineering, aiming to develop more systematic and effective methods for analyzing TAR trajectories. 6 Data Availability The data from our experiments are available at https://github.com/ andstor/agentic-ai-eval-replication-package. References [1] Fraol Batole, David OBrien, Tien N. Nguyen, Robert Dyer, and Hridesh Rajan. 2025. An LLM-Based Agent-Oriented Approach for Automated Code Design Issue Localization. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (Ottawa, Ontario, Canada) (ICSE â25). IEEE Press, 1320â1332. doi:10.1109/ICSE55347.2025.00100 [2] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. RepairAgent: An Autonomous, LLM-Based Agent for Program Repair. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (Ottawa, Ontario, Canada) (ICSE â25). IEEE Press, 2188â2200. doi:10.1109/ICSE55347.2025.00157 [3]Islem Bouzenia and Michael Pradel. 2025. Understanding Software Engineer- ing Agents: A Study of Thought-Action-Result Trajectories. arxiv,:2506.18824 (accepted by ICSE2026) [cs.LG] https://arxiv.org/abs/2506.18824 [4] Islem Bouzenia and Michael Pradel. 2025. You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects. Proc. ACM Softw. Eng. 2, ISSTA, Article ISSTA047 (June 2025), 23 pages. doi:10.1145/3728922 [5] Thanh-Long Bui, Hoa Khanh Dam, and Rashina Hoda. 2025. An LLM-based multi-agent framework for agile effort estimation. arxiv,:2509.14483 (accepted by ICSE2026) [cs.LG] https://arxiv.org/abs/2509.14483 [6]Zhi Chen, Wei Ma, and Lingxiao Jiang. 2025. Beyond Final Code: A Process- Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios. arxiv,:2503.12374 (accepted by ICSE2026) [cs.LG] https://arxiv.org/ abs/2503.12374 [7] Antoine Chevrot, Alexandre Vernotte, Jean-RĂŠmy Falleri, Xavier Blanc, Bruno Legeard, and Aymeric Cretin. 2025. Are Autonomous Web Agents Good Testers? Proc. ACM Softw. Eng. 2, ISSTA, Article ISSTA010 (June 2025), 23 pages. doi:10. 1145/3728879 [8]Eduard Andrei Cristea, Petter Molnes, and Jingyue Li. 2025. MalCVE: Malware Detection and CVE Association Using Large Language Models. arxiv,:2510.15567 (accepted by FORGE 2026) [cs.LG] https://arxiv.org/abs/2510.15567 [9]Venkata Sai Aswath Duvvuru, Bohan Zhang, Michael Vierhauser, and Ankit Agrawal. 2025. LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systems. In Proceedings of the IEEE/ACM 47th Interna- tional Conference on Software Engineering (Ottawa, Ontario, Canada) (ICSE â25). IEEE Press, 385â397. doi:10.1109/ICSE55347.2025.00223 [10]Yu-Fu Fu, Meng Xu, and Taesoo Kim. 2025. Agentic Specification Generator for Move Programs. arxiv,:2509.24515 (accepted by ICSE2026) [cs.LG] https: //arxiv.org/abs/2509.24515 [11]Sina Gogani-Khiabani, Ashutosh Trivedi, Diptikalyan Saha, and Saeid Tizpaz- Niari. 2025. An LLM Agentic Approach for Legal-Critical Software: A Case Study for Tax Prep Software. arxiv,:2509.13471 (accepted by ICSE2026) [cs.LG] https://arxiv.org/abs/2509.13471 [12]Kaiyao Ke. 2025. NIODebugger: A Novel Approach to Repair Non-Idempotent- Outcome Tests with LLM-Based Agent. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (Ottawa, Ontario, Canada) (ICSE â25). IEEE Press, 1014â1025. doi:10.1109/ICSE55347.2025.00226 FSE Companion â26, July 05â09, 2026, Montreal, QC, CanadaJingyue Li and AndrĂŠ Storhaug Aggregated Analysis ## Cross-Model Analysis: Agentic CVE Classification ### What Consistently Works Well **Temporal and contextual verification** emerges as the strongest success pattern. Llama-3.3-70B consistently outperforms others by checking foundational facts first-dates, versions, CVE metadata-before diving into code analysis. This "verify then analyze" discipline prevents the rabbit holes that trap other models. When Llama succeeds, it's because it treats timeline impossibilities and version mismatches as hard stops rather than details to work around. **Structured tool use with recovery** also works reliably. Gemma-3-27b demonstrates that rigorous, step-by-step tool application (CVE retrieval -> code search -> evidence synthesis) produces accurate results when executed properly. Llama's ability to recover from initial tool errors and still complete its analysis shows that resilience matters as much as capability. ### What Consistently Fails **Pattern matching without validation** is the dominant failure mode. Both Qwen3 and Gemma repeatedly fixate on superficial keyword overlaps-"CSRF" in message and CVE, "SQL injection" near "escaping," "normalize" implying security-without verifying functional relevance. This creates false confidence: models assign 5/5 ratings while missing six-year temporal gaps or conflating entirely different vulnerability classes (TLS padding vs. UTF-8 encoding). **Tool abandonment under pressure** appears across traces. When tools fail or return errors, models don't pause-they hallucinate. Qwen3 fabricates CVE details when retrieval fails; others skip required verification steps entirely. The availability of tools doesn't guarantee their use when reasoning shortcuts seem available. **Infinite examination loops** plague Qwen3 specifically. Despite strong analytical frameworks, it gets trapped iterating without conclusion, suggesting that reasoning depth without task adherence is worse than shallow but directed analysis. ### Recurring Behavioral Patterns **Epistemic humility varies dramatically**. Llama regularly acknowledges uncertainty and cross-checks evidence; Qwen and Gemma exhibit overconfidence bias, pressing forward with conclusions despite contradictory signals. This isn't about model size-Qwen's 235B parameters don't prevent basic logical errors like examining the wrong file version. **Architectural reasoning gaps** appear consistently. Models struggle to trace data flow from vulnerability entry points to modified code, often analyzing commits in isolation rather than verifying they address the actual CVE attack surface. Gemma shows flashes of coverage awareness (checking all seven vulnerable paths), but this rarely translates to execution. **Misdirection by commit messages** is a systemic weakness. Models repeatedly trust commit summaries over code reality, leading to backwards logic (interpreting relaxed controls as added security) or dismissal of relevant changes (ignoring path normalization as "just Windows handling"). ### Key Differentiator The decisive factor is **analytical discipline over reasoning sophistication**. Llama's 70B parameters outperform Qwen's 235B not through deeper reasoning chains, but through stricter adherence to verification protocols: check dates, check versions, check affected files, check tool outputs. Sophisticated analysis of the wrong question-Qwen's hallmark-produces confident wrong answers. Methodical checking of the right questions produces correct ones, even with imperfect technical explanations. Figure 5: Final results of the TAR trajectory analysis [13] Myeongsoo Kim, Tyler Stennett, Saurabh Sinha, and Alessandro Orso. 2025. A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM- Driven Inputs. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (Ottawa, Ontario, Canada) (ICSE â25). IEEE Press, 1409â1421. doi:10.1109/ICSE55347.2025.00179 [14] Feng Lin, Dong Jae Kim, and Tse-Hsun (Peter) Chen. 2025. SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents. IEEE Press, 1527â1539. https://doi.org/10.1109/ICSE55347.2025.00140 [15]Wei Ma, Daoyuan Wu, Yuqiang Sun, Tianwen Wang, Shangqing Liu, Jian Zhang, Yue Xue, and Yang Liu. 2025. Combining Fine-Tuning and LLM-Based Agents for Intuitive Smart Contract Auditing with Justifications. IEEE Press, 1742â1754. https://doi.org/10.1109/ICSE55347.2025.00027 [16]Mehil B Shah, Mohammad Masudur Rahman, and Foutse Khomh. 2025. Imita- tion Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agent. arxiv,:2512.14990 (accepted by ICSE2026) [cs.LG] https://arxiv.org/abs/2512.14990 [17]AndrĂŠ Storhaug, Jiamou Sun, and Jingyue Li. 2026. Favia: Forensic Agent for Vulnerability-fix Identification and Analysis. arXiv:2602.12500 [cs.SE] https: //arxiv.org/abs/2602.12500 [18]Kimi Team. 2026. Kimi K2.5: Visual Agentic Intelligence. arXiv:2602.02276 [cs.CL] https://arxiv.org/abs/2602.02276 [19]Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. De- mystifying LLM-Based Software Engineering Agents. Proc. ACM Softw. Eng. 2, FSE, Article FSE037 (June 2025), 24 pages. doi:10.1145/3715754 [20]Eray YapaÄcÄą, Yavuz Alp Sencer ĂztĂźrk, and Eray TĂźzĂźn. 2025. Agents in the Sandbox: End-to-End Crash Bug Reproduction for Minecraft. arxiv,:2503.20036 (accepted by ICSE2026) [cs.LG] https://arxiv.org/abs/2503.20036 [21]He Ye, Aidan Z.H. Yang, Chang Hu, Yanlin Wang, Tao Zhang, and Claire Le Goues. 2025. AdverIntent-Agent: Adversarial Reasoning for Repair Based on Inferred Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software EngineeringFSE Companion â26, July 05â09, 2026, Montreal, QC, Canada Program Intent. Proc. ACM Softw. Eng. 2, ISSTA, Article ISSTA062 (June 2025), 23 pages. doi:10.1145/3728939 [22]Zhengmin Yu, Yuan Zhang, Ming Wen, Yinan Nie, Wenhui Zhang, and Min Yang. 2025. CXXCrafter: An LLM-Based Agent for Automated C/C++ Open Source Software Building. Proc. ACM Softw. Eng. 2, FSE, Article FSE116 (June 2025), 23 pages. doi:10.1145/3729386 [23]Jie Zhu, Chihao Shen, Ziyang Li, Jiahao Yu, Yizheng Chen, and Kexin Pei. 2025. Lo- cus: Agentic Predicate Synthesis for Directed Fuzzing. arxiv,:2508.21302 (accepted by ICSE2026) [cs.LG] https://arxiv.org/abs/2509.13471