Paper deep dive
Challenges and Future Directions in Agentic Reverse Engineering Systems
Salem Radey, Jack West, Kassem Fawaz
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 4/18/2026, 1:33:24 AM
Summary
This paper provides a comprehensive analysis of agentic systems for binary reverse engineering (RE), categorizing them into static, dynamic, and hybrid analysis. It identifies critical limitations in current systems, including token constraints, susceptibility to obfuscation, lack of security guardrails, and challenges with real-time dynamic analysis. The authors propose future research directions, such as byte-pattern tokenization, deobfuscation components, and robust safety guardrails for autonomous agents.
Entities (7)
Relation Signals (4)
PentestGPT â employs â Human-in-the-loop
confidence 95% · PentestGPT represents a human-in-the-loop system
HackSynth â isa â Autonomous System
confidence 95% · HackSynth is completely automated.
Agentic Reverse Engineering Systems â utilizes â Static Analysis
confidence 90% · We divide binary RE into three categories: static, dynamic, and hybrid analysis.
Agentic Reverse Engineering Systems â utilizes â Dynamic Analysis
confidence 90% · Dynamic analysis, on the other hand, exhibits its own set of issues.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Agentic systems built on large language models (LLMs) are increasingly being used for complex security tasks, including binary reverse engineering (RE). Despite recent growth in popularity and capability, these systems continue to face limitations in realistic settings. Cutting-edge systems still fail in complex RE scenarios that involve obfuscation, timing, and unique architecture. In this work, we examine how agentic systems perform reverse engineering tasks with static, dynamic, and hybrid agents. Through an analysis of existing agentic tool usage, we identify several limitations, including token constraints, struggles with obfuscation, and a lack of program guardrails. From these findings, we outline current challenges and position future directions for system designers to overcome from a security perspective.
Tags
Links
- Source: https://arxiv.org/abs/2604.14317v1
- Canonical: https://arxiv.org/abs/2604.14317v1
Trouble viewing inline? Open PDF directly â
Full Text
44,200 characters extracted from source content.
Expand or collapse full text
Challenges and Future Directions in Agentic Reverse Engineering Systems Salem Radey, Jack West, Kassem Fawaz Abstract Agentic systems built on large language models (LLMs) are increasingly being used for complex security tasks, including binary reverse engineering (RE). Despite recent growth in popularity and capability, these systems continue to face limitations in realistic settings. Cutting-edge systems still fail in complex RE scenarios that involve obfuscation, timing, and unique architecture. In this work, we examine how agentic systems perform reverse engineering tasks with static, dynamic, and hybrid agents. Through an analysis of existing agentic tool usage, we identify several limitations, including token constraints, struggles with obfuscation, and a lack of program guardrails. From these findings, we outline current challenges and position future directions for system designers to overcome from a security perspective. 1 Introduction Agentic reverse engineering (RE) systems are collections of agents aimed to solve a given security task [36]. Each RE agent performs selected tasks in response to the systemâs needs at a given time. These tasks range from red-teaming [30], blue teaming [22], anomaly detection [8], and many other security tasks [36]. One of the most necessary capabilities for almost all RE problems is binary RE. Binary RE is the process of defining a binaryâs functionality using its compiled code. There are three methods in security research for which a binary can be fully explored: (1) static analysis, (2) dynamic analysis, and (3) hybrid analysis. Static analysis attempts to define the inner workings of a binary without executing the program itself [9]. Dynamic analysis, on the other hand, is the process of analyzing the binary while it is running. Finally, hybrid analysis combines static and dynamic analysis to analyze a binary. Modern security research [47, 6] that uses RE often employs hybrid analysis, as it captures almost all functionality of a given binary. Agents in a system are compartmentalized to perform one or more of the three binary types [11, 18, 30]. However, the technical details of how these systems implement these processes are opaque. There is no all-encompassing algorithm to RE an arbitrary binary [14]. Depending on how the binary is constructed, it could completely change the tool set and approach an agent must make. For example, Android apps require two different decompilers to fully examine a given app across two different runtimes [32]. Therefore, understanding the technical capabilities of an agentic system will define what binaries it can or cannot RE. In this work, we aim to define the current technical details for how agentic systems perform binary RE and lay out future directions and existing challenges that must be overcome. To define the details, we analyze open-source libraries for agentic RE systems to determine precisely how these systems perform binary RE. We divide binary RE into three categories: static, dynamic, and hybrid analysis. We also discuss 6 challenges and limitations that agents currently experience. We find that agents that perform static analysis are limited in capability. There are two main challenges agents currently face: (1) obfuscation and (2) tokenization. Obfuscated binaries cause confusion and unneeded complexity. Several studies have shown that obfuscation affects agent analysis [42] or that agents avoid obfuscated binaries entirely [12]. However, in realistic security scenarios, obfuscated binaries are common [42]. Binaries, especially after decompilation, are token-heavy [45, 32]. Limited tokens constrain the coverage an agent may have. Dynamic analysis, on the other hand, exhibits its own set of issues. We identify three core challenges: (1) lack of dynamic analysis guardrails, (2) timeouts, and (3) reliance on emulation. Upon reviewing codebases of existing agentic systems [10, 30, 49], we found that they commonly allow models to execute any command the agent deems necessary. RE tools are extremely powerful pieces of software, often requiring root-level access into compartmentalized memory spaces [21, 15, 13]. Recent work has shown [43, 31] that it is possible to trick an agent into executing malicious code. Thus, we argue that it may be possible to craft a binary that tricks a model into executing malicious code. Adversarial programs commonly employ timeouts to prevent processes from being monitored [2]. Given the time and latency agents naturally have, we anticipate that binaries will employ strict timeouts to block any dynamic analysis attempts. Adversarial applications also load dynamic binaries that may be encrypted or downloaded from an off-device source, further complicating the timing of an agentâs analysis. When performing safe dynamic analysis, experts commonly load malicious programs into virtual machines (VMs). Agents also analyze binaries within virtual environments as well for safe analysis [10, 49]. However, unlike their human RE counterparts, agents can be tricked into executing malicious code [44]. Furthermore, several virtual environments for edge devices have virtualization issues [50]. Finally, hybrid analysis is when an RE performs both static and dynamic analysis. Hybrid analysis shares the core challenges of static and dynamic analyses but also exhibits unique issues. The main challenge we identify in our research is that hybrid agents rely on a human-in-the-loop or perform all reasoning internally [40, 30, 49, 10]. Both approaches have their own issues. For example, human-in-the-loop is slow and requires an active RE to execute the instructions. Whereas, completely automated systems reason independently and may be susceptible to adversarially crafted binaries. We focus on automation issues since the end goal of an agentic system is to remove human dependency. In this work, we propose several research directions for static analysis agents, informed by our expert understanding of reverse engineering. First, we propose tokenizing static analysis based on byte patterns to lower the number of tokens the model may see. We also posit that exploring alternative decompilation methods within an agentic system could optimize an agentâs understanding of a given binary. Both byte-pattern filtering and ML model binary translation may result in lower tokenization. Furthermore, we argue that obfuscation must be accounted for in agentic RE systems. Work has shown that obfuscation reduces an agentâs capabilities; therefore, we must incorporate a deobfuscation component into the pipeline. Regarding dynamic analysis, we also provide researchers with directions for future exploration to improve agent performance. Our analysis reveals that agentic systems designers rely heavily on the modelâs judgment for executing commands. This lack of guardrails could lead to system hijacking by constructing a clever binary. Therefore, we present two avenues for research: (1) an adversarial direction to create a binary that takes control of the host system by utilizing the lack of guardrails, and (2) building guardrails for dynamic analysis. Another set of problems in dynamic analysis includes timeouts and the lack of sequential processing. Both issues may limit an agentâs ability to perform dynamic analysis. Thus, researchers should design specialized tools to overcome the limitations of sequential real-time data processing for agents. Finally, for hybrid agents, we propose a stronger reasoning methodology to prevent the execution of malicious code. Current systems lack a voting infrastructure and safety checks to prevent an agent from performing dynamic analysis. Hybrid systems are extremely vulnerable to malicious code as they interact with it. Thus, further verification methods across static and dynamic analysis should be explored. 2 Taxonomy In this section, we detail our taxonomy when discussing reverse engineering. Our focus will be on agents that can analyze a binary on a given device. Meaning, the agent has full access to a binary and unlimited resources to perform any type of analysis. 2.1 Reverse Engineering We examine papers that attempt to automate reverse engineering using agents. We define reverse engineering as the process of iteratively interacting with an arbitrary computer program. Specifically, we focus on reverse-engineering a given binary across multiple architectures. Therefore, our analysis will exclude reverse engineering that attempts to exploit binaries off a device on which the given agent exists. Reverse engineering can be described into three different types: (1) static, (2) dynamic, and (3) hybrid analysis. Static analysis is the process of examining a binary without executing it on the device. Dynamic analysis, on the other hand, is analyzing a binary while it is running. Hybrid analysis combines static and dynamic analysis to evaluate a binary. 2.2 AI Agent and Agentic Scope We define an AI agent as an LLM designed to solve a task, equipped with the tools to do so [40]. LLM agents plan and execute their own individual decisions by observing environmental feedback [36]. Agentic systems are collections of agents aiming to work together to accomplish a goal. In this work, we examine an agentic system where the goal is to reverse engineer a given binary. Rather than defining a threat model, we define the capabilities and goals of an arbitrary agent. First, we assume that the agent has no access to the source code of a given binary. We argue that the source code of a binary greatly simplifies RE tasks and is unrealistic for real-world security analysis. Our focus will purely be on local binary analysis. Some RE tasks require networking capabilities, such as port scanning or network traffic analysis. We do not discuss any RE task that requires networking. We assume no restrictions on the size, target device, and capabilities for a given binary. In all agents and agentic systems, we observed that they all decompile binaries. Decompiled binaries attempt to translate the bytes into a more human-readable form. Tools such as Ghidra [20] and IDA [23] translate binary code into a higher-level language. Agents that study decompiled binaries first preprocess the binaries to make them easier to read. We do not restrict agentic systems to a set of tools; any decompilation method is valid. 3 Overview of Agents This section presents a technical analysis of how agents perform various types of binary reverse engineering. Our focus is on the technical capabilities of the agents rather than their overall performance or goal. Several surveys have explored the diverse applications of LLM agents [36, 48, 34, 25], but our work focuses on their use in binary analysis from a security perspective. The focus on technical capabilities is intended to clarify how these agents perform binary reverse engineering. We segment agents into two categories: static analysis and hybrid analysis. Static analysis agents are common because static analysis can assist with a wide range of tasks beyond RE. Therefore, the communityâs understanding of static analysis agents exceeds that of hybrid agents. Our literature review found that no agent focuses on dynamic analysis; they would perform hybrid analysis. Thus, our discussion pertaining to hybrid analysis will detail current dynamic analysis capabilities. An overview of these capabilities can be seen in Figure 1. Figure 1: An overview of agent capabilities by type. 3.1 Static Analysis Agents Static analysis is an essential component of any binary reverse engineering pipeline and is, therefore, commonly performed by agents [11, 40, 31, 24, 7, 18, 26, 27, 52, 4, 45]. We found that agents perform static analysis on decompiled binaries. 3.1.1 Decompiled Code Binary decompilation is a common practice in reverse engineering [46]. The purpose of decompilation is to provide the human reverse engineer a readable interpretation of a given binary. The human reverse engineer then reads and interprets the outputs to determine the programâs goal. We find that decompilation is most commonly performed using Ghidra [20] or IDA Pro [23]. However, some studies also seek to train LLM agents to perform decompilation. Tan et al. [42] introduce LLM4Decompile, the first series of open source LLM trained for decompilation. They introduce both LLM4Decompile-End models which do decompilation directly, as well as LLM4Decompile-Ref models which refine Ghidra output. LLM4Decompile-End uses ObjDump to disassemble binaries prior to decompilation. Likewise, Rong et al. [35] present disasLLM, an end-to-end LLM disassembler for highly obfuscated executables. The system performs an initial disassembly and then uses an LLM-based classifier to validate decoded instructions. To improve efficiency, disasLLM batches LLM queries and outputs the corrected disassembly after all instructions are checked. Unlike LLM4Decompile, disasLLM is end-to-end and does not interact with traditional decompilers such as Ghidra or IDA Pro. Other works explore web runtimes and languages such as WebAssembly (WASM). She et al. [39] present WaDec, an approach that converts Wasm binaries to the textual wat format and performs block-wise decompilation using a fine-tuned LLM. By segmenting functions into Wasm loop blocks and applying a unified variable renaming scheme, WaDec improves scalability and readability, producing decompiled code with minimal code inflation. Other agents turn to JADx [41] when reverse engineering Android dex and APK files. Qian et al. [32] introduce LAMD: an LLM-powered framework that utilizes key context extraction and tier-wise code reasoning to detect malware more accurately than standalone LLMs. The LAMD framework uses existing models, with the paper evaluating GPT-4o-mini. However, since GPT-4o-mini and many other LLM cannot process APKs directly, the authors use JADx to decompile the code as a pre-processing step before sending it to the LLM. Since Android APK files consist of both C and Java code [3], there is additional pre-processing overhead to account for the differing methods required to decompile each language. Most studies tend to use IDA Pro or Ghidra. Shang et al. [37] introduce BinMetric, a benchmark of 1000 questions pertaining to common real-world binary analysis tasks. To obtain data to feed the LLM, the authors compile and strip the source code, then use IDA Pro to decompile and disassemble the compiled binaries. Although they choose to use IDA Pro, the authors note the prevalence of tools such as Ghidra, Radare2 [21], and Clang Static Analyzer [9] as static analysis tools. However, they note that the tools require significant manual effort and expert knowledge to use. Additionally, the authors use the SrcML tool [29] to extract information from the source files and convert it into XML format as additional data. 3.2 Hybrid Analysis Hybrid analysis combines static and dynamic analysis to leverage the strengths of both approaches when reverse engineering. We define a hybrid analysis agent as an agent that can use tools for both static and dynamic analysis. All agents we observed that perform dynamic analysis do so in tandem with static analysis. In this section, we provide an overview of dynamic agent capabilities and discuss hybrid implementations. 3.2.1 Dynamic Analysis Agents Dynamic analysis is the process of observing code behavior at runtime. Due to dynamic code loading [2], many behaviors of a program are not inferable through static analysis and are only observed during runtime. To observe these behaviors, reverse engineers use tools for dynamic analysis, including debuggers and dynamic instrumentation. Dynamic analysis allows a reverse engineer to define complex behaviors using captured values, execution traces, and breakpoints. Despite the advantages of dynamic analysis, it remains more difficult to scale than static analysis. Currently, the interactive and adaptive components of dynamic analysis make it difficult for LLMs to perform independently. Therefore, human analysts must still account for a large portion of the work [5]. Additionally, pipelines used for dynamic analysis are tightly coupled to the codeâs architecture, making portability a challenge [47, 2, 6]. An example tool used for dynamic analysis is the GNU Debugger (GDB) [16, 51, 1]. GDB allows for step-by-step instruction-level analysis of a binary. Beyond stepping through code, GDB also features disassembly, which can be done at runtime, decompilation, the ability to view the contents of registers, and breakpoints [28]. Zhuo et al. [53] present CTF-Dojo, an executable runtime designed to train cybersecurity agents using CTF challenges. CTF-Dojo builds upon EnIGMA+ [19, 51] for its agent, which implements dynamic reverse engineering capabilities using GDB. EnIGMA+ implements GDB only with the commands break, which adds a breakpoint, continue, which continues execution of the binary, stepi, which steps to a specific line, and debug_exec, which allows for arbitrary GDB command execution[10]. Disassembly and decompilation are handled separately with Ghidra. Likewise, Muzsai et al. [30] introduce HackSynth, an LLM agent capable of autonomous penetration testing. HackSynthâs architecture consists of a planner and summarizer that iteratively generate and interpret bash commands within a containerized Linux environment. Unlike CTF-Dojo, HackSynth uses any bash command available to the agent. 3.2.2 Hybrid Implementations While dynamic analysis agents appear only in hybrid systems, there are many ways to design a hybrid agent. There are two methods to perform hybrid analysis: (1) human-in-the-loop and (2) fully autonomous. Human-in-the-loop systems execute the agentsâ instructions and verify their feasibility and security. The humans in this process are expected to run the commands so that the agent can execute commands safely. Whereas autonomous systems are left completely to their own devices when performing analysis. Meaning, all of the reasoning for RE comes from the collection of agents communicating. Deng et al. [11] present PentestGPT, an LLM-driven framework for automated pentesting. Similar to HackSynth, PentestGPT comprises multiple agents. Whereas HackSynth used a planner and summarizer, PentestGPT employs reasoning, generation, and parsing agents. PentestGPT does not enable tool usage; rather, it operates with a human-in-the-loop strategy, in which a human expert serves solely as an executor, strictly following the LLMâs suggestions. PentestGPT represents a human-in-the-loop system whereas HackSynth is completely automated. Another case of an agentic hybrid approach is by Pasquini et al. [31], which creates MANTIS: a defense against LLM-driven cyberattacks. MANTIS operates autonomously, responds to detected anomalous interactions, and employs both active (dynamic) and passive (static) defenses. The static component analyzes the attackerâs goals, constraints, and control logic to misdirect them. The dynamic component instruments a counterattack by leading the adversarial LLM to open a reverse shell on the adversaryâs machine. 4 Current Challenges Reverse engineers face a broad range of challenges that stem from the increasing complexity of modern software. Obfuscation techniques, large-scale code bases, and dynamic runtime behavior all serve to complicate RE. These methods fundamentally limit the understanding an analyst can glean from a programâs bytes, structure, and behavior. As with many technical disciplines, reverse engineering is continually evolving. As RE improves, so do anti-RE techniques hindering models. 4.1 Static Analysis Challenges Static reverse engineering comes with a set of inherent challenges that analysts face regardless of their expertise. Techniques such as code obfuscation through symbol stripping, variable name changes, control flow changes, and comment removal obscure code semantics and impede understanding for both human and LLM reverse engineers. Additionally, the code bases of reversing targets are often large, which requires additional time and understanding to rationalize. 4.1.1 Obfuscation Just as in non-agentic reverse engineering, obfuscation remains a hurdle for agentic systems. For agentic systems, obfuscation provides new challenges. Since LLM-based agents rely on their ability to detect syntax patterns they have been trained to recognize, previously unseen or unconventional obfuscation techniques disrupt the agentsâ ability to function optimally [42]. This increases the likelihood of misclassifications or an incorrect prioritization of code importance. Tan et al. [42] find that Control Flow Flattening (CFF) and Bogus Control Flow (BCF) both decrease the success rate of decompilation by over 70%, showing that even with common obfuscation techniques, there exists a drastic decrease in reversing accuracy. This implies that complex obfuscation techniques could drop accuracies even lower. As a result, many works dealing with binary code choose to forgo including obfuscated binaries altogether, since they pose challenges for LLMs [38, 37, 5]. However, closed-source real-world applications commonly employ obfuscation techniques, meaning that when given an obfuscated binary, these agents are likely to fail. 4.1.2 Tokenization LLM-based agents operate with a bounded context depending on their token limit, which constrains the amount of decompiled code that can be considered at once [32]. The natural workaround for this is to segment the input into chunks that fit the token limit, then analyze them sequentially. However, this implementation fails to account for long control flows that span the entire code and for functions or variables defined outside the current section. This lack of context impairs the agentâs ability to properly analyze the codeâs relationships as a whole. Udeshi et al. [45] propose a framework using LLMs to generate summarizations of binary diffs for malware detection in supply chain scenarios. The study was conducted on 104 binaries using five LLMs and required token usage ranging from 100M to 500M per model. This study represents a best-case scenario, where the authors removed exact matches from the diff, thereby reducing token counts. However, this removal may affect an agentâs ability to fully understand the context of a binary. Current work demonstrates that larger binaries require a balance between tokens and functionality. 4.2 Dynamic Analysis Challenges Dynamic analysis poses challenges for agentic reverse-engineering systems from both architectural and tooling perspectives. Current works demonstrate an over-reliance on shell commands, which may constrain the agentâs ability to perform more diverse runtime analysis and, depending on the implementation, introduce security risks. Additionally, LLM-based agents are hindered by their prolonged reasoning times, which may impact time-sensitive interactions and lead reverse engineers to reach incorrect conclusions due to timeouts and other unintended behaviors. In addition, low-level emulation environments are hard to emulate, limiting the dynamic analysis an agent can perform on niche hardware. 4.2.1 Lack of Guardrails Some agents lack guardrails for the commands they are allowed to execute. One example of this is CTF-Dojo by Zhuo et al. [53] which builds upon Enigma+ [19]. Enigma+ allows unrestricted dynamic analysis on a given binary. This means an attacker could purposefully trick the agent into executing unsafe commands on the device. Another example is Muzsai et al. [30], who developed HackSynth, an LLM-based pentesting agent tested on CTF challenges. Similar to CTF-Dojo, HackSynth allows commands to be input without validation. 4.2.2 Timeouts A key challenge for executing dynamic analysis with agentic systems arises from the variable processing times of LLMs. LLMs utilize multi-step reasoning and planning, which may require an extended period of time to âthinkâ. This inconsistency in execution time can trigger timeouts [2] or miss key interactions altogether. Reverse engineers interacting with these systems may draw spurious conclusions from these timeouts, assuming the agent is operating under proper conditions. 4.2.3 Reliance on Emulation Almost all existing agentic systems perform analysis on desktops. These systems use Docker and/or QEMU to create virtual machines for arbitrary devices and architectures. However, several edge devices are notoriously hard to simulate [50]. Furthermore, some operating systems require multiple application components compiled with different compilers [32]. To our knowledge, edge device RE is not currently being explored due to emulation restrictions. 4.3 Hybrid Analysis Challenges Hybrid analysis shares challenges with static and dynamic analysis. However, hybrid systems exhibit one major challenge regardless of technical capabilities: system reasoning. Binaries are not definable deterministically [14]. Therefore, unchecked reasoning in agentic systems is more likely to lead to failure on RE tasks for complex applications. For example, if an agent hyperfocuses on a specific region of the binary, which may be obvious to a human, it could waste hours of compute time without any human intervention. Another issue found in hybrid systems is the reliance on a human-in-the-loop. Human-in-the-loop adds time to the RE process and limits the modelâs capabilities while retaining security and guidance. Interestingly, the challenges of full automation and Human-in-the-loop mirror each other. In this section, we describe the challenges associated with automation, as the goal for the community is automation rather than a human-in-the-loop approach. 4.3.1 Never Ending Automation As with any cat-and-mouse chase in security, the attacker adapts. If agentic systems attempt to fully automate RE practices, we foresee adversarial binary development to maximize the agentâs processing time. For example, new obfuscation techniques could emerge that force agents to triage long execution chains to extract a single piece of information. Rather than optimizing for human deobfuscation, the adversary will likely obfuscate to maximize hindrance to agentic communication and time. 5 Future Directions Agentic RE is still young. Many systems and agents are still vulnerable. Secure systems, including agentic ones, require solutions for edge cases to prevent system compromise. Below, we outline several research directions to address existing challenges. 5.1 Static Analysis Directions Our analysis indicates that static analysis agents are the most common. Agentic systems have several methods for understanding and interacting with decompiled code from a given binary. However, several challenges exist. In this section, we detail potential ways to improve static analysis agents from a security researcherâs perspective. 5.1.1 Raw Binary Analysis Currently, no agents deal with raw binary inputs. This is by design, as agents work with natural languages rather than bits and bytes. However, adversaries can exploit reliance on decompilation tools to hinder agentic reverse engineering. Since Ghidra and IDA deploy their own decompilation algorithms, it is possible to exploit the decompilation process to hide functionality [33]. Qin et al. [33] demonstrate an attack where an adversary can design assembly instructions that create a semantic deviation. One of those types is disassembly deviation, where, depending on the compiler, a set of instructions may yield different results. Thus, future work should explore alternative methods for binary decompilation such that they avoid or detect decompilation evasion. 5.1.2 Add Deobfuscation to Agentic Pipelines Obfuscation is a challenge in RE for both scalable [6] and manual methods [47]. Agentic frameworks are no different [8, 7], with many frameworks forgoing [7] or lightly deobfuscating [8]. Thus, we argue that research should be done to allow agents to perform deobfuscation. Deobfuscating a binary will likely increase the accuracy of malware detection for agentic systems, as the system will spend less time understanding instructions. 5.1.3 Clever Tokenization We found that most agentic systems that perform decompilation using Ghidra/IDA feed the output directly to the agent [11]. The problem with feeding decompilations to agents is that they contain repetitive information (e.g., stack frame construction, local variable allocation) that is irrelevant to the high-level functionality. Humans actively ignore repetitive functionality and focus on core logic for understanding the binary. LLM agents, on the other hand, may not ignore irrelevant blocks of instructions, which can lead to large tokenized decompilations and limit their capabilities. Some works, such as Udeshi et al. [45], cleverly leverage the fact that the agent is likely aware of open-source projects and can remove irrelevant functionality. While a clever solution, if presented with a binary that the system is unaware of, it may fail. 5.2 Dynamic Analysis Directions Work has shown that manipulating agents performing dynamic analysis is possible [31]. We anticipate that these attacks will become more complex as researchers explore techniques to interfere with agentic RE. This section describes potential solutions to the challenges in dynamic analysis. 5.2.1 Lack of Guardrails One common theme among the agents we review is a lack of guardrails on their capabilities when exploring a binary. Binary analysis tools are extremely powerful and may allow for a cleverly crafted binary to take control of a system. We propose that researchers attack agents during the dynamic analysis phase, as the binary serves as a dynamic adversary and can adjust its behavior in real time. It is expected that an adversarial binary has situational awareness and can adapt to human RE efforts. 5.2.2 Timeouts Adversarial binaries employ strict timeouts when executing critical tasks to prevent dynamic analysis [2]. Agents are more susceptible to timeouts than their human counterparts due to network latency and limited thinking time. A binary built to attack high-resource systems could wrap important functionality around strict thread timeouts that easily defeat thinking models. A solution to this problem is for the agent to control the time-estimation mechanisms in the observed binary. However, this solution requires precise function hooks and control over which time value the program observes. Timeouts offer a unique opportunity to develop agent-specific RE tools that build around natural weaknesses agents have. If a human RE built a generalized tool to control the clock based on an agentâs response time, the agent could automatically adjust the clock to account for its thinking time while the binary remains unaware. 5.2.3 Improving Emulation Environments While the technology does not currently exist, as a community, we should build stronger emulation environments for agentic systems. Several components could hypothetically be simulated using Gem5 [17], enabling deeper security analysis of hardware systems. Therefore, we propose building a similar tool that evaluates binaries on lesser-known hardware components that are challenging to emulate. This hypothetical tool will allow for scalable IoT analysis by an agentic system. 5.3 Hybrid Analysis Directions We argue that agents designed for hybrid analysis are susceptible to all of the challenges above as well as being vulnerable to their own. 5.3.1 Protection from Infinite Tool Calls We need to design protections from novel obfuscation schemes designed to impede hybrid agents. The goal of the hypothetical scheme would be to force an agent to waste its time potentially indefinitely. While this does not yet exist, we argue that it may in the near future and must be addressed to ensure robustness and convergence. 6 Conclusion In this work, we examine how agentic systems perform binary reverse engineering across static, dynamic, and hybrid settings. Additionally, we identify 6 challenges within the space. For static analysis, we identify challenges with obfuscation and tokenization. To address these challenges, we propose a future research attempt to perform raw binary analysis with agents and to add deobfuscation to agentic pipelines. For dynamic analysis, we note a lack of guardrails, issues stemming from timeouts, and a reliance on emulation. We outline future directions for binary design defenses, clock-based timeout protection, and improved emulation environments. Finally, we describe how hybrid agents may lead to an infinite sequence of tool calls. Our solution is to design defenses against infinite-agentic RE such that future obfuscation techniques will always converge. References [1] T. Abramovich, M. Udeshi, M. Shao, K. Lieret, H. Xi, K. Milner, S. Jancheska, J. Yang, C. E. Jimenez, F. Khorrami, P. Krishnamurthy, B. Dolan-Gavitt, M. Shafique, K. Narasimhan, R. Karri, and O. Press (2025) EnIGMA: interactive tools substantially assist lm agents in finding security vulnerabilities. External Links: 2409.16165, Link Cited by: §3.2.1. [2] A. Afianian, S. Niksefat, B. Sadeghiyan, and D. Baptiste (2019) Malware dynamic analysis evasion techniques: a survey. ACM Computing Surveys (CSUR) 52 (6), p. 1â28. Cited by: §1, §3.2.1, §3.2.1, §4.2.2, §5.2.2. [3] () Application fundamentals â App architecture â Android Developers â developer.android.com. Note: ://developer.android.com/guide/components/fundamentals Cited by: §3.1.1. [4] F. Balassone, V. Mayoral-Vilches, S. Rass, M. Pinzger, G. Perrone, S. P. Romano, and P. Schartner (2025-10) Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs. arXiv (en). Note: arXiv:2510.17521 [cs] External Links: Link, Document Cited by: §3.1. [5] Z. L. Basque, S. Doria, A. Soneji, W. Gibbs, A. DoupĂ©, Y. Shoshitaishvili, E. Losiouk, R. Wang, and S. Aonzo Decompiling the synergy: an empirical study of humanâllm teaming in software reverse engineering. Cited by: §3.2.1, §4.1.1. [6] J. Cao, F. Guo, and Y. Qu (2025) JNFuzz-droid: a lightweight fuzzing and taint analysis framework for native code of android applications. Empirical Software Engineering 30 (5), p. 113. Cited by: §1, §3.2.1, §5.1.2. [7] G. Chen, H. Sun, D. Liu, Z. Wang, Q. Wang, B. Yin, L. Liu, and L. Ying (2025-05) ReCopilot: Reverse Engineering Copilot in Binary Analysis. arXiv (en). Note: arXiv:2505.16366 [cs] External Links: Link, Document Cited by: §3.1, §5.1.2. [8] X. Chen, A. Zhou, C. Ye, and C. Zhang (2025) ClearAgent: agentic binary analysis for effective vulnerability detection. In Proceedings of the 1st ACM SIGPLAN International Workshop on Language Models and Programming Languages, p. 130â137. Cited by: §1, §5.1.2. [9] () Clang Static Analyzer â clang-analyzer.llvm.org. Note: ://clang-analyzer.llvm.org/ Cited by: §1, §3.1.1. [10] () Cyber-Zero/enigma-plus/config/commands/debug.sh at main · amazon-science/Cyber-Zero â github.com. Note: ://github.com/amazon-science/Cyber-Zero/blob/main/enigma-plus/config/commands/debug.sh Cited by: §1, §1, §1, §3.2.1. [11] G. Deng, Y. Liu, A. Robotics, A. Klagenfurt, P. Liu, Y. Li, T. Zhang, Y. Liu, A. Klagenfurt, and S. Rass PentestGPt: Evaluating and Harnessing Large Language Models for Automated Penetration Testing. (en). Cited by: §1, §3.1, §3.2.2, §5.1.3. [12] L. Dramko, D. Bölöni-Turgut, C. Le Goues, and E. Schwartz (2025) Quantifying and mitigating the impact of obfuscations on machine-learning-based decompilation improvement. In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment, p. 244â266. Cited by: §1. [13] () Dynamorio.org. Note: ://dynamorio.org/ Cited by: §1. [14] D. Engel, F. Verbeek, and B. Ravindran (2024) On the decidability of disassembling binaries. In International Symposium on Theoretical Aspects of Software Engineering, p. 127â145. Cited by: §1, §4.3. [15] () Frida âą A world-class dynamic instrumentation toolkit â frida.re. Note: ://frida.re/ Cited by: §1. [16] () GDB: The GNU Project Debugger â sourceware.org. Note: ://w.sourceware.org/gdb/ Cited by: §3.2.1. [17] () Gem5: The gem5 simulator system â gem5.org. Note: ://w.gem5.org/ Cited by: §5.2.3. [18] R. Ghosh, H. v. Stockhausen, M. Schmitt, G. M. Vasile, S. K. Karn, and O. Farri (2025-04) CVE-LLM: Ontology-Assisted Automatic Vulnerability Evaluation Using Large Language Models. Proceedings of the AAAI Conference on Artificial Intelligence 39 (28), p. 28757â28765 (en). External Links: ISSN 2374-3468, Link, Document Cited by: §1, §3.1. [19] (2025) GitHub - amazon-science/Cyber-Zero: Cyber-Zero: Training Cybersecurity Agents Without Runtime â github.com. Note: ://github.com/amazon-science/Cyber-Zero Cited by: §3.2.1, §4.2.1. [20] () GitHub - NationalSecurityAgency/ghidra: Ghidra is a software reverse engineering (SRE) framework â github.com. Note: ://github.com/NationalSecurityAgency/ghidra Cited by: §2.2, §3.1.1. [21] () GitHub - radareorg/radare2: UNIX-like reverse engineering framework and command-line toolset â github.com. Note: ://github.com/radareorg/radare2 Cited by: §1, §3.1.1. [22] X. He, D. Wu, Y. Zhai, and K. Sun (2025-05) SentinelAgent: Graph-based Anomaly Detection in Multi-Agent Systems. arXiv (en). Note: arXiv:2505.24201 [cs] External Links: Link, Document Cited by: §1. [23] Hex-Rays () Note: ://hex-rays.com/ida-pro Cited by: §2.2, §3.1.1. [24] L. Hu, G. Chen, X. Shang, S. Cheng, B. Wu, G. Li, X. Zhu, W. Zhang, and N. Yu CompileAgent: Automated Real-World Repo-Level Compilation with Tool-Integrated LLM-based Agent System. (en). Cited by: §3.1. [25] X. Hu, Z. Fu, S. Xie, S. H. H. Ding, and P. Charland (2025-09) SoK: Potentials and Challenges of Large Language Models for Reverse Engineering. arXiv (en). Note: arXiv:2509.21821 [cs] External Links: Link, Document Cited by: §3. [26] H. Kong, D. Hu, J. Ge, L. Li, T. Li, and B. Wu (2025-01) VulnBot: Autonomous Penetration Testing for A Multi-Agent Collaborative Framework. arXiv (en). Note: arXiv:2501.13411 [cs] External Links: Link, Document Cited by: §3.1. [27] X. Lin, J. Zhang, G. Deng, T. Liu, T. Zhang, Q. Guo, and R. Chen (2025-10) IRCopilot: Automated Incident Response with Large Language Models. arXiv (en). Note: arXiv:2505.20945 [cs] External Links: Link, Document Cited by: §3.1. [28] O. L. &. C. LLC (2024) Debuggers - CTF Handbook â ctf101.org the gnu debugger (gdb). Note: ://ctf101.org/reverse-engineering/what-is-gdb/ Cited by: §3.2.1. [29] J. I. Maletic and M. L. Collard (2015-05) Exploration, analysis, and manipulation of source code using srcml. IEEE. External Links: Link, Document Cited by: §3.1.1. [30] L. Muzsai, D. Imolai, and A. LukĂĄcs (2024) HackSynth: llm agent and evaluation framework for autonomous penetration testing. External Links: 2412.01778, Link Cited by: §1, §1, §1, §1, §3.2.1, §4.2.1. [31] D. Pasquini, E. M. Kornaropoulos, and G. Ateniese (2024-11) Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks. arXiv (en). Note: arXiv:2410.20911 [cs] External Links: Link, Document Cited by: §1, §3.1, §3.2.2, §5.2. [32] X. Qian, X. Zheng, Y. He, S. Yang, and L. Cavallaro (2025) LAMD: context-driven android malware detection and classification with llms. External Links: 2502.13055, Link Cited by: §1, §1, §3.1.1, §4.1.2, §4.2.3. [33] S. Qin, C. Zhang, K. Chen, and Z. Li (2021) IDEV: exploring and exploiting semantic deviations in arm instruction processing. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, p. 580â592. Cited by: §5.1.1. [34] S. Ren and S. Chen Large Language Models for Cybersecurity Intelligence, Threat Hunting, and Decision Support. (en). Cited by: §3. [35] H. Rong, Y. Duan, H. Zhang, X. Wang, H. Chen, S. Duan, and S. Wang (2024) Disassembling obfuscated executables with llm. External Links: 2407.08924, Link Cited by: §3.1.1. [36] A. Shahriar, M. N. Rahman, S. Ahmed, F. Sadeque, and M. R. Parvez (2025) A survey on agentic security: applications, threats and defenses. arXiv preprint arXiv:2510.06445. Cited by: §1, §2.2, §3. [37] X. Shang, G. Chen, S. Cheng, B. Wu, L. Hu, G. Li, W. Zhang, and N. Yu (2025) BinMetric: a comprehensive binary analysis benchmark for large language models. External Links: 2505.07360, Link Cited by: §3.1.1, §4.1.1. [38] X. Shang, S. Cheng, G. Chen, Y. Zhang, L. Hu, X. Yu, G. Li, W. Zhang, and N. Yu (2024) How far have we gone in binary code understanding using large language models. External Links: 2404.09836, Link Cited by: §4.1.1. [39] X. She, Y. Zhao, and H. Wang (2024) WaDec: decompiling webassembly using large language model. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE â24, New York, NY, USA, p. 481â492. External Links: ISBN 9798400712487, Link, Document Cited by: §3.1.1. [40] X. Shen, L. Wang, Z. Li, Y. Chen, W. Zhao, D. Sun, J. Wang, and W. Ruan (2025) Pentestagent: incorporating llm agents to automated penetration testing. In Proceedings of the 20th ACM Asia Conference on Computer and Communications Security, p. 375â391. Cited by: §1, §2.2, §3.1. [41] Skylot () GitHub - skylot/jadx: Dex to Java decompiler â github.com. Note: ://github.com/skylot/jadx Cited by: §3.1.1. [42] H. Tan, Q. Luo, J. Li, and Y. Zhang (2024) LLM4Decompile: decompiling binary code with large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 3473â3487. External Links: Link, Document Cited by: §1, §3.1.1, §4.1.1, §4.1.1. [43] H. Triedman, R. Jha, and V. Shmatikov (2025) Multi-agent systems execute arbitrary malicious code. arXiv preprint arXiv:2503.12188. Cited by: §1. [44] H. Triedman, R. Jha, and V. Shmatikov (2025) Multi-agent systems execute arbitrary malicious code. External Links: 2503.12188, Link Cited by: §1. [45] M. Udeshi, V. S. C. Putrevu, P. Krishnamurthy, P. Anantharaman, S. Carrick, R. Karri, and F. Khorrami (2025-09) Binary Diff Summarization using Large Language Models. arXiv (en). Note: arXiv:2509.23970 [cs] External Links: Link, Document Cited by: §1, §3.1, §4.1.2, §5.1.3. [46] Y. Wang, X. Xu, X. Zhu, X. Gu, and B. Shen (2025) SALT4Decompile: inferring source-level abstract logic tree for llm-based binary decompilation. External Links: 2509.14646, Link Cited by: §3.1.1. [47] J. West, L. Thiemt, S. Ahmed, M. Bartig, K. Fawaz, and S. Banerjee (2024) A picture is worth 500 labels: a case study of demographic disparities in local machine learning models for instagram and tiktok. In 2024 IEEE Symposium on Security and Privacy (SP), p. 369â387. Cited by: §1, §3.2.1, §5.1.2. [48] J. P. A. Yaacoub, H. N. Noura, O. Salman, and G. Pujolle (2025-11) Large language models: applications, limitations, challenges, and recommendations in cybersecurity, digital forensics, and ethical hacking. Annals of Telecommunications (en). External Links: ISSN 1958-9395, Link, Document Cited by: §3. [49] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press (2024) SWE-agent: agent-computer interfaces enable automated software engineering. External Links: 2405.15793, Link Cited by: §1, §1, §1. [50] W. Zhou, L. Zhang, L. Guan, P. Liu, and Y. Zhang (2022) What your firmware tells you is not how you should emulate it: a specification-guided approach for firmware emulation. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, p. 3269â3283. Cited by: §1, §4.2.3. [51] T. Y. Zhuo, D. Wang, H. Ding, V. Kumar, and Z. Wang (2025) Cyber-zero: training cybersecurity agents without runtime. External Links: 2508.00910, Link Cited by: §3.2.1. [52] T. Y. Zhuo, D. Wang, H. Ding, V. Kumar, and Z. Wang (2025-09) Training Language Model Agents to Find Vulnerabilities with CTF-Dojo. arXiv (en). Note: arXiv:2508.18370 [cs] External Links: Link, Document Cited by: §3.1. [53] T. Y. Zhuo, D. Wang, H. Ding, V. Kumar, and Z. Wang (2025) Training language model agents to find vulnerabilities with ctf-dojo. External Links: 2508.18370, Link Cited by: §3.2.1, §4.2.1.