Paper deep dive
Learning from the Test: Self-Referential Differential Testing for Deep RL Agents
Junda He, Jieke Shi, Zhou Yang, Mingfei Cheng, David Lo
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deep Reinforcement Learning (DRL) has achieved significant success in complex decision-making problems. As DRL systems are increasingly deployed in real-world applications, ensuring their quality and reliability is paramount. Current works primarily focus on detecting safety-critical failures, often neglecting policy optimality, which can lead to reduced efficiency, user distrust, and economic losses. This oversight, compounded by the inherent "testing oracle problem" for optimality, leaves a significant gap in comprehensively evaluating DRL systems. To address this gap, we propose Delta (Differential Testing for DRL Agents), a novel and comprehensive framework that automatically identifies both safety-critical and optimality bugs in DRL agents. Delta employs a two-phase approach: (1) Safety Testing, where the Agent Under Test (AUT) is evaluated for catastrophic failures while collecting data from its decision-making policy, and (2) Optimality Testing, where this collected data from the prior phase is used to train a challenger agent via Offline Reinforcement Learning. Differential testing is then performed by comparing the challenger agent against the AUT; instances where the challenger achieves higher cumulative rewards indicate optimality issues in the AUT. We demonstrate Delta's effectiveness across five environments. We investigate the effectiveness of three offline RL algorithms (BC, BCQ, and CQL) in generating challenger agents. Experimental results demonstrate that safety testing datasets are valuable for training competent DRL agents. Challenger agents trained with BCQ proved most effective for identifying optimality issues within the framework of Delta. Across the five environments, Delta uncovered an average of 2,518 optimality issues, outperforming the baseline methods by 50.2%.
Tags
Links
- Source: https://arxiv.org/abs/2608.22284v1
- Canonical: https://arxiv.org/abs/2608.22284v1
Trouble viewing inline? Open PDF directly →
Full Text
99,273 characters extracted from source content.
Expand or collapse full text
Learning from the Test: Self-Referential Differential Testing for Deep RL AgentsJournal: PACMSEVolume: 3ISSTAISSTA01110DOI: 10.1145/3832102issta26main-p101-pCCS: Computing methodologies Markov decision processesCCS: Software and its engineering Search-based software engineering Junda He OrcID: 0000-0003-3370-8585 email: jundahe.2022@phdcs.smu.edu.sg Affiliation: Singapore Management University , Singapore , Singapore , Jieke Shi OrcID: 0000-0002-0799-5018 email: jiekeshi@smu.edu.sg Affiliation: Singapore Management University , Singapore , Singapore , Zhou Yang OrcID: 0000-0001-5938-1918 email: zy25@ualberta.ca Affiliation: University of Alberta , Edmonton , Canada Affiliation: Canada CIFAR AI Chair, Amii , Edmonton , Canada , Mingfei Cheng OrcID: 0000-0002-8982-1483 email: mfcheng.2022@phdcs.smu.edu.sg Affiliation: Singapore Management University , Singapore , Singapore and David Lo OrcID: 0000-0002-4367-7201 email: davidlo@smu.edu.sg Affiliation: Singapore Management University , Singapore , Singapore 2026; © c; Received 2026-06-25 Abstract. Deep Reinforcement Learning (DRL) has achieved significant success in complex decision-making problems. As DRL systems are increasingly deployed in real-world applications, ensuring their quality and reliability is paramount. Current works primarily focus on detecting safety-critical failures, often neglecting policy optimality, which can lead to reduced efficiency, user distrust, and economic losses. This oversight, compounded by the inherent “testing oracle problem” for optimality, leaves a significant gap in comprehensively evaluating DRL systems. To address this gap, we propose Delta (Differential Testing for DRL Agents), a novel and comprehensive framework that automatically identifies both safety-critical and optimality bugs in DRL agents. Delta employs a two-phase approach: (1) Safety Testing, where the Agent Under Test (AUT) is evaluated for catastrophic failures while collecting data from its decision-making policy, and (2) Optimality Testing, where this collected data from the prior phase is used to train a challenger agent via Offline Reinforcement Learning. Differential testing is then performed by comparing the challenger agent against the AUT; instances where the challenger achieves higher cumulative rewards indicate optimality issues in the AUT. We demonstrate Delta’s effectiveness across five environments. We investigate the effectiveness of three offline RL algorithms (BC, BCQ, and CQL) in generating challenger agents. Experimental results demonstrate that safety testing datasets are valuable for training competent DRL agents. Challenger agents trained with BCQ proved most effective for identifying optimality issues within the framework of Delta. Across the five environments, Delta uncovered an average of 2,518 optimality issues, outperforming the baseline methods by 50.2%. Keywords: Deep Reinforcement Learning, Differential Testing, Fuzz Testing †c-license: by 1. Introduction In recent years, Deep Reinforcement Learning (DRL) (76; 49; 1) has gained significant attention for its ability to tackle complex decision-making problems. By interacting with unknown environments through trial-and-error, DRL agents learn a decision-making policy to maximize cumulative reward over a sequence of actions. DRL has demonstrated remarkable success in various domains such as healthcare (82; 8), autonomous driving (29; 58; 60), and robotic manipulation (27; 31; 52), outperforming traditional control methods by a substantial margin (46; 20). As DRL systems, and AI-based software systems more broadly (25), are increasingly deployed in real-world applications (68; 53), ensuring their quality and reliability becomes paramount (81). Recent testing methodologies (55; 26; 38; 62) for DRL systems primarily focus on detecting catastrophic failures (safety-critical issues), e.g., a DRL-controlled robot colliding with obstacles. We refer to these testing methods as safety testing. Despite this progress, current safety testing methods largely overlook a critical dimension: the decision optimality of DRL systems. Evaluating policy optimality is as crucial as ensuring safety. For instance, a robotic arm performs a task with unnecessary energy consumption. Such non-optimal DRL decisions can incur higher economic costs, reduce operational efficiency, and decrease user satisfaction. More critically, unresolved optimality issues can increase the risk of escalating into safety-critical hazards; for example, a delivery robot selecting congested routes is more likely to face traffic accidents or battery depletion. True system reliability, therefore, demands both safety and optimality. To bridge the critical gap between safety and optimality evaluation, this paper presents a unified framework for testing DRL agents on detecting both safety-critical failures and optimality issues. However, a fundamental challenge in testing the optimality of DRL policies is the absence of a clear testing oracle. DRL is predominantly applied in complex and dynamic environments. In these environments, the optimal solution that achieves the theoretical maximum cumulative reward is either unknown or analytically intractable. Without a ground-truth optimal solution to serve as a benchmark, it becomes difficult to assess the optimality of an agent’s decisions quantitatively. This is akin to verifying the output of a software program without knowing the correct output. While manual assessment by domain experts is an option, it is prohibitively time-consuming and expertise-intensive, rendering it infeasible to scale for the vast and diverse scenarios DRL agents encounter. Automated, learned surrogates are therefore increasingly explored as scalable evaluators when a ground-truth oracle is unavailable, e.g., for judging software artifacts (24) and assessing the correctness of automatically generated patches (84). The impracticality of manual assessment underscores the critical challenge of addressing the oracle problem and developing automated testing methods for DRL policy optimality. To overcome this, differential testing (47; 15) emerges as a promising solution. This widely used software testing technique compares the outputs of multiple implementations of the software, thereby eliminating the need to know the correct output. Similarly, we adapt this principle to DRL. For a given DRL agent to be tested—we refer to this agent as the agent under test (AUT)—by comparing the AUT’s cumulative reward against other DRL instances, we can reveal the AUT’s optimality issues if it is outperformed in cumulative reward. However, this approach introduces a new challenge: generating suitable DRL instances for comparison. We can repeat the training process several times to obtain multiple DRL agents for differential testing; however, developers are typically only interested in testing high-performing DRL agents. Training such policies requires extensive interaction with the environment, consuming considerable computational resources and substantial time (12; 11). For large-scale systems, restarting this process can take weeks or even months, requiring the processing of billions of data points (12). To address this limitation, we introduce a more efficient approach founded on a core insight: the detailed records of an AUT’s decision-making process generated during safety testing are a vibrant and underutilized resource for training highly competent DRL agents. The reasons are: (1) existing testing methodologies (26; 55; 38) employ various mechanisms to generate a broad spectrum of scenarios for evaluating the AUT and triggering diverse agent behaviors. These scenarios often include edge cases that are rarely encountered during standard training, deliberately pushing the AUT into less explored regions. (2) Developers are typically only interested in testing high-performing DRL agents. Consequently, the AUT is a well-trained agent. Its behaviors, therefore, constitute a high-quality dataset of successful decisions. Leveraging this data provides a new policy with a strong starting point, jump-starts its training process, and bypasses the need for extensive, and often inefficient, initial exploration from scratch. (3) Finally, the resulting dataset offers a mixture of learning signals, which is crucial for robust training. It contains not only the AUT’s successful trajectories as strong positive examples but also its documented failures as critical negative examples. This mixture provides clear guidance, teaching a new agent what behaviors to replicate and which to avoid. As a result, we train a new agent using the AUT’s own interaction data captured during safety testing. We name this new agent the challenger agent. Because it is trained solely on the AUT’s own history, the challenger agent then serves as a potent self-referential testing oracle for suboptimality detection. This approach pragmatically relaxes the oracle problem. Acknowledging that true optimal policies are often unknown, if the challenger agent finds a superior solution, it exposes a concrete instance where the AUT’s policy can be improved. As a result, this method effectively enables automated optimality testing without requiring a predefined perfect solution. Notably, this approach yields value regardless of the testing outcome. A superior challenger identifies optimality issues and guides policy improvement. Conversely, a challenger that fails to outperform the AUT validates that the AUT is already near-optimal—without ever knowing what the true optimum is. In this paper, we introduce Delta (Differential Testing for DRL Agents), a comprehensive framework designed to automatically test DRL agents. Delta seamlessly integrates established safety testing methods, such as CureFuzz (26), MDPFuzz (55), and GMT (38), with our proposed differential testing method, enabling holistic evaluation of agent quality in terms of both catastrophic failures and suboptimal behavior. Delta consists of two main phases: (1) Safety Testing: The AUT is evaluated across a diverse set of scenarios to identify safety-critical issues. We adopt the state-of-the-art technique CureFuzz (26) for this phase. Simultaneously, Delta records the AUT’s interaction trajectories, creating a rich dataset that underpins subsequent optimality analysis. (2) Optimality Testing: Using the collected trajectories, Delta applies Offline Reinforcement Learning (36; 32) to train a challenger agent. We then perform differential testing by comparing the challenger’s performance against the original AUT. Cases where the challenger achieves higher cumulative rewards are flagged as optimality issues in the AUT. To assess Delta’s effectiveness, we conducted evaluations across five diverse environments: three classical control tasks (CartPole, MountainCar, Acrobot) and two MuJoCo locomotion tasks (Hopper, Walker2D). We also evaluated the performance of three offline RL algorithms (BC (75), BCQ (21), and CQL (34)) in generating challenger agents. Experimental results demonstrate that safety testing datasets are valuable for training competent DRL agents. Challenger agents trained with BCQ proved most effective for identifying optimality issues within Delta. Leveraging the BCQ-trained challenger agents, Delta uncovered an average of 2,518 optimality issues per environment. Delta significantly outperforms the baseline methods and identifies an average of 50.2% more optimality bugs across all environments. The contributions of this paper include: • We propose a novel self-referential testing oracle for detecting optimality bugs in DRL agents. • We develop Delta, a unified two-stage framework comprising a safety testing stage (which adopts state-of-the-art method CureFuzz) and our novel optimality testing stage, enabling DRL agents to be evaluated for both catastrophic failures and suboptimal decisions in a single pipeline. • We provide extensive experimental evidence demonstrating that Delta effectively detects a significant number of optimality issues across diverse environments and DRL algorithms. 2. Preliminaries 2.1. Markov Decision Process In general, reinforcement learning addresses the problem of learning to control a dynamical system (67). Many RL problems are formalized using Markov Decision Processes (MDPs) (56), which are defined by a tuple ⟨S,S0,A,T,R,γ⟩ S,S_0,A,T,R,γ where: • S is a set of states. A state s∈Ss∈ S represents a specific situation the agent encounters within the environment. • S0⊆S_0 S is the set of possible starting states. An initial state s0∈S0s_0∈ S_0 is selected for each episode. • A is a set of actions. An action a∈Aa∈ A is a decision made by the agent that affects the current state. • T:S×A×S→[0,1]T:S× A× S→[0,1] is the transition probability function, where T(s′|s,a)T(s |s,a) represents the probability of transitioning from state s to state s′s by taking action a. • R:S×A→ℝR:S× A is the reward function, where R(s,a)R(s,a) is the immediate reward received for taking action a in state s. Typically, the agent’s objective is to maximize the cumulative reward over time. • γ∈(0,1]γ∈(0,1] is a scalar discount factor that determines the present value of future rewards. Markov Property. The Markov property implies that the future state depends solely on the current state and the action taken without being influenced by the sequence of past events. In this paper, we assume that the environments used in our experiments adhere to the Markov property. Trajectory. A trajectory is a sequence of (state, action, reward) tuples. Episode. An episode is always a trajectory that represents a complete run from a start state to a terminal state. Policy. A policy π:S→Aπ:S→ A is a strategy that specifies the action a to be taken when in state s. The goal is to discover an optimal policy π∗π^* that maximizes the expected cumulative reward over time. This objective can be formalized as the maximization of the expected sum of discounted rewards: (1) max∑t=1∞πst,at∼π[γtR(st,at)] _π _t=1^∞E_s_t,a_t π [γ^tR(s_t,a_t) ] where R(st,at)R(s_t,a_t) represents the reward received at time step t for taking action ata_t in state sts_t, and E denotes the expectation over the states and actions under policy π. 2.2. Offline Reinforcement Learning Figure 1. Illustration of Offline Reinforcement Learning.Illustration of Offline Reinforcement Learning. Offline Reinforcement Learning (Offline RL) (36; 32) is an emerging paradigm in RL research. Unlike traditional RL, where agents learn by continuously interacting with the environment, offline RL focuses on creating effective policies exclusively from pre-collected datasets of trajectories without needing further environmental interaction (See Figure 1 as an example). This approach is particularly valuable in scenarios where interaction with the environment is expensive, time-consuming, or poses safety risks, such as in healthcare (71; 70), robotics (64; 33), or autonomous driving (16; 63). Formally, in the offline RL setting (36; 32), learning occurs from a static dataset =(si,ai,ri,si′)i=1ND=\(s_i,a_i,r_i,s _i)\_i=1^N. The objective is to learn a policy π:S→Aπ:S→ A using only D that maximizes the expected cumulative discounted reward. The dataset D is generated by an existing policy (or set of policies), which is termed the behavior policy (65; 19). This behavior policy can encompass a wide range of data collection strategies, from random exploration or rule-based approaches to expert demonstrations or policies from previously trained agents. 3. Approach Figure 2. The Overall Workflow of Delta.The Overall Workflow of Delta. 3.1. Assumption Our methodology assumes a black-box testing paradigm: we do not require access to the AUT’s internal states or parameters. Observations are limited to the AUT’s interactions with the environment (i.e., state-action-reward-next_state tuples) generated during test execution. Furthermore, we assume the AUT adheres to a deterministic policy and the environment exhibits deterministic transition dynamics. While the approach is designed for environments modeled as Markov Decision Processes (56), it does not necessitate explicit knowledge of the environment’s transition dynamics. This characteristic renders the method applicable to scenarios involving opaque AUTs, such as proprietary DRL agents or third-party services. 3.2. Overview Delta aims to detect both catastrophic failures and optimality bugs in DRL agents. As illustrated in Figure 2, it consists of two main phases: 1) Safety Testing. This phase aims to uncover safety-critical issues of the AUT, such as crashes or some hard constraint violations. In our current implementation, we utilize CureFuzz (26), a state-of-the-art safety testing method for DRL agents. Notice that Delta can be seamlessly integrated with any other safety testing frameworks. Concurrently, all AUT behaviors observed during this fuzzing process are collected and stored as a dataset D. 2) Optimality Testing. In the second phase, the dataset D collected from the safety testing phase is used to train a “challenger” agent, utilizing offline RL algorithms. This challenger agent serves as the testing oracle, guiding a differential fuzzing process to detect the AUT’s optimality issues. An optimality bug is subsequently flagged in any scenario where the challenger agent achieves a higher cumulative reward than the AUT. We introduce these two phases in detail as follows. 3.3. Safety Testing Several safety testing methods have recently been proposed to detect crashes and hard constraint violations in DRL agents (26; 55; 38). For our safety testing, we did not propose a new safety testing method. In our implementation, we adopt the state-of-the-art testing method, CureFuzz (26). CureFuzz is a black-box fuzz testing approach. CureFuzz employs a “curiosity mechanism,” inspired by Random Network Distillation (5) and a multi-objective seed selection technique to optimize the fuzzing process to uncover a broad spectrum of crash-triggering scenarios; for details, we refer to the original CureFuzz paper (26). Dataset Construction. Concurrently with the safety testing process, all observed trajectories are collected to form a static dataset D. Each trajectory τ′τ , initiated from a seed σ′σ (via its initial state s0′s _0), consists of a sequence of state-action-reward-next_state tuples: τ′=(st,at,rt,st+1)t=0H−1τ =\(s_t,a_t,r_t,s_t+1)\_t=0^H-1, where H is the length of the episode. The dataset =τi′i=1ND=\τ _i\_i=1^N thus aggregates all such trajectories recorded during fuzzing; the transition tuples they contain form the static dataset used for offline RL training (Section 2). 3.4. Optimality Testing 3.4.1. Overview Optimality testing mainly consists of two key stages. First, a challenger agent is trained via offline RL algorithms (36; 35; 63) on the dataset D collected during safety testing. Then, this challenger is employed within a differential fuzzing process to identify optimality bugs of the AUT. Algorithm 1 Differential Fuzzing for Optimality Testing Input: Agent Under Test (AUT), Challenger Agent (CA) Initialize seed corpus S; curiosity networks T (fixed), P (trainable) while time budget not exhausted do Select seed σ from S by energy E(σ)E(σ) (Eq. 7); mutate to σ′σ Execute AUT and CA on σ′σ ; obtain trajectories τaut′,τca′τ _aut,τ _ca and rewards Raut(σ′),Rca(σ′)R_aut(σ ),R_ca(σ ) if Rca(σ′)>Raut(σ′)+δR_ca(σ )>R_aut(σ )+δ then Report optimality bug end if Compute diversity D(σ′)D(σ ) over τaut′τ _aut (Eq. 3) Update P to minimize ‖T(s)−P(s)‖2\|T(s)-P(s)\|^2 over s∈τaut′s∈τ _aut Compute regret A(σ′)A(σ ) (Eq. 4) and inconsistency I(σ′)I(σ ) (Eq. 6) if D(σ′)>D¯D(σ )> D_S or A(σ′)>1A(σ )>1 or I(σ′)<I¯I(σ )< I_S then Add σ′σ to S end if Update energy E(σ)E(σ) for all seeds in S end while 3.4.2. Oracle Formulation We define an optimality issue as a scenario where the AUT achieves lower cumulative rewards than the challenger agent from the same initial state. The oracle, therefore, relies on the reward function R, and its validity depends on what role R plays in our setting. We treat R as the reference objective for testing—the same objective the AUT was trained against. This follows standard DRL practice (67). The design of R is a separate research problem, orthogonal to Delta. That said, Delta is naturally compatible with extended objectives. A practitioner can encode a new non-functional concern into R, and rerun Delta against the extended objective. Formally, the oracle condition for detecting an optimality issue is: (2) R(τca)>R(τaut)+δR( _ca)>R( _aut)+δ where τaut _aut and τca _ca are the trajectories generated by the AUT and the challenger agent from the same initial state s0s_0, R(τ)R(τ) denotes the cumulative reward for trajectory τ, and δ≥0δ≥ 0 is a user-configurable magnitude threshold (default δ=0δ=0, recovering the strict inequality). All experiments in this paper use δ=0δ=0. Challenger Agent Training. The dataset D, gathered during the Safety Testing phase (Section 3.3), is utilized to train a challenger agent. We employ offline RL algorithms for this training, with specific algorithmic details provided in Section 4. The quality of the dataset D is critical to the challenger agent’s performance, and the data gathered during safety testing is uniquely suitable for this purpose for several reasons. First, safety testing naturally explores a wide spectrum of scenarios, including rare edge cases not encountered during standard training. This data diversity is crucial for training a challenger that can generalize across a broad range of situations (59; 66). Second, the dataset contains a rich mixture of learning signals. By including both the AUT’s successful trajectories and its documented failures, it provides clear positive and negative examples that guide the challenger on which behaviors to imitate and which to avoid (36; 54). Finally, this approach is highly efficient. Since the dataset D is generated by a competent AUT, it provides a strong baseline that jump-starts the challenger’s training, bypassing the need to learn from scratch. Moreover, the training is conducted entirely offline, requiring no additional, costly environment interactions. Another advantage of Delta’s design is the direct utilization of these existing safety testing logs. This ensures seamless integration with established safety testing and pragmatically eliminates the need for any separate, resource-intensive data collection phase specifically for training the challenger agent. Figure 3. Delta’s Differential Fuzzing Workflow.Delta Differential Fuzzing Workflow. 3.4.3. Differential Fuzzing After training the challenger agent, Delta employs differential fuzzing to identify optimality bugs. As depicted in Figure 3, this process leverages the challenger agent as a testing oracle to systematically evaluate the AUT’s decisions. Overview. Delta follows an energy-guided, iterative fuzzing process. To begin, Delta randomly generates an initial seed corpus, where each seed represents one initial state of the given environment. For example, in autonomous driving, a scenario can be described by the initial position of the autonomous vehicle and surrounding vehicles. In each iteration, Delta selects a high energy seed from the corpus and mutates it slightly to generate a new seed (a new initial state). Delta then observes both the AUT and the challenger’s actions under the new initial state to assess if the AUT performs suboptimally. An optimality bug is flagged if the challenger agent achieves a higher cumulative reward than the AUT. This cycle continues until the allocated time budget is exhausted. To guide the generation of diverse and compelling test cases, Delta estimates the energy of the seed. Higher energy seeds, which indicate either novel, unexplored scenarios or greater potential to expose optimality bugs, are prioritized for mutation to produce new test cases. If the mutated seed is deemed “interesting,” it would be added back to the seed corpus for consideration in subsequent iterations. Seed Energy Measurement. Seed energy is determined by three factors: Diversity, Regret, and Inconsistency. Hereafter, we denote a seed by σ. Each seed corresponds to an initial state, typically s0s_0, from which the AUT generates a trajectory. We use s or sts_t to represent a generic state within such a trajectory. 1) Diversity: Indicating a seed’s novelty compared to previously encountered seeds, diversity is key for exploring new AUT behaviors and scenarios. It is quantified using a curiosity mechanism (26). The intuition behind this mechanism is to identify states that are “novel” to the fuzzer. It employs two neural networks: a target network T and a predictor network P. Both networks share the same architecture (e.g., a Multi-Layer Perceptron (MLP) (9)) and accept states encountered by the AUT as input. The target network T is initialized with random weights and remains fixed throughout the fuzzing process. The predictor network P, however, is continuously trained to mimic the output of T using the states observed during fuzzing. Note that a randomly initialized DNN is fundamentally different from an unstructured random labeling. Although a DNN’s weights may be sampled randomly, the network itself is a deterministic, continuous function of its input: similar inputs produce similar outputs. This locality propagates to the predictor: once P has been trained to mimic T on a set of observed states, it also approximates T’s output well for unseen states near them. The prediction error therefore stays low in well-explored regions and rises sharply in unexplored ones, which is precisely what we exploit as a novelty signal. Replacing T with per-state random labels would destroy this property: nearby inputs would carry uncorrelated targets, leaving P with no basis for extrapolation, and every unseen state would yield a similarly large error regardless of its proximity to observed ones. Concretely, for an individual state s encountered in a trajectory, this novelty is measured by the Mean Squared Error (77) between the outputs of the two networks: ‖T(s)−P(s)‖2 T(s)-P(s) ^2. The diversity of a seed σ (with initial state s0s_0), denoted D(σ)D(σ), is then calculated as the average of these novelty scores over all states in the trajectory τ generated by executing the AUT starting from s0s_0: (3) D(σ)=1|τ|∑st∈τ‖T(st)−P(st)‖2D(σ)= 1|τ| _s_t∈τ T(s_t)-P(s_t) ^2 2) Regret: Regret, denoted A(σ)A(σ), measures the performance difference between the AUT and the challenger agent for a specific initial state s0s_0. It quantifies the AUT’s suboptimal performance relative to the challenger, reflecting the potential improvement had the AUT followed the challenger’s policy. A regret value A(σ)>1A(σ)>1 indicates that the challenger agent achieves a higher cumulative reward (RπcaR _ca) than the AUT (RπautR _aut) for that scenario. Formally, A(σ)A(σ) is calculated as: (4) A(σ)=exp(Rπca(s0)−Rπaut(s0))A(σ)= (R _ca(s_0)-R _aut(s_0) ) This exponential formulation of regret serves to significantly amplify scenarios where the challenger agent markedly outperforms the AUT. By emphasizing this relative performance gain, A(σ)A(σ) guides the fuzzer towards test cases demonstrating clear potential for the AUT’s policy improvement, rather than merely towards situations of poor AUT performance that might arise from inherently challenging or unsolvable aspects of the environment for both agents. 3) Inconsistency: The Inconsistency score, I(σ)I(σ), quantifies the behavioral overlap between the AUT and the challenger agent when starting from the same initial seed σ; a lower score indicates greater behavioral divergence. It aims to identify scenarios where the two agents explore dissimilar state sequences, potentially revealing instances where the challenger’s actions differ significantly from those of the AUT. To measure inconsistency, we employ Locality Sensitive Hashing (LSH) (28). LSH utilizes a hash function h(s)h(s) to map high-dimensional states s to lower-dimensional hash codes. We implement LSH based on Euclidean distance (10). The process can be conceptualized as follows: (a) Random Hyperplane Generation: A set of k random hyperplanes is defined within the high-dimensional state space. Each hyperplane j passes through the origin and is characterized by its random normal vector rjr_j. (b) State Projection and Discretization: For any given state s, its projection onto each random normal vector rjr_j is calculated as the dot product: pj(s)=s⋅rjp_j(s)=s· r_j. The resulting projected value is then binarized based on its sign. Specifically, a single bit gj(s)g_j(s) is generated for each hyperplane, indicating on which side of the j-th hyperplane the state s lies: gj(s)=1if pj(s)>00if pj(s)≤0g_j(s)= cases1&if p_j(s)>0\\ 0&if p_j(s)≤ 0 cases (c) Binary Hash Code: The resulting sequence of k binary digits, (g1(s),g2(s),…,gk(s))(g_1(s),g_2(s),…,g_k(s)), forms the discrete hash code, h(s)h(s), for state s. The key property is that if two states sis_i and sjs_j are close in Euclidean distance, their projections onto most random lines are also likely to be close, causing them to fall into the same or nearby segments. This results in a high probability that their composite hash codes, h(si)h(s_i) and h(sj)h(s_j), will be identical. Conversely, states far apart in Euclidean distance are more likely to be separated into different segments on many lines, yielding different hash codes. For a given seed σ, let τaut _aut and τca _ca be the trajectories of the AUT and the challenger agent, respectively. We apply the LSH function h to each state in these trajectories, yielding two sets of unique hash codes that represent the distinct regions explored by each agent: (5) Haut=h(st)∣st∈τaut,Hca=h(st)∣st∈τcaH_aut=\h(s_t) s_t∈ _aut\, H_ca=\h(s_t) s_t∈ _ca\ The inconsistency metric I(σ)I(σ) is calculated as the number of LSH hash codes commonly visited by both the AUT and the challenger agent, divided by the total number of unique hash codes visited by the AUT. (6) I(σ)=|Haut∩Hca||Haut|I(σ)= |H_aut∩ H_ca||H_aut| This energy function E(σ)E(σ) for optimality testing is defined as: (7) E(σ)=α⋅D(σ)+β⋅A(σ)+λ⋅(1−I(σ))E(σ)=α· D(σ)+β· A(σ)+λ·(1-I(σ)) where D(σ)D(σ) is the diversity score, A(σ)A(σ) is the regret (Equation 4), and I(σ)I(σ) is the inconsistency score (Equation 6). The terms α,β,λα,β,λ are weighting factors for these components. Mutation. Mutation is the process that generates a new test scenario (σ′σ ) from an existing seed (σ) by applying a slight random perturbation to the initial state vector represented by the seed. The nature of this perturbation is adapted to the specific initial state space of each environment. For instance, in the CartPole environment, a seed’s initial state is defined by a vector of four values: cart position, cart velocity, pole angle, and pole angular velocity. To mutate a CartPole seed, a small random value is added to each of these four components, creating a new, slightly different starting condition for the subsequent test run. The specific descriptions of the initial states for all environments are detailed in Section 4. Fuzzing. Algorithm 1 provides a detailed description of the differential fuzzing. It mainly consists of two parts: Initialization and Fuzzing Iteration. 1) Initialization. Delta generates the initial population of seeds S by randomly sampling the initial state of the environment based on a uniform distribution. Meanwhile, Delta initializes the Curiosity module to assign the diversity of each seed in the seed corpus. 2) Fuzzing Iteration. The core fuzzing loop proceeds as follows for each iteration until the time budget is exhausted: A seed σ is selected from the corpus S, where the probability of selecting any seed σi _i is proportional to its energy E(σi)E( _i) relative to the total energy of all seeds in the corpus: (8) P(σi)=E(σi)∑σj∈E(σj)P( _i)= E( _i) _ _j E( _j) The selected seed σ is then mutated by applying a slight random perturbation, yielding a new seed σ′σ . The AUT and challenger’s behavior, when initiated from σ′σ (i.e., its initial state s0′s _0), is subsequently monitored. Dynamic Thresholding. Manually setting fixed thresholds to identify “interesting” seeds for fuzzing can be challenging and often requires domain-specific expertise. Instead, Delta utilizes a dynamic thresholding mechanism during its differential fuzzing phase to determine which new seeds σ′σ are added to the corpus S. A seed σ′σ is considered interesting and added if it meets at least one of the following criteria: (1) Its diversity D(σ′)D(σ ) exceeds the average diversity D¯ D_S of seeds currently in the corpus (i.e., D(σ′)>D¯D(σ )> D_S), indicating it explores novel states. (2) Its regret A(σ′)A(σ ) is greater than 1 (i.e., A(σ′)>1A(σ )>1). This signifies that the challenger agent achieved a higher cumulative reward than the AUT for seed σ′σ , directly indicating an instance of AUT suboptimality. (3) Its inconsistency I(σ′)I(σ ), which measures trajectory overlap as defined in Equation 6, falls below the average inconsistency I¯ I_S of seeds in the corpus (i.e., I(σ′)<I¯I(σ )< I_S). A lower I(σ′)I(σ ) value highlights the greater behavioral divergence between the AUT and the challenger agent. Subsequently, at the end of each iteration, Delta updates the energy values (calculated using Equation 7) for all seeds remaining in the corpus S. This process of seed selection, mutation, evaluation, and corpus management continues until the allocated time budget is exhausted. 4. Experiment Setting Environment To assess the feasibility of Delta, we apply it to multiple case studies, including three classical control environments: CartPole, MountainCar, and Acrobot, as well as two MuJoCo environments: Hopper and Walker2D. These environments are widely used in previous work on testing DRL agents (85; 79; 41). • CartPole: The agent balances a pole on a cart by moving left or right. Success is keeping it upright; episodes last up to 500 steps. Initial states vary by cart position, cart velocity, pole angle, and angular velocity. • MountainCar: The objective is to navigate an underpowered car from a random starting position and zero velocity to a hilltop. The agent controls the car by applying directional forces. The agent receives a reward of -1 per step until success or 200 steps. • Acrobot: The agent swings a two-link pendulum to a target height. It receives -1 reward per step until reaching the target or 500 steps. The initial state involves the cosine and sine values of the angles and their corresponding angular velocities. • Hopper: A one-legged robot aims to hop forward by applying torques to its three actuated joints. Initial states include noise on joint angles and velocities. Rewards are based on forward velocity, survival, and penalties for excessive action. An episode terminates if the Hopper enters an “unhealthy” state or after 1000 timesteps. • Walker2D: A bipedal robot learns to walk forward by applying torques to its six actuated joints. Initial states include slight random variations in joint angles and velocities. Similar to Hopper, rewards favor forward progress and upright posture, and episodes end upon entering an “unhealthy” state or after 1000 timesteps. Safety Testing Oracle A safety violation or catastrophic failure is defined for each environment as follows: We define safety violations based on standard environment termination criteria (17; 18). For CartPole, failure occurs if the pole tilts >12∘>12 or the cart moves >2.4>2.4 units. MountainCar fails if the goal is not reached in 200 steps. Acrobot is considered unsatisfactory if the reward drops below -100. For Hopper and Walker2D, a violation occurs if the robot becomes “unhealthy” (e.g., falling or exceeding joint limits). AUT Implementation We train the AUT using stable and verified implementations of DRL algorithms from the d3rlpy open-source library (59). Specifically, for environments with discrete action spaces (CartPole, MountainCar, and Acrobot), we employ the Double Deep Q-Network (DDQN) algorithm (76). For environments with continuous action spaces (Hopper and Walker2D), we utilize the Soft Actor-Critic (SAC) algorithm (22). We confirmed the high performance of all AUTs prior to safety testing. We report the average cumulative reward and 95% confidence interval margin of error over 100 runs for each environment, with detailed results presented in Table 1. Challenger Implementation To train the challenger agent, we leverage offline Reinforcement Learning (RL) algorithms, a rapidly evolving subfield of RL. We experiment with three prominent offline RL methods: • BC (75) directly learns a policy by mimicking state-action pairs from a static dataset. • BCQ (21) mitigates distributional shift by constraining the policy to select actions similar to those in the provided data batch. • CQL (34) addresses the overestimation of Q-values for out-of-distribution actions by learning a conservative Q-function; it incorporates a regularizer that minimizes Q-values for actions not well-represented in the dataset while maximizing them for dataset actions. For the MuJoCo environments, we utilized hyperparameters from the D4RL replication package (19). Since D4RL does not provide hyperparameters for the classical control environments, we adopted the hyperparameters from the Scope-RL replication package (30). All experiments were implemented in Python 3.9 and conducted on an Ubuntu 22.04 server equipped with an AMD EPYC 7643 48-core Processor, 504 GB RAM, and four NVIDIA RTX A5000 GPUs. 5. Results RQ1: Which offline RL algorithm is most effective for training challenger agents to identify optimality bugs? Safety Testing. We first perform the safety testing phase to identify catastrophic failures in the AUT. Following common practice (44; 23; 83), each run is executed for two hours, which includes a 30-minute sampling phase to construct the initial seed corpus. We perform five independent runs of safety testing for each environment using distinct random seeds. Throughout the entire safety testing process, covering both the sampling and fuzzing phases, every trajectory executed by the AUT is recorded into the dataset D used for challenger training. Table 1 reports, for each environment, the average number of detected failures and the 95% confidence interval margin. Performance of Challenger Agents. We detail our experimental setup for training challenger agents as follows: 1) Dataset Generation: Safety testing is conducted five times per environment, yielding five distinct datasets for each environment. 2) Challenger Agent Training: Each of these five datasets was utilized to train three challenger agents using different offline RL algorithms (BC, BCQ, and CQL). This approach yielded a total of 75 challenger agents (5 environments×5 datasets×3 algorithms5 environments× 5 datasets× 3 algorithms). 3) Evaluation Metrics: We calculate the Mean Cumulative Reward of one DRL agent under 100 independent runs. The overall performance metrics for a specific offline RL algorithm in a given environment are calculated as the average of its five respective challenger agents. From Table 1, we can observe no single offline RL algorithm uniformly achieves the highest cumulative rewards across all environments; both CQL and BCQ frequently achieved comparable or superior performance. In general, the challenger agents perform very closely to the AUT. This comparable performance implies they are valuable and competent agents for conducting differential analysis with the AUT. Efficiency of Training Challenger Agents. The requirement to train a challenger agent using offline RL may raise concerns about computational overhead. In Figure 4, we illustrate the training time of the AUT and challenger agents. Since there are five challenger agents trained for each offline RL algorithm in each environment, we report the median value. However, we notice that the training time is almost identical across runs. In general, the training time for the challenger agents is significantly lower than that of the AUT across all environments. This efficiency gain is mainly due to the different learning paradigms employed (78). The AUT is trained using traditional reinforcement learning, which requires extensive and often time-consuming interaction with the environment to gather data and learn a policy. In contrast, the challenger agents learn exclusively from a pre-collected, static dataset of trajectories, eliminating the need for any further interaction with the environment. Table 1. Number of safety failures detected in the AUT, AUT success rate, and mean cumulative reward over 100 episodes for the AUT and challengers (BC, BCQ, CQL). Brackets [lower, upper] denote the 95% Wilson score CI for the success rate; ± denotes the 95% CI margin of error for the other columns. Bold values indicate the best-performing challenger agent (among BC, BCQ, and CQL) for each environment. Environment Number of Safety Failures of AUT AUT Success Rate (%) AUT Performance BC Performance BCQ Performance CQL Performance CartPole 77.4±10.377.4± 10.3 97.0[91.6,99.0]97.0\ [91.6,99.0] 495.7±6.8495.7± 6.8 497.1±2.5497.1± 2.5 496.0±2.6496.0± 2.6 499.8±1.6499.8± 1.6 MountainCar 167.2±8.7167.2± 8.7 99.0[94.6,99.8]99.0\ [94.6,99.8] −129.8±2.8-129.8± 2.8 −129.0±1.4-129.0± 1.4 −128.2±1.4-128.2± 1.4 -126.6±1.5-126.6± 1.5 Acrobot 1183.6±41.91183.6± 41.9 93.0[86.3,96.6]93.0\ [86.3,96.6] −81.3±3.0-81.3± 3.0 −83.7±2.1-83.7± 2.1 −84.2±3.2-84.2± 3.2 -83.3±1.5-83.3± 1.5 Hopper 154.4±9.9154.4± 9.9 97.0[91.6,99.0]97.0\ [91.6,99.0] 3162.3±60.93162.3± 60.9 3173.4±41.63173.4± 41.6 3204.4±32.43204.4± 32.4 3173.8±20.83173.8± 20.8 Walker2D 251.8±23.8251.8± 23.8 95.0[88.8,97.9]95.0\ [88.8,97.9] 3925.5±192.23925.5± 192.2 3961.3±95.83961.3± 95.8 4004.0±92.14004.0± 92.1 3996.5±68.53996.5± 68.5 (a) (b) (c) (d) (e) Figure 4. Running time of training the Agent Under Test (AUT), and the challenger agents (BC, BCQ, and CQL). Bar charts of the training time of the AUT and the challenger agents (BC, BCQ, and CQL) in each of the five environments. The notably long AUT training time in MountainCar stems from the environment’s challenging reward system. In MountainCar, an agent is penalized at every timestep (reward of -1) and only receives a positive reward upon reaching the hilltop goal. This lack of any intermediate positive feedback creates a major exploration challenge: at the beginning of the training, all actions appear equally poor to the agent. It cannot learn incrementally and must instead discover a rare, successful trajectory through prolonged random exploration, which accounts for the significant training time observed. Optimality Testing. Further, we evaluate the effectiveness of the different challenger agents (BC, BCQ, and CQL) in identifying optimality bugs. We conduct five differential fuzzing executions for every combination of environment and offline RL algorithm. In each of these five executions, the AUT is compared against one of the five previously generated challenger agents. Following the same setup as safety testing, each optimality testing execution had a duration of two hours. We focus on the following evaluation metrics: • # of Bugs: The total count of optimality issues. • # of Distinct Bugs: The space of the initial state is discretized into a grid by dividing each dimension into a predefined number of intervals. We use five intervals, following previous work (45), except for the MountainCar environment, where 100 intervals are employed due to its simpler nature. The number of Distinct Bugs is then the number of unique grid cells containing at least one detected bug. • Distance: Assesses the sparseness of the detected optimality issues within the state space. It is calculated as the average Euclidean distance over all pairs of scenarios where optimality issues were identified. Table 2. Experiment Results of RQ1. We compare the performance of different offline RL algorithms for training the challenger agents to detect optimality bugs. We report the Number of Bugs, the Number of Distinct Bugs, and the Distance. The average result over five runs is reported. The ± values represent the 95% confidence interval margin of error. Bold values indicate the best result in each column. # of Bugs CartPole MountainCar Acrobot Hopper Walker2D BC 79.4 ± 8.5 1243.2 ± 318.3 2900.0 ± 797.3 1231.8 ± 236.8 1251.4 ± 148.9 BCQ 105.6 ± 17.7 3752.0 ± 649.9 5450.0 ± 1241.7 1738.8 ± 315.6 1543.2 ± 146.6 CQL 84.4 ± 10.7 2144.0 ± 797.6 4811.0 ± 760.1 875.0 ± 123.5 778.6 ± 75.7 # of Distinct Bugs CartPole MountainCar Acrobot Hopper Walker2D BC 65.4 ± 9.8 19.8 ± 6.9 298.2± 12.2 1183.6 ± 228.8 1238.4 ± 150.8 BCQ 81.0 ± 9.8 58.2 ± 9.9 467.4 ± 10.0 1648.8 ± 280.8 1508.2 ± 213.4 CQL 67.4 ± 8.5 23.8 ± 5.6 432.8± 15.4 852.8 ± 115.8 771.0 ± 75.8 Distance CartPole MountainCar Acrobot Hopper Walker2D BC 0.067 ± 3e-3 0.019 ± 8e-3 0.155 ± 3e-3 0.014± 2e-4 0.018± 2e-4 BCQ 0.071± 3e-3 0.060± 2e-2 0.157± 3e-3 0.014± 2e-4 0.019± 2e-4 CQL 0.068± 2e-3 0.020± 8e-3 0.156± 3e-3 0.014± 2e-4 0.017± 2e-4 Results. For all evaluation metrics, we report the average performance over the five fuzzing executions and the margin of error at 95% confidence interval. Table 2 demonstrates the capability of BC, BCQ, and CQL in training challenger agents to detect optimality bugs. Across all five environments, the BCQ-trained challenger agents consistently proved to be the most effective at identifying optimality bugs. They detected the highest Number of Bugs and Number of Distinct Bugs in every environment. For instance, in the MountainCar environment, the BCQ challenger found an average of 3752.0 bugs, significantly outperforming both BC (1243.2) and CQL (2144.0). BCQ-trained challenger agents are therefore established as the default choice to form the oracle when implementing Delta. An interesting insight arises when comparing these bug detection results (Table 2) with the performance of the challenger agents (Table 1). In cases where CQL-trained challenger agents achieved the highest cumulative rewards, they were less effective at optimality bug detection than BCQ agents. This divergence suggests that the agent characteristics that lead to the highest task rewards are not necessarily the same as those that are most effective for uncovering optimality issues under Delta. Answer to RQ1: BCQ-trained challenger agents are the most effective for Delta in detecting optimality bugs, yielding an average of 2,517.9 issues per environment, and are therefore established as the default choice when implementing Delta. RQ2: How effective is Delta at detecting optimality bugs compared to baseline methods? Baseline Methods. To the best of our knowledge, Delta is the first work on this topic and there are no direct baselines. Metamorphic testing (e.g., Decictor (6)) relies on manually-defined domain-specific metamorphic relations. General ML testers (e.g., DeepGauge (43)) target static problems. Consequently, we select three state-of-the-art safety testing methods. These methods also prioritize exploration diversity, ensuring broad coverage of the state space. • CureFuzz (26): As we described in Section 3, CureFuzz is the state-of-the-art fuzz testing approach to uncover crashes of DRL agents. • MDPFuzz (55): A black-box fuzz testing framework for deep learning models that solve problems modeled as Markov Decision Processes. • GMT (38): A framework that utilizes a generative diffusion model to create test cases and uses novelty-based guidance to diversify agent behaviors. For a fair and direct comparison, all testing methodologies operated under identical conditions: they used the same BCQ-trained challenger agent (from RQ1) to form the differential testing oracle and were allotted a two-hour time budget. To ensure the statistical reliability of our findings, we repeated each experiment five times and report the average performance. Results. The experimental results are presented in Table 3. To summarize, Delta achieves the best overall performance when compared against the baseline methods across all environments and nearly all evaluation metrics. On average, Delta identifies 50.2% more optimality bugs across all environments. Statistical analysis using the Mann-Whitney U test (48) and Cohen’s d effect size (7) confirms that Delta’s superior performance in finding both the total number of bugs and distinct bugs is statistically significant and substantial. Regarding the total Number of Bugs (# of Bugs), Delta outperformed all baseline methods in all five environments, consistently identifying the highest total number of optimality issues. This highlights its effectiveness in locating instances of suboptimal agent behavior. The most significant advantage of Delta is its ability to find a diverse set of unique bugs. It found the highest number of distinct bugs (# of Distinct Bugs) in every environment, often by a large margin. For example, in the Hopper environment, Delta found more than double the number of distinct bugs compared to the closest competitor (1648.8 vs. 769.2). In the Acrobot environment, Delta identifies over five times more distinct bugs than the runner-up (467.4 vs. 93.2). On the Distance metric, which measures the spread of the discovered bugs, Delta’s performance is competitive but not universally dominant. It achieved the highest or joint-highest distance score in four of the five environments (MountainCar, Acrobot, Hopper, and Walker2D). The only environment where a baseline method achieved a distinctly better result was CartPole. In CartPole, the generative approach of GMT found fewer distinct bugs than Delta (43.6 vs. 81.0), yet it achieved the highest distance score (0.084 vs. 0.071). This implies that the few bugs GMT finds are likely to be far apart and unrelated. In contrast, Delta finds a dense cluster of many unique bugs that are, on average, closer together. In summary, while not leading in every single metric, the consistent dominance of Delta in discovering the most total and the most unique bugs makes it the most effective testing framework overall. Answer to RQ2: Delta significantly outperforms the baseline methods in terms of nearly all evaluation metrics in optimality bug detection. On average, Delta identifies 50.2% more optimality bugs. Table 3. Performance comparison of Delta against baseline methods (MDPFuzz, CureFuzz, and GMT) in detecting optimality bugs, including the total number of bugs found (# of Bugs), the number of unique bugs (# of Distinct Bugs), and Distance. All results are averaged over five runs, with the ± values representing the 95% confidence interval margin of error. Bold values indicate the best performance in each row. The percentages in parentheses for Delta denote its improvement over the best-performing baseline for that metric. Environment # of Bugs Delta MDPFuzz CureFuzz GMT CartPole 105.6 ± 17.7 (+46.7%) 54.0 ± 17.6 72.0 ± 8.1 60.6 ± 15.5 MountainCar 3752.0 ± 649.9 (+91.5%) 1854.4 ± 600.6 1959.2 ± 383.6 739.8 ± 559.1 Acrobot 5450.0 ± 1241.7 (+40.9%) 3869.0 ± 1425.4 2556.6 ± 861.9 1850.8 ± 598.5 Hopper 1738.8 ± 315.6 (+37.9%) 674.2 ± 101.3 815.0 ± 160.1 1261.2 ± 286.2 Walker2D 1543.2 ± 146.6(+34.1%) 598.6 ± 141.1 838.8 ± 342.7 1150.4 ± 193.9 Environment # of Distinct Bugs Delta MDPFuzz CureFuzz GMT CartPole 81.0 ± 9.8 (+71.6%) 43.4 ± 8.2 47.2 ± 3.9 43.6 ± 7.9 MountainCar 58.2 ± 9.9 (+19.3%) 48.8 ± 6.7 47.8 ± 10.3 35.6 ± 4.7 Acrobot 467.4 ± 10.0 (+401.5%) 93.2 ± 14.9 75.8 ± 9.3 63.6 ± 5.2 Hopper 1648.8 ± 280.8 (+114.4%) 641.8 ± 103.1 769.2 ± 149.1 616.4 ± 124.7 Walker2D 1508.2 ± 213.4 (+81.8%) 592.4 ± 140.0 829.6 ± 339.1 487.2 ± 128.7 Environment Distance Delta MDPFuzz CureFuzz GMT CartPole 0.071 ± 3e-3 0.070 ± 3e-3 0.070 ± 3e-3 0.084 ± 3e-3 MountainCar 0.060± 2e-2 0.052 ± 1e-2 0.060 ± 2e-2 0.025 ± 5e-3 Acrobot 0.157± 3e-3 (+61.9%) 0.071 ± 3e-3 0.068 ± 2e-3 0.097 ± 5e-3 Hopper 0.014± 2e-4 0.014 ± 1e-4 0.014 ± 1e-4 0.010 ± 8e-4 Walker2D 0.019± 2e-4 (+11.8%) 0.017 ± 1e-4 0.017 ± 2e-4 0.014 ± 3e-4 RQ3: To what extent does each component of the energy function (i.e., diversity, regret, and inconsistency) contribute to Delta’s optimality bug detection performance? To investigate this research question, we implement and experiment with three ablated variants of Delta (w/o diversity, w/o regret, w/o inconsistency). Table 4 summarizes the experimental results. The removal of the Regret component (w/o-Regret) causes the most substantial drop in the total number of bugs found. For example, in the Acrobot environment, the bug count falls by approximately 59% from 5450.0 to 2238.4. This pattern confirms that prioritizing test cases where the challenger agent shows a clear performance advantage is the single most effective strategy for finding a high volume of optimality bugs. The Diversity component is crucial for discovering a wide range of unique bugs. Its removal (w/o-Diversity) leads to a dramatic reduction in the number of Distinct Bugs. This is most clearly seen in the CartPole environment, where the number of distinct bugs is more than halved, dropping from 81.0 to 39.4. Further, removing Inconsistency (w/o-Inconsistency) also results in a noticeable decrease in both total and distinct bugs across all environments. The ablation study concludes that all three energy components, i.e., Regret, Diversity, and Inconsistency, are integral to the success of Delta. They all play synergistic and distinct roles. Regret is the primary factor for the quantity of bugs. Table 4. Performance comparison of Delta against ablated variants (w/o diversity, w/o regret, w/o inconsistency) in detecting optimality bugs, including the total number of bugs found (# of Bugs), the number of unique bugs (# of Distinct Bugs), and Distance. All results are averaged over five runs, with the ± values representing the 95% confidence interval margin of error. Bold values indicate the best performance in each column. Method # of Bugs CartPole MountainCar Acrobot Hopper Walker2D Delta 105.6± 17.7 3752.0± 649.9 5450.0± 1241.7 1738.8± 315.6 1543.2± 146.6 w/o-Diversity 93.8± 7.7 2373.0± 269.4 2963.0± 446.4 1464.6± 271.6 1252.0± 311.7 w/o-Regret 74.6± 11.3 2328.0± 252.6 2238.4± 165.9 1409.8± 214.1 1009.4± 247.9 w/o-Inconsistency 67.6± 9.0 2509.2± 526.3 2532.4± 295.2 1419.4± 350.9 1147.2± 206.0 Method # of Distinct Bugs CartPole MountainCar Acrobot Hopper Walker2D Delta 81.0± 9.8 58.2± 9.9 467.4± 10.0 1648.8± 280.8 1508.2± 213.4 w/o-Diversity 39.4± 2.4 43.8± 6.4 343.2± 38.8 1329.4± 197.9 1235.8± 308.7 w/o-Regret 63.0± 8.9 48.0± 10.5 409.4± 17.2 1299.0± 280.8 895.6± 240.2 w/o-Inconsistency 60.0± 7.7 47.6± 10.2 448.4± 16.7 1315.8± 212.3 1037.4± 155.7 Method Distance CartPole MountainCar Acrobot Hopper Walker2D Delta 0.071± 3e-3 0.060± 2e-2 0.157± 3e-3 0.014± 2e-4 0.019± 2e-4 w/o-Diversity 0.070± 1e-3 0.057± 2e-2 0.157± 3e-3 0.014± 1e-4 0.018± 2e-4 w/o-Regret 0.069± 2e-3 0.061± 2e-2 0.158± 3e-3 0.014± 1e-4 0.019± 2e-4 w/o-Inconsistency 0.070± 2e-3 0.060± 2e-3 0.157± 3e-3 0.014± 1e-4 0.018± 2e-4 Answer to RQ3: The ablation study demonstrates that Diversity, Regret, and Inconsistency are all vital for Delta’s performance. Regret is the single most crucial factor for maximizing the total number of optimality bugs found. RQ4: How useful are the identified optimality bugs to improve the performance of DRL agents? We investigate whether the optimality bugs identified by Delta can be leveraged to enhance the performance of the AUT. The core strategy involves fine-tuning the AUT using the superior trajectories generated by the challenger agent during the optimality testing phase. The experimental methodology is structured into the following stages: (1) Curation of a Superior Trajectory Dataset: First, based on the identified optimality bugs, we create a new dataset, superiorD_superior. This dataset is formed by collecting every trajectory where the challenger agent outperformed the AUT. This dataset is thus composed entirely of high-performing trajectories that demonstrate policies superior to the AUT’s original behavior. (2) Offline Fine-Tuning: The original AUTs are further trained using this new dataset. Notice that the AUT cannot be effectively fine-tuned on the static dataset superiorD_superior using its original online training algorithm (e.g., DDQN or SAC), as these methods perform poorly in offline settings (35; 34; 16). Therefore, our approach is to retain the AUT’s existing neural network architecture but switch its training objective to one specifically designed for the offline paradigm. We utilize Conservative Q-Learning (CQL) for this purpose. By applying the CQL objective, we can update the agent’s existing policy using only the static data, ensuring the fine-tuning process is both stable and robust. Finally, we refer to this fine-tuned agent as AUT-Improved. Result. We evaluate AUT-Improved in the same environment used to train the original AUT, with no modifications to the transition dynamics, reward function, or action space. The primary metric for comparison is the mean cumulative reward, averaged over 100 independent runs. The experiment results are detailed in Table 5, where AUT-Improved shows a consistent performance increase against the original AUT across all five environments. The enhancements were particularly notable in several cases: the CartPole agent achieved the theoretical maximum score of 500 with perfect stability (± 0.0), and the Walker2D agent’s performance increased by over 12%. These results confirm that the optimality bugs identified by Delta are highly useful for improving cumulative reward. Repairing the Safety Failure. We further experiment with the robustness of AUT-Improved. For each environment, AUT-Improved is evaluated again on the known safety failure-inducing scenarios from the safety testing stage (the safety testing stage from RQ1). The result demonstrates the enhanced robustness of the AUT. AUT-Improved exhibits a substantial reduction in the number of such failures. In CartPole and MountainCar, all safety failures are repaired. Furthermore, AUT-Improved in Walker2D reduces its safety failures by more than 97%. These results demonstrate that our fine-tuning process not only leads to policies that are higher-performing but also significantly more robust. Re-evaluation with CureFuzz. To further validate the robustness of the AUT-Improved, we launch a new fuzzing campaign with CureFuzz against the AUT-Improved. The fuzzing campaign follows the same execution setup described in RQ1. Table 5 presents the results. AUT-Improved consistently exhibits fewer safety failures than the original AUT. Notably, in CartPole, the AUT-Improved is completely robust, with zero safety failures detected. In Acrobot, the number of failures is reduced by approximately 77.6% (from 1183.6 to 265). Answer to RQ4: The identified optimality bugs are highly useful in improving the performance of the AUT. An independent re-evaluation with CureFuzz confirms that the AUT-Improved is substantially more robust, reducing safety failures by up to 100% across all environments. Table 5. Comparison of the original Agent Under Test (AUT) and the AUT-Improved. The first two columns report the mean cumulative reward for each agent. The third and fourth columns present the average number of safety failures on known safety-critical scenarios, with percentages in parentheses quantifying the failure reduction after repair. The last column reports the results of the re-evaluation of CureFuzz against the AUT-Improved. All ± values represent the 95% confidence interval margin of error. Environment Mean Reward (AUT) Mean Reward (AUT-Improved) Safety Failures of AUT Safety Failures of AUT-Improved (after repair) CureFuzz Re-eval of AUT-Improved CartPole 495.7 ± 6.8 500 ± 0.0 77.4 ± 10.3 0.0(-100.0%) ± 0.0 0.0 ± 0.0 MountainCar -129.8 ± 2.8 -125.4 ± 3.2 167.2 ± 8.7 0.0(-100.0%) ± 0.0 37.4 ± 6.4 Acrobot -81.3 ± 3.0 -75.2 ± 2.9 1183.6 ± 41.9 118.2(-90.0%) ± 11.3 265.0 ± 18.5 Hopper 3162.3 ± 60.9 3247.0 ± 30.5 154.4 ± 9.9 40.0(-74.1%) ± 6.0 106.2 ± 19.5 Walker2D 3925.5 ± 192.2 4399.3 ± 93.8 251.8 ± 23.8 5.6(-97.8%) ± 2.4 105.0 ± 17.2 6. Discussion 6.1. Behavior Clustering In this subsection, we analyze the AUT’s behavior on bug-triggering initial states. We group the AUT’s trajectories from these initial states into clusters and treat each cluster as a behavioral pattern, following prior work (37). Specifically, we adopt unsupervised hierarchical clustering (50): Each trajectory initially forms a singleton cluster, and at every step, the pair of clusters with the smallest pairwise distance is merged. This bottom-up process continues until all trajectories collapse into a single cluster, yielding a dendrogram with maximum merge distance dmaxd_ . For pairwise distance, we use Dynamic Time Warping (DTW) (3) computed over the state sequences, which natively accommodates trajectories of varying length without padding. Unlike K-means (40), which requires the number of clusters to be fixed in advance, hierarchical clustering imposes no target count; clusters emerge naturally from the trajectory structure given a cut threshold τ, expressed as a fraction of dmaxd_ . We report results at three values, τ∈0.10,0.20,0.30τ∈\0.10,0.20,0.30\, in Table 6. Table 6. Number of behavioral patterns detected by Delta across the five environments at three relative distance thresholds τ∈0.10,0.20,0.30τ∈\0.10,0.20,0.30\. Results are averaged over five independent runs; ± values denote the 95% confidence interval margin of error. τ CartPole MountainCar Acrobot Hopper Walker2D 0.10 100.2 ± 15.2 4.2 ± 1.0 279.0 ± 10.5 1029.8 ± 139.0 1505.6 ± 204.6 0.20 25.4 ± 3.4 3.2 ± 0.6 130.6 ± 8.9 93.8 ± 6.4 1111.2 ± 152.7 0.30 3.8 ± 1.4 3.0 ± 0.0 54.8 ± 2.4 56.6 ± 5.9 445.2 ± 116.3 Findings. The number of behavioral patterns varies with both the environment and the cut threshold τ, decreasing monotonically as τ grows. MountainCar yields only 3–4 behavioral patterns across the three thresholds, reflecting the environment’s low-dimensional state space and small discrete action space. In contrast, higher-dimensional environments such as Walker2D exhibit hundreds of distinct behavioral patterns even at τ=0.30τ=0.30. 6.2. Extension to Non-Deterministic Environments Our experiment setting assumes a deterministic environment, which is the same setting as in previous work (14; 85). However, many real-world environments exhibit stochasticity due to random disturbances or noisy dynamics. To demonstrate that Delta remains applicable in such settings, we present a natural extension of our oracle formulation based on statistical hypothesis testing, and empirically validate it on a non-deterministic benchmark. Statistical Oracle Formulation. For a given initial state s0s_0, we execute both the AUT and the challenger agent N times (default N=5N=5), obtaining two sets of cumulative rewards: ℛaut=Raut(1),…,Raut(N)R_aut=\R^(1)_aut,…,R^(N)_aut\ and ℛca=Rca(1),…,Rca(N)R_ca=\R^(1)_ca,…,R^(N)_ca\. An optimality bug is reported when the challenger’s rewards are statistically significantly higher than the AUT’s: (9) OptimalityBug(s0)=[p<α]OptimalityBug(s_0)=1 [p<α ] where p is the p-value from a one-sided Mann-Whitney U test (48) with the alternative hypothesis H1:Rca>RautH_1:R_ca>R_aut, and α is the significance level (default α=0.05α=0.05). Empirical Validation. To demonstrate the effectiveness of the statistical oracle, we conducted additional experiments on Hopper-Noise, a non-deterministic variant of Hopper that introduces random wind disturbances (42; 57). Following the same protocol as RQ1, the AUT achieves a mean reward of 3086.9±61.03086.9± 61.0 over 100 episodes, with 477.4±91.2477.4± 91.2 safety failures detected during safety testing. For optimality testing, we adopt the same two-hour budget per run and repeat each experiment five times, with N=5N=5 rollouts per initial state for the statistical oracle. Table 7 reports the number of optimality bugs detected. BCQ remains the strongest offline RL algorithm for training the challenger. Using the BCQ challenger, Delta detects 171% more optimality bugs than the best-performing baseline, confirming that our differential testing methodology remains effective and statistically sound under non-determinism. Table 7. Optimality bug detection on the non-deterministic Hopper-Noise environment. Top: comparison of offline RL algorithms for training the challenger. Bottom: Delta against baseline methods. All results are averaged over five runs; ± values denote the 95% confidence interval margin of error. Challenger Algorithm BC BCQ CQL # Bugs 44.6 ± 11.3 87.8 ± 23.3 65.2 ± 22.8 Testing Method Delta CureFuzz GMT MDPFuzz # Bugs 87.8 ± 23.3 (+171%) 32.4 ± 11.3 25.2 ± 6.7 24.6 ± 6.4 6.3. Reliance on Thorough Safety Testing The challenger’s quality depends on the diversity of the safety-testing dataset D used to train it. As long as safety testing is conducted thoroughly, D can be trusted to be diverse; if not, this would imply that safety testing itself has failed to adequately explore the AUT’s behavior—a far more serious problem than missed optimality issues. Delta therefore operates under an implicit precondition: it should be applied only when developers are satisfied that safety testing has been rigorous enough to expose the AUT’s safety-critical failures. Under this precondition, D can be reasonably regarded as diverse, and Delta’s optimality testing serves as a complementary layer on top of safety testing. In practice, we adopt a two-hour safety-testing budget per environment, consistent with prior work (44; 23; 83). Developers working with more complex environments may need to extend this budget accordingly. How long a testing campaign should run is itself a long-standing open question in the software testing community (4; 39). 6.4. Practical Considerations When Can the Challenger Surpass the AUT? The challenger can outperform the AUT when: (1) the AUT has not converged to the optimal policy, and room exists for an alternative policy to outperform it in certain regions of the state space; (2) the safety testing dataset naturally contains diverse scenarios including edge cases. This diversity mitigates distributional shift and enables stitching of suboptimal trajectory segments into improved policies (19). Overall, the challenger need not outperform the AUT globally. It functions as a local oracle: surpassing the AUT from specific initial states suffices to reveal faults. When the Challenger Fails to Outperform. Importantly, this oracle yields value regardless of outcome. A superior challenger exposes optimality issues and guides policy improvement. When the AUT is already near-optimal, the challenger is unlikely to outperform it. This does not indicate oracle failure; rather, it provides empirical confirmation of the high performance of the AUT. The goal of testing is to assess the optimality, not necessarily to find faults. Confirming that the AUT performs well is as valuable as uncovering defects. Training Efficiency. Our efficiency comparison (Section 5) reports only offline training time, excluding the data collection performed during safety testing. Safety testing is a prerequisite for deploying DRL systems, and is not optional, as safety concerns take precedence over optimality. The data collection period is therefore a zero-cost byproduct of an already-required process. Repair vs. Replace. RQ4 shows that the optimality bugs detected by Delta have practical value for repairing the AUT via fine-tuning. A natural question is whether the challenger should instead be deployed directly as the final agent. We view fine-tuning and replacement as complementary downstream choices enabled by Delta’s bug detection. When the AUT’s weights are inaccessible, fine-tuning is not feasible, and the challenger becomes a candidate for direct deployment. The choice between these modes is ultimately a deployment decision determined by access constraints and operational requirements. 7. Threats to Validity One threat to internal validity is the randomness of the testing algorithms used in our experiments. To mitigate it, we repeat each experiment five times and use appropriate statistical tests to account for both statistical significance and effect size. When implementing CureFuzz (26), MDPFuzz (55), and GMT (38), we reuse their officially released replication packages. In terms of external validity, one threat is that the results of our analysis may not be generalizable. To mitigate it, we have carefully selected diverse tasks and diverse AUTs to evaluate our method, which ensures that our results are not biased. One potential threat to construct validity is that the evaluation metrics may not fully capture the performance of our baselines. The number of bugs, distance, and distinct bugs are widely adopted metrics for fuzzing DRL agents (55; 26; 45). A further threat to construct validity concerns how optimality bugs are counted. Following established practice in DRL testing (55; 26; 45), we count bugs at the granularity of the initial state, as initial states are concrete, reproducible inputs that developers can directly use for debugging and retraining. However, counting at this granularity does not directly reflect the diversity of underlying agent behaviors. To complement this initial-state count, we additionally report a behavioral-pattern count in Section 6.1. Another threat to construct validity concerns the possibility of false negatives. Delta’s challenger is trained on data collected during safety testing; if the safety tester underexplores certain regions of the state space, the challenger may be undertrained in those regions and fail to surpass the AUT even where the AUT is genuinely suboptimal. To mitigate this, Delta is designed to integrate with any state-of-the-art safety testing method, and our current implementation adopts CureFuzz (26), which explicitly promotes exploration diversity via a curiosity-driven mechanism. We further discuss this dependency in Section 6.3. Nonetheless, Delta’s reported bug counts should be interpreted as a sound-but-not-complete lower bound on the AUT’s optimality issues. 8. Related Work Safety Testing of DRL Agents. Pang et al. introduce MDPFuzz (55), which represents the pioneering effort in black-box fuzz testing for deep learning models addressing Markov Decision Processes, of which DRL agents are a straightforward application. He et al. further propose CureFuzz (26). CureFuzz proposes a curiosity mechanism to measure the novelty of a scenario, which aims to reveal a diverse set of crash-triggering scenarios. Li et al. (38) present GMT, a testing framework that utilizes a generative diffusion model (80) to generate practical test cases and novelty-based guidance, thereby diversifying agent behaviors and improving test effectiveness. Shi et al. (62) propose Synthify, which accelerates the falsification of AI-enabled control systems by synthesizing lightweight proxy programs that approximate the neural controller. Complementary to offline testing, they further synthesize efficient and permissive programmatic runtime shields that correct unsafe actions of neural policies at deployment time (61). As multi-agent systems are increasingly adopted for complex real-world tasks (25), testing their reliability has attracted growing attention. Ma et al. (44) focus on testing multi-agent DRL systems and propose MASTest, which incorporates both individual diversity and team diversity. Furthermore, they propose AdvTest (45) to evaluate competitive game agents through constraint-guided adversarial agent training. Search-Based Testing of DRL Agents. Zolfagharian et al. (85) propose STARLA, which uses a genetic algorithm to find faulty episodes and actions leading to crashes. Tappler et al. (72) introduce a search-based testing approach for RL agents with stochastic policies. Their method employs a depth-first backtracking search algorithm to identify reference traces that solve RL tasks and boundary states leading to unsafe states. Tappler et al. (73) further present a framework that integrates search-based testing with Reinforcement Learning from Demonstrations (RLfD) (51). Mutation Testing of DRL Agents. Tambon et al. (69) and Lu et al. (41) introduce several RL-specific mutation operators. Thomas et al. (74) introduce muPRL, which utilizes a systematic taxonomy of real-world faults to design specialized mutation operators and evaluate the effectiveness of testing suites in detecting realistic DRL bugs. Metamorphic Testing of DRL Agents. Eniser et al. apply metamorphic testing using manually designed relaxations as metamorphic oracles to assess DRL policies (14). Subsequently, Eisenhut et al. introduce new metamorphic oracles alongside a search-based testing method to identify bugs in DRL agents (13). More recently, Cheng et al. propose Decictor (6), a method designed to generate non-optimal decision scenarios, specifically in cases where an autonomous driving system fails to plan optimal paths for autonomous vehicles (AVs). A limitation of Decictor is its reliance on pre-existing knowledge of some optimal paths and domain-specific metamorphic relations. In contrast, our work tackles optimality issues using a differential testing approach, making it more broadly applicable. 9. Conclusion and Future Work In this paper, we introduced Delta, a two-phase testing framework that automatically identifies both safety-critical and optimality bugs. Delta first conducts safety testing on the AUT while collecting its interaction trajectories. Subsequently, these trajectories are used to train a challenger agent via Offline Reinforcement Learning; if the challenger achieves higher cumulative rewards, an optimality issue is flagged in the AUT. Our extensive evaluations across five diverse environments demonstrated Delta’s effectiveness in uncovering optimality issues. Challenger agents trained with BCQ proved most effective for Delta in identifying an average of 2,518 optimality issues per environment, and Delta significantly outperformed baseline methods. In future work, we plan to evaluate Delta on more complex real-world domains, such as autonomous driving, and to explore emerging offline RL algorithms for training stronger challenger agents. 10. Data Availability A replication package, including source code, experimental datasets, and detailed results, is available on Figshare (2) and can be accessed at https://doi.org/10.6084/m9.figshare.29196884.v1. References Arulkumaran et al. (2017) K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath Deep reinforcement learning: a brief survey. IEEE Signal Processing Magazine 34 (6), p. 26–38. External Links: Document Cited by: §1. Authors (2026) A. Authors Replication Package for "Learning from the Test: Self-Referential Differential Testing for Deep RL Agents". Figshare. External Links: Document, Link Cited by: §10. Berndt and Clifford (1994) D. J. Berndt and J. Clifford Using dynamic time warping to find patterns in time series. In Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining, p. 359–370. Cited by: §6.1. Böhme (2019) M. Böhme Assurances in software testing: a roadmap. In 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), p. 5–8. External Links: Document Cited by: §6.3. Burda et al. (2019) Y. Burda, H. Edwards, A. Storkey, and O. Klimov Exploration by random network distillation. In International Conference on Learning Representations, External Links: Document Cited by: §3.3. Cheng et al. (2025) M. Cheng, X. Xie, Y. Zhou, J. Wang, G. Meng, and K. Yang Decictor: towards evaluating the robustness of decision-making in autonomous driving systems. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering, p. 424–436. External Links: Document, ISBN 9798331505691, Link Cited by: §5, §8. Cohen (1992) J. Cohen Statistical power analysis. Current Directions in Psychological Science 1 (3), p. 98–101. External Links: Document Cited by: §5. Coronato et al. (2020) A. Coronato, M. Naeem, G. De Pietro, and G. Paragliola Reinforcement learning for intelligent healthcare applications: a survey. Artificial Intelligence in Medicine 109, p. 101964. External Links: Document Cited by: §1. Cybenko (1989) G. Cybenko Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems 2 (4), p. 303–314. External Links: Document Cited by: §3.4.3. Danielsson (1980) P. Danielsson Euclidean distance mapping. Computer Graphics and Image Processing 14 (3), p. 227–248. External Links: Document Cited by: §3.4.3. Ding and Dong (2020) Z. Ding and H. Dong Challenges of reinforcement learning. Deep Reinforcement Learning: Fundamentals, Research and Applications, p. 249–272. External Links: Document Cited by: §1. Dulac-Arnold et al. (2021) G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning 110 (9), p. 2419–2468. External Links: Document Cited by: §1. Eisenhut et al. (2023) J. Eisenhut, Á. Torralba, M. Christakis, and J. Hoffmann Automatic metamorphic test oracles for action-policy testing. In Proceedings of the International Conference on Automated Planning and Scheduling, Vol. 33, p. 109–117. External Links: Document Cited by: §8. Eniser et al. (2022) H. F. Eniser, T. P. Gros, V. Wüstholz, J. Hoffmann, and M. Christakis Metamorphic relations via relaxations: an approach to obtain oracles for action-policy testing. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, p. 52–63. External Links: Document Cited by: §6.2, §8. Evans and Savoia (2007) R. B. Evans and A. Savoia Differential testing: a new approach to change detection. In The 6th Joint Meeting on European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering: Companion Papers, p. 549–552. External Links: Document Cited by: §1. Fang et al. (2022) X. Fang, Q. Zhang, Y. Gao, and D. Zhao Offline reinforcement learning for autonomous driving with real world driving data. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), p. 3417–3422. External Links: Document Cited by: §2.2, §5. Farama Foundation (2024a) Farama Foundation Acrobot. Note: https://gymnasium.farama.org/environments/classic_control/acrobot/ Cited by: §4. Farama Foundation (2024b) Farama Foundation Hopper. Note: https://gymnasium.farama.org/environments/mujoco/hopper/ Cited by: §4. Fu et al. (2020) J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine D4rl: datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219. External Links: Document Cited by: §2.2, §4, §6.4. Fuchs et al. (2021) F. Fuchs, Y. Song, E. Kaufmann, D. Scaramuzza, and P. Dür Super-human performance in gran turismo sport using deep reinforcement learning. IEEE Robotics and Automation Letters 6 (3), p. 4257–4264. External Links: Document Cited by: §1. Fujimoto et al. (2019) S. Fujimoto, D. Meger, and D. Precup Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, p. 2052–2062. External Links: Document Cited by: §1, 2nd item. Haarnoja et al. (2018) T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, p. 1861–1870. External Links: Document Cited by: §4. Haq et al. (2023) F. U. Haq, D. Shin, and L. C. Briand Many-objective reinforcement learning for online testing of dnn-enabled systems. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), p. 1814–1826. External Links: Document Cited by: §5, §6.3. He et al. (2026) J. He, J. Shi, T. Y. Zhuo, C. Treude, J. Sun, Z. Xing, X. Du, and D. Lo LLM-as-a-judge for software engineering: literature review, vision, and the road ahead. ACM Transactions on Software Engineering and Methodology. External Links: Document Cited by: §1. He et al. (2025) J. He, C. Treude, and D. Lo LLM-based multi-agent systems for software engineering: literature review, vision, and the road ahead. ACM Transactions on Software Engineering and Methodology 34 (5), p. 1–30. External Links: Document Cited by: §1, §8. He et al. (2024) J. He, Z. Yang, J. Shi, C. Yang, K. Kim, B. Xu, X. Zhou, and D. Lo Curiosity-driven testing for sequential decision-making process. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, p. 1–14. External Links: Document Cited by: §1, §1, §1, §3.2, §3.3, §3.4.3, 1st item, §7, §7, §8. Ibarz et al. (2021) J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research 40 (4-5), p. 698–721. External Links: Document Cited by: §1. Jafari et al. (2021) O. Jafari, P. Maurya, P. Nagarkar, K. M. Islam, and C. Crushev A survey on locality sensitive hashing algorithms and their applications. arXiv preprint arXiv:2102.08942. External Links: Document Cited by: §3.4.3. Kiran et al. (2021) B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez Deep reinforcement learning for autonomous driving: a survey. IEEE Transactions on Intelligent Transportation Systems 23 (6), p. 4909–4926. External Links: Document Cited by: §1. Kiyohara et al. (2024) H. Kiyohara, R. Kishimoto, K. Kawakami, K. Kobayashi, K. Nakata, and Y. Saito Towards assessing and benchmarking risk-return tradeoff of off-policy evaluation. In International Conference on Learning Representations, External Links: Document Cited by: §4. Kober et al. (2013) J. Kober, J. A. Bagnell, and J. Peters Reinforcement learning in robotics: a survey. The International Journal of Robotics Research 32 (11), p. 1238–1274. External Links: Document Cited by: §1. Kostrikov et al. (2022) I. Kostrikov, A. Nair, and S. Levine Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, External Links: Document Cited by: §1, §2.2. Kumar et al. (2023) A. Kumar, A. Singh, F. Ebert, M. Nakamoto, Y. Yang, C. Finn, and S. Levine Pre-training for robots: offline rl enables learning new tasks from a handful of trials. In Robotics: Science and Systems, External Links: Document Cited by: §2.2. Kumar et al. (2020) A. Kumar, A. Zhou, G. Tucker, and S. Levine Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems 33, p. 1179–1191. External Links: Document Cited by: §1, 3rd item, §5. Lee et al. (2022) S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, p. 1702–1712. External Links: Document Cited by: §3.4.1, §5. Levine et al. (2020) S. Levine, A. Kumar, G. Tucker, and J. Fu Offline reinforcement learning: tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643. External Links: Document Cited by: §1, §2.2, §3.4.1, §3.4.2. Li et al. (2024) Z. Li, R. Krohn, T. Chen, A. Ajay, P. Agrawal, and G. Chalvatzaki Learning multimodal behaviors from scratch with diffusion policy gradient. Advances in Neural Information Processing Systems 37, p. 38456–38479. External Links: Document Cited by: §6.1. Li et al. (2023) Z. Li, X. Wu, D. Zhu, M. Cheng, S. Chen, F. Zhang, X. Xie, L. Ma, and J. Zhao Generative model-based testing on decision-making policies. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), p. 243–254. External Links: Document Cited by: §1, §1, §1, §3.3, 3rd item, §7, §8. Lipp et al. (2023) S. Lipp, D. Elsner, S. Kacianka, A. Pretschner, M. Böhme, and S. Banescu Green fuzzing: a saturation-based stopping criterion using vulnerability prediction. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, p. 127–139. External Links: Document Cited by: §6.3. Lloyd (1982) S. Lloyd Least squares quantization in pcm. IEEE Transactions on Information Theory 28 (2), p. 129–137. External Links: Document Cited by: §6.1. Lu et al. (2022) Y. Lu, W. Sun, and M. Sun Towards mutation testing of reinforcement learning systems. Journal of Systems Architecture 131, p. 102701. External Links: Document Cited by: §4, §8. Luo et al. (2025) B. Luo, A. Pettet, A. Laszka, A. Dubey, and A. Mukhopadhyay Scalable decision-making in stochastic environments through learned temporal abstraction. In The Thirteenth International Conference on Learning Representations, External Links: Document, Link Cited by: §6.2. Ma et al. (2018) L. Ma, F. Juefei-Xu, F. Zhang, J. Sun, M. Xue, B. Li, C. Chen, T. Su, L. Li, Y. Liu, et al. Deepgauge: multi-granularity testing criteria for deep learning systems. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, p. 120–131. External Links: Document Cited by: §5. Ma et al. (2024a) X. Ma, Y. Wang, J. Wang, X. Xie, B. Wu, S. Li, F. Xu, and Q. Wang Enhancing multi-agent system testing with diversity-guided exploration and adaptive critical state exploitation. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, p. 1491–1503. External Links: Document Cited by: §5, §6.3, §8. Ma et al. (2024b) X. Ma, Y. Wang, J. Wang, X. Xie, B. Wu, Y. Yan, S. Li, F. Xu, and Q. Wang Diversity-oriented testing for competitive game agent via constraint-guided adversarial agent training. IEEE Transactions on Software Engineering. External Links: Document Cited by: 2nd item, §7, §7, §8. Matsuo et al. (2022) Y. Matsuo, Y. LeCun, M. Sahani, D. Precup, D. Silver, M. Sugiyama, E. Uchibe, and J. Morimoto Deep learning, reinforcement learning, and world models. Neural Networks 152, p. 267–275. External Links: Document Cited by: §1. McKeeman (1998) W. M. McKeeman Differential testing for software. Digital Technical Journal 10 (1), p. 100–107. Cited by: §1. McKnight and Najab (2010) P. E. McKnight and J. Najab Mann-whitney u test. The Corsini Encyclopedia of Psychology, p. 1–1. External Links: Document Cited by: §5, §6.2. Mnih et al. (2013) V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Riedmiller Playing atari with deep reinforcement learning. CoRR abs/1312.5602. External Links: Document, Link, 1312.5602 Cited by: §1. Murtagh and Contreras (2012) F. Murtagh and P. Contreras Algorithms for hierarchical clustering: an overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 2 (1), p. 86–97. External Links: Document Cited by: §6.1. Nair et al. (2018) A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference on Robotics and Automation (ICRA), p. 6292–6299. External Links: Document Cited by: §8. Nguyen and La (2019) H. Nguyen and H. La Review of deep reinforcement learning for robot manipulation. In 2019 Third IEEE International Conference on Robotic Computing (IRC), p. 590–595. External Links: Document Cited by: §1. Nguyen et al. (2020) T. T. Nguyen, N. D. Nguyen, and S. Nahavandi Deep reinforcement learning for multiagent systems: a review of challenges, solutions, and applications. IEEE Transactions on Cybernetics 50 (9), p. 3826–3839. External Links: Document Cited by: §1. Paine et al. (2020) T. L. Paine, C. Gulcehre, B. Shahriari, M. Denil, M. Hoffman, H. Soyer, R. Tanburn, S. Kapturowski, N. Rabinowitz, D. Williams, et al. Making efficient use of demonstrations to solve hard exploration problems. In International Conference on Learning Representations, External Links: Document Cited by: §3.4.2. Pang et al. (2022) Q. Pang, Y. Yuan, and S. Wang Mdpfuzz: testing models solving markov decision processes. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, p. 378–390. External Links: Document Cited by: §1, §1, §1, §3.3, 2nd item, §7, §7, §8. Puterman (1990) M. L. Puterman Markov decision processes. Handbooks in Operations Research and Management Science 2, p. 331–434. External Links: Document Cited by: §2.1, §3.1. Rigter et al. (2023) M. Rigter, B. Lacerda, and N. Hawes One risk to rule them all: a risk-sensitive perspective on model-based offline reinforcement learning. Advances in Neural Information Processing Systems 36, p. 77520–77545. External Links: Document Cited by: §6.2. Sallab et al. (2017) A. E. Sallab, M. Abdou, E. Perot, and S. Yogamani Deep reinforcement learning framework for autonomous driving. In IS&T International Symposium on Electronic Imaging: Autonomous Vehicles and Machines, p. 70–76. External Links: Document Cited by: §1. Seno and Imai (2022) T. Seno and M. Imai D3rlpy: an offline deep reinforcement learning library. Journal of Machine Learning Research 23 (315), p. 1–20. External Links: Document Cited by: §3.4.2, §4. Shalev-Shwartz et al. (2016) S. Shalev-Shwartz, S. Shammah, and A. Shashua Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295. External Links: Document Cited by: §1. Shi et al. (2025a) J. Shi, J. He, Z. Yang, Đ. Žikelić, and D. Lo Synthesizing efficient and permissive programmatic runtime shields for neural policies. ACM Transactions on Software Engineering and Methodology. External Links: Document Cited by: §8. Shi et al. (2025b) J. Shi, Z. Yang, J. He, B. Xu, D. Kim, D. Han, and D. Lo Finding safety violations of ai-enabled control systems through the lens of synthesized proxy programs. ACM Transactions on Software Engineering and Methodology. External Links: Document Cited by: §1, §8. Shi et al. (2021) T. Shi, D. Chen, K. Chen, and Z. Li Offline reinforcement learning for autonomous driving with safety and exploration enhancement. arXiv preprint arXiv:2110.07067. External Links: Document Cited by: §2.2, §3.4.1. Sinha et al. (2022) S. Sinha, A. Mandlekar, and A. Garg S4rl: surprisingly simple self-supervision for offline reinforcement learning in robotics. In Conference on Robot Learning, p. 907–917. External Links: Document Cited by: §2.2. Sonabend-W et al. (2020) A. Sonabend-W, J. Lu, L. A. Celi, T. Cai, and P. Szolovits Expert-supervised reinforcement learning for offline policy learning and evaluation. Advances in Neural Information Processing Systems 33, p. 18967–18977. External Links: Document Cited by: §2.2. Suttle et al. (2025) W. A. Suttle, A. Suresh, and C. Nieto-Granda Behavioral entropy-guided dataset generation for offline reinforcement learning. arXiv preprint arXiv:2502.04141. External Links: Document Cited by: §3.4.2. Sutton and Barto (1998) R. S. Sutton and A. G. Barto Reinforcement learning: an introduction. MIT Press, Cambridge, MA. Cited by: §2.1, §3.4.2. Talpaert et al. (2019) V. Talpaert, I. Sobh, B. R. Kiran, P. Mannion, S. Yogamani, A. El-Sallab, and P. Perez Exploring applications of deep reinforcement learning for real-world autonomous driving systems. In International Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP), p. 564–572. External Links: Document Cited by: §1. Tambon et al. (2023) F. Tambon, V. Majdinasab, A. Nikanjam, F. Khomh, and G. Antoniol Mutation testing of deep reinforcement learning based on real faults. In 2023 IEEE Conference on Software Testing, Verification and Validation (ICST), p. 188–198. External Links: Document Cited by: §8. Tang et al. (2022) S. Tang, M. Makar, M. Sjoding, F. Doshi-Velez, and J. Wiens Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. Advances in Neural Information Processing Systems 35, p. 34272–34286. External Links: Document Cited by: §2.2. Tang and Wiens (2021) S. Tang and J. Wiens Model selection for offline reinforcement learning: practical considerations for healthcare settings. In Machine Learning for Healthcare Conference, p. 2–35. External Links: Document Cited by: §2.2. Tappler et al. (2022) M. Tappler, F. C. Córdoba, B. K. Aichernig, and B. Könighofer Search-based testing of reinforcement learning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, L. D. Raedt (Ed.), p. 503–510. External Links: Link, Document Cited by: §8. Tappler et al. (2024) M. Tappler, A. Pferscher, B. K. Aichernig, and B. Könighofer Learning and repair of deep reinforcement learning policies from fuzz-testing data. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, p. 1–13. External Links: Document Cited by: §8. Thomas et al. (2025) D. Thomas, M. Biagiola, N. Humbatova, M. Wardat, G. Jahangirova, H. Rajan, and P. Tonella MuPRL: a mutation testing pipeline for deep reinforcement learning based on real faults. In IEEE/ACM International Conference on Software Engineering, External Links: Document Cited by: §8. Torabi et al. (2018) F. Torabi, G. Warnell, and P. Stone Behavioral cloning from observation. In International Joint Conference on Artificial Intelligence, p. 4950–4957. External Links: Document Cited by: §1, 1st item. van Hasselt et al. (2016) H. van Hasselt, A. Guez, and D. Silver Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, D. Schuurmans and M. P. Wellman (Eds.), p. 2094–2100. External Links: Link, Document Cited by: §1, §4. Wackerly (2008) D. D. Wackerly Mathematical statistics with applications. Thomson Brooks/Cole. Cited by: §3.4.3. Xie et al. (2021) T. Xie, N. Jiang, H. Wang, C. Xiong, and Y. Bai Policy finetuning: bridging sample-efficient offline and online reinforcement learning. Advances in Neural Information Processing Systems 34, p. 27395–27407. External Links: Document Cited by: §5. Yahmed et al. (2023) A. H. Yahmed, R. Bouchoucha, H. B. Braiek, and F. Khomh An intentional forgetting-driven self-healing method for deep reinforcement learning systems. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), p. 1314–1325. External Links: Document Cited by: §4. Yang et al. (2023) L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui, and M. Yang Diffusion models: a comprehensive survey of methods and applications. ACM Computing Surveys 56 (4), p. 1–39. External Links: Document Cited by: §8. Yang et al. (2022) Z. Yang, J. Shi, J. He, and D. Lo Natural attack for pre-trained models of code. In Proceedings of the 44th International Conference on Software Engineering, p. 1482–1493. External Links: Document Cited by: §1. Yu et al. (2021) C. Yu, J. Liu, S. Nemati, and G. Yin Reinforcement learning in healthcare: a survey. ACM Computing Surveys (CSUR) 55 (1), p. 1–36. External Links: Document Cited by: §1. Zheng et al. (2019) Y. Zheng, X. Xie, T. Su, L. Ma, J. Hao, Z. Meng, Y. Liu, R. Shen, Y. Chen, and C. Fan Wuji: automatic online combat game testing using evolutionary deep reinforcement learning. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), p. 772–784. External Links: Document Cited by: §5, §6.3. Zhou et al. (2024) X. Zhou, B. Xu, K. Kim, D. Han, H. H. Nguyen, T. Le-Cong, J. He, B. Le, and D. Lo Leveraging large language model for automatic patch correctness assessment. IEEE Transactions on Software Engineering 50 (11), p. 2865–2883. External Links: Document Cited by: §1. Zolfagharian et al. (2023) A. Zolfagharian, M. Abdellatif, L. C. Briand, M. Bagherzadeh, and S. Ramesh A search-based testing approach for deep reinforcement learning agents. IEEE Transactions on Software Engineering. External Links: Document Cited by: §4, §6.2, §8.