Paper deep dive
Resilience Matters for Embodied Agents System: New Metrics, Systematic Evaluation, and Optimization
Yapeng Liu, Yuanzhao Zhai, Xudong Gong, Dawei Feng, Bo Ding, Lin Wang, Huaimin Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/26/2026, 4:46:55 AM
Summary
This paper introduces a novel resilience evaluation framework for Embodied Agents Systems (EAS), addressing the limitations of outcome-centric metrics like success rate. The authors define three core resilience metrics: Rebound (recovery cost), Stability (policy stability under perturbations), and Graceful Extensibility (stress response mapping and capacity). They implement a non-intrusive evaluation layer applied to 10 EAS methods across 400 household tasks in Habitat-Sim. Results show that resilience metrics reveal process-level distinctions hidden by traditional metrics, such as significant differences in recovery costs among agents with similar success rates. The framework also enables metrics-guided optimization, demonstrating that resilience evaluation can diagnose issues and improve agent performance.
Entities (10)
Relation Signals (9)
Resilience → hasaspect → Rebound
confidence 95% · we define the first comprehensive resilience metrics suite for EASs system that exposes Rebound, Stability, and Graceful Extensibility
Resilience → hasaspect → Stability
confidence 95% · we define the first comprehensive resilience metrics suite for EASs system that exposes Rebound, Stability, and Graceful Extensibility
Resilience → hasaspect → Graceful Extensibility
confidence 95% · we define the first comprehensive resilience metrics suite for EASs system that exposes Rebound, Stability, and Graceful Extensibility
Rebound → measuredby → Recovery Cost
confidence 94% · Rebound measures how much extra work the agent must expend... We use the Recovery Cost as the primary indicator
Stability → measuredby → Policy Stability
confidence 94% · We take Policy Stability(β) as the primary indicator, because it directly measures whether small changes in task instruction or execution condition amplify into process fluctuations.
Graceful Extensibility → measuredby → Stress Response Mapping
confidence 94% · We therefore model Graceful Extensibility as a stress-response mapping λ↦Mf(λ)
Resilience Evaluation Framework → evaluatedon → Habitat-Sim
confidence 93% · We conduct experiments on Habitat-Sim with 400 household tasks and 10 representative EAS methods.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Embodied Agents System (EAS) are increasingly deployed in open-world physical domains, where reliability directly dictates deployment quality and human-agent trust. However, existing evaluations rely on outcome-centric metrics as success rate or safety scores that collapse diverse execution trajectories into coarse scores, obscuring the dynamic processes underlying agent behavior. Therefore, they ignore a critical property of EAS -- which we define as the Resilience -- that reflects how EASs recover, stabilize, and extend under perturbations and across iterative updates. The lack of resilience is particularly critical in open-world environments due to continuous unexpected disruptions, thus directly affecting the quality of EAS deployment. To address this problem, we gain insight from the resilience-engineering concepts to EAS groundings and propose a novel resilience evaluation framework that can be flexibly applied to any EAS. Specifically, we define the first comprehensive resilience metrics suite for EASs system that exposes Rebound, Stability, and Graceful Extensibility across embodied tasks execution, providing a practical grounding for EAS resilience analysis. We further implement the resilience evaluation layer that transforms execution process into assessments for diagnosis and optimization. Across 400 household tasks with 10 EAS, we reveal the process-level distinction hidden by outcome metrics, including recovery cost differences among successful episodes ($\Delta C_{rec}=25.2$), increased instability and task-family degradation. Metrics-guided optimizations reduce recovery cost and increase stability, graceful extensibility completion, showing the diagnostic effect of resilience evaluation. Our results reveal a trade-off among resilience characteristics, suggesting that a resilient EAS construction should be configured according to deployment-specific requirements.
Tags
Links
- Source: https://arxiv.org/abs/2608.23839v1
- Canonical: https://arxiv.org/abs/2608.23839v1
Trouble viewing inline? Open PDF directly →
Full Text
49,634 characters extracted from source content.
Expand or collapse full text
Resilience Matters for Embodied Agents System: New Metrics, Systematic Evaluation, and Optimization Yapeng Liu Yuanzhao Zhai Xudong Gong Affiliation: PDL Lab, College of Computer Science and Technology, National University of Defense Technology Affiliation: State Key Laboratory of Complex & Critical Software Environment, Changsha 410073, Hunan, China Affiliation: EmPACT Lab, Nanyang Technological University, Singapore Code: Eas-Resilience-Evaluation | Website: Eas-Resilience-Evaluation Dawei Feng Bo Ding Lin Wang Huaimin Wang Affiliation: PDL Lab, College of Computer Science and Technology, National University of Defense Technology Affiliation: State Key Laboratory of Complex & Critical Software Environment, Changsha 410073, Hunan, China Affiliation: EmPACT Lab, Nanyang Technological University, Singapore Code: Eas-Resilience-Evaluation | Website: Eas-Resilience-Evaluation Abstract Embodied Agents System (EAS) are increasingly deployed in open-world physical domains, where reliability directly dictates deployment quality and human-agent trust. However, existing evaluations rely on outcome-centric metrics as success rate or safety scores that collapse diverse execution trajectories into coarse scores, obscuring the dynamic processes underlying agent behavior. Therefore, they ignore a critical property of EAS – which we define as the Resilience – that reflects how EASs recover, stabilize, and extend under perturbations and across iterative updates. The lack of resilience is particularly critical in open-world environments due to continuous unexpected disruptions, thus directly affecting the quality of EAS deployment. To address this problem, we gain insight from the resilience-engineering concepts to EAS groundings and propose a novel resilience evaluation framework that can be flexibly applied to any EAS. Specifically, we define the first comprehensive resilience metrics suite for EASs system that exposes Rebound, Stability, and Graceful Extensibility across embodied tasks execution, providing a practical grounding for EAS resilience analysis. We further implement the resilience evaluation layer that transforms execution process into assessments for diagnosis and optimization. Across 400 household tasks with 10 EAS, we reveal the process-level distinction hidden by outcome metrics, including recovery cost differences among successful episodes (ΔCrec=25.2 C_rec=25.2), increased instability and task-family degradation. Metrics-guided optimizations reduce recovery cost and increase stability, graceful extensibility completion, showing the diagnostic effect of resilience evaluation. Our results reveal a trade-off among resilience characteristics, suggesting that a resilient EAS construction should be configured according to deployment-specific requirements. 1 Introduction Embodied Agents System (EAS) shows great potential in domains where physical interaction and long-term task completion are essential, especially for the collaboration domain [18, 4]. Recent advances in large language models (LLMs) further strengthen EAS capabilities of reasoning and interaction, along with increased complexity in reliability evaluation [32, 30, chang2024partnr]. As illustrated in Fig. 1, two EASs receive the same household instruction and achieve the same outcome, but their thought and execution process can be fundamentally different. This difference matters because EAS operates in dynamic environments where disruptions have continuously emerged [19, 8], indicating that reproducible reliability is critical for physical tasks and collaboration trust during execution [9, 36]. However, current EAS evaluation largely remains outcome or safety metrics, such as success rate or aggregated safety triggers [chang2024partnr, 35, 16, 30]. They collapse different execution processes into the same score, making brittle success indistinguishable from trustworthy adaptation success [22, 23]. This limitation points to a missing property of EAS – which we define as the Resilience in EAS – that reflects how EASs recover, stabilize, and extend under stress. Figure 1: Outcome hide process resilience difference in EAS. (A) A resilient success trajectory handles inner and outer stress through resilient mechanisms, (B) A costly success relies on repeated backtrack, replan and unsafe behaviors. To address this problem, we gain insights from the resilience-engineering concepts to EAS groundings. System resilience evaluation captures the intrinsic property of a system, which maintains its acceptable execution and effective recovery when unexpected disturbances happen [11, 3]. Woods further decomposes system resilience into four capabilities, including rebound, robustness, graceful extensibility under stress, and sustained adaptability [33]. Specifically, we define the first comprehensive Resilience Metrics for EAS that expose Rebound, Stability, and Graceful Extensibility throughout execution: recovery cost CrecC_rec, stability sensitivity β, stress response mapping Mf(λ)M_f(λ) and stress capacityλf∗ _f^*. We further implement the Resilience Evaluation Layer that extracts signals from trajectories, monitors, and LLM judge, aggregates them through stage baselines and controlled stress intensities. This layer non-intrusively enables comparison, diagnosis and optimization guidance for EAS resilience evaluation. We conduct experiments on Habitat-Sim with 400 household tasks and 10 representative EAS methods. Focus on EAS resilience Metrics Validity, Benchmark and Practical Optimization, results show that resilience evaluation reveals process-level disparities, benchmarking EAS methods and has practical diagnosis effectiveness. Among success episodes, physical perturbations increase recovery cost from 11.611.6 to 36.336.3, semantic perturbations raise instability to Δβ=0.32 β=0.32, and stress tests expose degradation, recovery cost and perturbations intensities are strongly correlated (Spearman ρ=0.82ρ=0.82). For methods, Reflexion and CoPAL achieve comparable SR (56.3% vs. 55.8%) but differ by 4.8×4.8× in recovery cost, showing that EASs have distinct resilience signatures. There is usually a trade-off in resilience aspects for EAS on our bench. Metrics-guided optimization further improve resilience performance, demonstrating that our evaluation layer is both diagnostic and actionable. In summary, our work makes three major contributions. 1. Embodied Resilience Metrics and Evaluation Layer: We formalize resilience metrics as three aspects: Rebound, Stability, and Graceful Extensibility. Then we complement EAS evaluation with the resilience evaluation layer, a practical grounding for measuring EAS resilience property quantification. 2. Evidence and benchmarking: We perform a experimental study on 10 representative EAS methods, demonstrating that our metrics successfully distinguish between brittle and resilient behaviors (Sec 3.2.2), we benchmark them, and show their resilience signatures under our resilience evaluation (Sec 3.2.3). 3. Resilience evaluation provides practical optimization: We provide quantitative evidence linking specific diagnosis to resilience improvements (Sec 4), showing that resilience-guided diagnosis can support practical optimization, improving EAS resilience while holding the resilience trade-off design principle. 2 Related Works LLM-Enhanced EAS Evaluation: From Outcome to Resilience Metrics. LLM-enhanced EAS close the perception-execution loop by taking language-based reasoning into embodied actions generation, and they evolved from instruction grounding to feedback reasoning and self-correction [2, 17, 13, 28, 29, 10]. Recent works [14, 20, 38, 31] further explore adaptation, consistency, and open-ended skill acquisition to build trustworthy EAS. However, classical EAS evaluation remains largely outcome-centric as PARTNR emphasizing success metrics [36, 39, 5, 24]. Under embodied scenarios – physical irreversibility, partial observability, and dynamic constraints, such outcomes cannot distinguish robust adaptation from brittle success [18, chang2024partnr, 30]. Focused on EAS reliability evaluation, recent works construct perturbation test to measure safety-aware planning [35, 34, 6]. They provide reliability diagnostics, but mainly whether an agent violates constraints or approaches a hazardous state in perturbation tests rather than normal process-level execution [16, 40]. We present the resilience metrics to evaluate EAS Resilience as an intrinsic property of EAS, measuring how the system behaves from inner stress or perturbations. In this way, we complement EAS evaluation with our resilience evaluation layer to measure how the resilience of EAS exhibits and provide practical optimization guidance. Resilience Concepts Practical Evaluation in EAS Execution. In systems engineering, resilience emphasizes the capability to endure, adapt to, and recover from disruptions [25, 7]. Woods organizes the technical expression of resilience into four concepts: rebound, robustness, graceful extensibility and sustained adaptability [33]. This perspective holds that handling open-world complexities relies not only on failure prevention, but on continuous, dynamic adaptation during execution [12]. Classical resilience evaluation works evaluate loss severity and recovery latency, but their reliance on exact state matching or crash detection proves insufficient for embodied scenarios [15, 27, 24]. Recently, resilience-relevant evaluating methods for EAS emerged, they evaluate on semantic safety under hazardous instructions and physical cases [41, 37, 19]. However, these methods mix EAS resilience with adversarial security or perturbation stability. They evaluate the failure under static perturbation, but rarely capture the process-aware dynamics like recovery or extensibility. Grounding with resilient EAS construction, we implement the resilience concepts into three resilient aspects. Rebound captures the burden of closed-loop recovery; Stability captures robustness as decision consistency; and Graceful Extensibility captures bounded degradation and remaining operational margin under stress. Sustained adaptability is treated as a longitudinal extension because it is less directly observable within dynamic EAS execution. Our resilient evaluation realize the practical evaluation layer, providing the validated instrument that reveals the EAS resilience property. 3 The Proposed EAS Resilience Evaluation Framework Overview. We formalize the Resilience Metrics and Resilience Evaluation Layer for EAS, which goal is to evaluate system resilience from execution dynamics. Given an episode artifact (τ,ℒ,,)(τ,L,C,J), where τ=(ot,at,rt)t=1Tτ=\(o_t,a_t,r_t)\_t=1^T is the execution trajectory, ℒL is the runtime log, C is the monitor, and J is the LLM Judge signal, the evaluation layer maps execution dynamics to three resilience aspects: ℳ:(τ,ℒ,,)↦(ℳrebound,ℳstability,ℳGE).M:(τ,L,C,J) (M_rebound,M_stability,M_GE). For episode i, let t denote a runtime step. At each step, we record task progress pt∈[0,1]p_t∈[0,1], proposition completion qt∈[0,1]q_t∈[0,1], and execution mode mtm_t. We define a local execution anchor as at:=(f,btp,btq,mt)a_t:=(f,b_t^p,b_t^q,m_t), where f is the task family, and btqb_t^q is the residual proposition load. For each anchor, we estimate a local Stage Baseline from clean or reference episodes under the same ata_t: (at):=(τ∗(at),Wrem∗(at),T¯cog(at),N¯phy(at),Δp¯(at),Δq¯(at),r¯(at)).N(a_t):= (τ^*(a_t),W^*_rem(a_t), T^cog(a_t), N^phy(a_t), p(a_t), q(a_t), r(a_t) ). Here, τ∗(at)τ^*(a_t) is the nominal cycle time, Wrem∗(at)W^*_rem(a_t) is the estimated remaining work from the anchor, T¯cog(at) T^cog(a_t) and N¯phy(at) N^phy(a_t) summarize cognitive and physical effort, (Δp¯(at),Δq¯(at))( p(a_t), q(a_t)) summarize task progress and proposition completion, r¯(at) r(a_t) summarizes nominal risk load. The Stage Baseline provides a common local reference for resilience metrics computation and the reliability of statistical generalization on resilience metrics. In parallel, we take LLM-as-a-Judge to densify sparse execution feedback. The LLM judges our process from instruction, state, trajectory, and env feedback, and then produces structured reward for goal progress, rational action, and efficiency; implementation details are available through our Code Repository and Project Website. Based on this shared evidence substrate, we instantiate system resilience from three complementary aspects: Rebound, Stability, and Graceful Extensibility as Table 1. Table 1: Overview of resilience metrics with aspect, type (D: deterministic outcome, S: statistical, A: analysis, T: time-series), and data source Sym. Metric Name Calculation Aspect Type Data Source CrecC_rec Recovery Cost Crec=1Wrem∗(atd)∑tdtrΔτtτ∗(at)⋅gtrecC_rec= 1W_rem^* (a_t_d ) _t_d^t_r _tτ^* (a_t )· g_t^rec Rebound D, S System Logs β Policy Stability β=supz,ρ|ℓ(π,ρ(z))−ℓ(π,z)|‖ρ‖β= _z,ρ | (π,ρ(z))- (π,z) |\|ρ\| Stability S Sensitivity Test Mf(λ)M_f(λ) Stress Curve Mapping Mf(λ)=1K∑j=1Kr~(j),f(λ)−1M_f(λ)= 1K _j=1^K r_(j),f(λ)-1 Extensibility D, S Stress Test λf∗ _f^* Stress Capacity λf∗=supλ∈[0,1]∣Mf(λ)≥0 _f^*= \λ∈[0,1] M_f(λ)≥ 0\ Extensibility S, A Stress Grid 3.1 Metrics Design 1) Rebound Metrics: Recovery Cost to Availability. Rebound measures how much extra work the agent must expend to return the acceptable execution after disruption. We use the Recovery Cost as the primary indicator, revealing the efficient rebound and brute-force rebound. Detailed implementation is available through our Code Repository and Project Website. For a rebound window [td,tr][t_d,t_r], tdt_d is the first step at which execution departs from the nominal continuation under the local anchor ata_t, and trt_r is the first step at which the task re-enters the acceptable execution. Execution StageBaseline (at)N(a_t) (execution local reference, refer to Sec 3.2) provides the nominal local reference, including the remaining nominal work Wrem∗(at)W^*_rem(a_t), and the expected local execution statistics. We decompose recovery cost into three aligned parts: Cognitive recovery, Physical recovery, and State debt in Eq. 1. We aggregate their excess variables as tcog=(Ttcog−T¯cog(at))+T¯cog(at)+ϵ,tphy=(Ntphy−N¯phy(at))+N¯phy(at)+ϵ,tdeb=(rt−r¯(at))+r¯(at)+ϵ.z_t^cog= (T_t^cog- T^cog(a_t))_+ T^cog(a_t)+ε, _t^phy= (N_t^phy- N^phy(a_t))_+ N^phy(a_t)+ε, _t^deb= (r_t- r(a_t))_+ r(a_t)+ε. (1) where tcogz_t^cog measures excess planning, perception effort; tphyz_t^phy measures excess navigation, retry effort; and tdebz_t^deb measures lagged progress leakage, risk load. (⋅)+(·)_+ indicates the ReLU function, we record the excess costs while discarding pre-complete [1]. Implementation details and illustrative calculations are available through our Code Repository and Project Website. To make the three channels comparable under the same anchor-conditioned regime, we convert each local excess vector into a covariance-adjusted deviation as Mahalanobis Distance[21]: gtu=tu⊤(Σu(at)+ηI)−1tu,u∈cog,phy,deb,g_t^u= z_t^u ( _u(a_t)+η I )^-1z_t^u, u∈\cog,phy,deb\, (2) where Σu(at) _u(a_t) is the covariance estimated from standard stage baselines, ηIη I is the regularizer. This step removes scale inconsistency and accounts for correlation within each recovery channel. We then aggregate the three channel deviations into recovery intensity gtrec=1U∑uU(gtu)2g_t^rec= 1U _u^U(g_t^u)^2. Finally, for episode i, the Rebound Cost is defined as Crec(i)=1Wrem∗(atd)+ϵ∑t=tdtrΔτtτ∗(at)+ϵ⋅gtrec.C_rec^(i)= 1W^*_rem(a_t_d)+ε _t=t_d^t_r _tτ^*(a_t)+ε· g_t^rec. (3) This quantity measures the total extra work required to restore acceptable execution after disruption. Normalization by Wrem∗(atd)W^*_rem(a_t_d) ensures that recovery is judged against the remaining nominal work at the disturbance point. We aggregate rebound signatures using the statistical procedures documented in our Code Repository and Project Website. 2) Stability Metrics: Execution Fluctuations. Stability quantifies the system’s ability to remain consistent under stresses, whether arising from stochasticity (Execution), perturbations (Sensitivity), or self-evolution. It is reflected in execution as frequent replanning, unstable progress, and abrupt shifts in the progress. We take Policy Stability(β) as the primary indicator, because it directly measures whether small changes in task instruction or execution condition amplify into process fluctuations. We describe how β is computed from execution representations; implementation details, including the value-network design, are available through our Code Repository and Project Website. We use the value function V(⋅)V(·) for each EAS execution representation, which is used to diagnose execution progress changes trend. Given the encoded state sts_t, the value function is V(st)=π[∑u=tTγu−tr~u∣st],V(s_t)=E_π\! [ _u=t^Tγ^u-t r_u s_t ], (4) where π is the agent policy and r~u r_u is the shaped reward by LLM judge and Wrem∗W^*_rem. For each state transition, we compute the one-step temporal-difference residual δt=r~t+γ(1−dt)V(st+1)−V(st) _t= r_t+γ(1-d_t)V(s_t+1)-V(s_t), where dtd_t is the termination signal. We further compute the generalized advantage estimate AtA_t as an exponentially decayed accumulation of TD residuals: At=∑ℓ=0T−t(γλ)ℓδt+ℓA_t= _ =0^T-t(γλ) _t+ . Then our resilience loss for shape reward update is ℓ(π,zi)=1Ti∑t=1Tit⊤(Σ(at)+ηI)−1t, (π,z_i)= 1T_i _t=1^T_i x_t ( (a_t)+η I )^-1x_t, (5) where ziz_i denotes an episode, Σ(at) (a_t) is the standard stage baselines covariance, and ηIη I is regularizer. This loss increases when the trajectory tx_t contains large value variance, TD spikes, and advantages. We define β-Stability as this perturbations sensitivity to execution progress: βf=supz∈f,ρ∈f|ℓ(π,ρ(z))−ℓ(π,z)|‖ρ‖+ϵ. _f= _z _f,ρ _f | (π,ρ(z))- (π,z) |\|ρ\|+ε. (6) Here, fZ_f is the episode set from task family f, fP_f is the controlled perturbations set. A low βf _f means that similar conditions lead to similar results, while high βf _f means more replanning, value fluctuation, and progress instability. 3) Graceful Extensibility Metrics: Stress Response. Graceful Extensibility measures how an EAS performance extends as stress increases. It ensures that when stressors exceed the system’s adaptive capacity, performance declines gracefully and predictably rather than collapsing catastrophically. This is a dynamic stress-response, revealing the resilience property of EAS. We therefore model Graceful Extensibility as a stress-response mapping λ↦Mf(λ)λ M_f(λ), where λ is the stress severity and Mf(λ)M_f(λ) is the operational margin. Let λT_λ denote a stress operator with severity λ∈[0,1]λ∈[0,1], which is related to perturbations and EAS self-evolvement. In our resilience evaluation, we unify external perturbation and internal evolution through the same stress-response formulation, as documented in our Code Repository and Project Website. For an episode i from task family f under stress λT_λ, we first define a hard boundary monitor: mi(λ)=minp,i(λ)τp,q,i(λ)τq,τrecCrec,i(λ)+ϵ,τstabβstab,i(λ)+ϵ,τsafeUi(λ)+ϵ−1.m_i(λ)= \ p_T,i(λ) _p, q_T,i(λ) _q, _recC_rec,i(λ)+ε, _stab _stab,i(λ)+ε, _safeU_i(λ)+ε \-1. (7) where UiU_i is the constraints violation load. The thresholds define the minimal acceptable task contract, mi(λ)<0m_i(λ)<0 means that the contract is violated. After accumulation of stage baselines from the task family f, we aggregate the statistical extensibility of the operation. For compact notation, let Qαf[X](λ)≜Qα(X(λ)∣f)Q_α^f[X](λ) Q_α\! (X(λ) f ), where Qαf[X](λ)Q_α^f[X](λ) denotes the α-quantile of variable X over episodes from family f under stress λ. We construct the statistical extensibility vector f(λ)=[Qαf[pT](λ)τp,Qαf[qT](λ)τq,τrecQ1−αf[Crec](λ)+ϵ,τstabQ1−αf[Sstab](λ)+ϵ,τsafeQ1−αf[U](λ)+ϵ]⊤. _f(λ)= [ Q_α^f[p_T](λ) _p,\, Q_α^f[q_T](λ) _q,\, _recQ_1-α^f[C_rec](λ)+ε,\, _stabQ_1-α^f[S_stab](λ)+ε,\, _safeQ_1-α^f[U](λ)+ε ] . (8) We take lower and upper quantiles for benefit variables and cost variables, which makes our metrics sensitive to graceful extensibility. Let r(1),f(λ)≤⋯≤r(Kmax),f(λ)r_(1),f(λ)≤·s≤ r_(K_ max),f(λ) be the sorted entries of f(λ)r_f(λ). The statistical operational margin is defined as Mf(λ)=1K∑j=1Kr(j),f(λ)−1,1<K≤Kmax.M_f(λ)= 1K _j=1^Kr_(j),f(λ)-1, 1<K≤ K_ max. (9) This bottom-K aggregation preserves the semantics of extensibility, but avoids the rigid behavior of a minimum optimization. For practical diagnosis, dynamic mapping between Mf(λ)M_f(λ) extensibility and execution performance is an important indicator to analyze EAS resilience. Further, the quantitative Graceful Extensibility indicator is the largest stress level where the EAS remains acceptable execution: λf∗=supλ∈[0,1]∣Mf(λ)≥0. _f^*= \λ∈[0,1] M_f(λ)≥ 0 \. (10) We call λf∗ _f^* the stress capacity. It gives the operational boundary of task family f: the maximum stress severity at which the system can still preserve its positive task progress in task family f. 3.2 Resilience Evaluation Benchmark The Resilience Evaluation Layer Construction We construct the Resilience Evaluation Layer as a non-intrusive evaluation instrument that complements the EAS reliability evaluation process. We present our resilience evaluation pipeline in Alg 1, it attaches probes during the EAS execution, then computes collected signals to resilience metrics. This design allows resilience to be evaluated as an execution property rather than an additional task objective. Algorithm 1 Resilience Metrics Evaluation Pipeline 0: Task family set ℱF; episode artifacts (τi,ℒi,i,i)i=1N\( _i,L_i,C_i,J_i)\_i=1^N; StageBaseline estimation; perturbation set fP_f; stress grid Λ=λjj=1m =\ _j\_j=1^m; fixed thresholds and windows. 0: ℳreboundM_rebound, ℳstabilityM_stability, ℳGEM_GE 1: Estimate Stage Baselines (a)N(a) from executions. 2: for each task family f∈ℱf do 3: for each stress level λ∈Λλ∈ do 4: for each episode i, artifact (τi,ℒi,i,i)( _i,L_i,C_i,J_i) do 5: Extract stepwise signals: progress pt,qtp_t,q_t, execution mode mtm_t, execution ℒiL_i, critics i,iC_i,J_i. 6: Assign local execution anchors at=(f,btp,btq,mt)a_t=(f,b_t^p,b_t^q,m_t). 7: if Detect Rebound window [td,tr][t_d,t_r] then 8: Compute tcogz_t^cog, tphyz_t^phy and tdebz_t^deb. 9: Compute rebound cost Crec(i)C_rec^(i) from gtrecg_t^rec. 10: Return to acceptable (at)N(a_t). 11: end if 12: Collect Stability values V(st)V(s_t), TD residuals δt _t, and GAE advantages AtA_t. 13: Aggregate to vector tx_t and βi _i. 14: Backpropagation Critic ℓstab(π,zi) _stab(π,z_i) and Stage Baselines f(a)N_f(a). 15: end for 16: Retrieve stressed executions under λT_λ. 17: Compute hard boundary monitors mi(λ)m_i(λ) in family f. 18: Aggregate episode statistics f(λ)r_f(λ). 19: Compute stress mapping: Mf(λ)←1K∑j=1Kr(j),f(λ)−1M_f(λ)← 1K _j=1^Kr_(j),f(λ)-1. 20: end for 21: Compute Graceful Extensibility stress capacity: λf∗←supλ∈Λ∣Mf(λ)≥0 _f^*← \λ∈ M_f(λ)≥ 0\. 22: Update Stage Baseline N and Statistical Aggregation Metrics 23: end for 24: return CrecC_rec, β, MfM_f, λf∗ _f^*, trajectory. 3.2.1 Resilience EAS Testbed Setup We conduct experiments on Habitat-Sim 3.0 [26] with PARTNR-style [chang2024partnr] collaborative household tasks, including navigation, object manipulation, and object transportation scenarios. To expose different EAS resilience, we introduce controlled perturbations over physical states, semantic instructions, object states, and stress severity. The full perturbation specification, evaluated EAS methods, and dataset setting are documented in our Code Repository and Project Website. 3.2.2 Resilience Metrics Validation: Execution Differences Hidden by Outcomes We first validate whether resilience metrics (ℳrebound,ℳstability,ℳGE)(M_rebound,M_stability,M_GE) capture distinct resilience property hidden by outcome-centric evaluation, and verify them as the measurement instruments. Figure 2: Different CrecC_rec under outcomes. Resilience metrics disperse differences in same outcome. Resilience metrics reveal significant performance differences that both SR and safety scores fail to capture, demonstrating discrimination capability for EAS with similar success rates and safety scores. We take CrecC_rec as an example, Fig. 2 illustrates "the same outcome, different resilience" effect. Resilience Metrics Construct Validity and Sensitivity. Fig. 3 further confirms the construct validity of the three metric families. Rebound CrecC_rec separates executions by recovery burden: perturbations increase recovery cost from 11.611.6 in clean episodes to 36.336.3 in average, with a significant distributional shift (Mann–Whitney p<0.001p<0.001). Stability β captures sensitivity to semantic-preserving instruction changes: semantic perturbations inflate instability to β=0.32β=0.32, mainly driven by divergent action generation instability (βout=0.42 _out=0.42). Graceful Extensibility characterizes degradation as a stress-response curve rather than a binary failure event: as stress severity increases over λ∈[0.2,1.0]λ∈[0.2,1.0], the relative margin ΔM M decreases while recovery cost rises, with strong correlation (Spearman ρ=0.82ρ=0.82). These results show that the metrics measure complementary resilience properties and are not reducible to SR, safety, or completion alone. Figure 3: Resilience metrics construct validity results (zoom in for better view). In addition, our Code Repository and Project Website provide analyses of the statistical reliability and sensitivity of the resilience metrics. These analyses indicate that resilience metrics are related to EAS inherent property, which maintains the consistency across embodied tasks execution. Figure 4: Resilience benchmark landscape. Also, a scalability sensitivity analysis across Qwen3-8B-Instruct, Qwen2.5-7B-Instruct, and Llama3.1-8B-Instruct shows that stronger backbones improve baseline task performance while preserving similar resilience profile patterns, suggesting that resilience is a system-level property rather than only a backbone-size effect (see our Code Repository and Project Website). 3.2.3 Benchmarking EAS Resilience Results We apply the validated resilience metrics to benchmark representative EAS methods under identical environments and perturbation settings. Table 2 reports the resilience benchmark scores, and Fig. 4 summarizes the normalized resilience profiles. For example, Reflexion and CoPAL achieve comparable SR (56.3% vs. 55.8%), but Reflexion incurs 4.8×4.8× higher recovery cost than CoPAL (57.357.3 vs. 11.911.9). This indicates that EAS take different recovery processes, and resilience evaluation reveals these features. Table 2: Comparative EAS resilience benchmark results in resilience aspects. Method Resilience Metrics Outcome Ref. Sim. CrecC_rec (↓ ) β (↓ ) Stress Cap. λ∗λ^* (↑ ) SR Comp. Steps ReAct 6.5 ± 0.46 0.239 ± 0.011 0.50 ± 0.042 38.5% ± 3.1% 57.8% ± 4.6% 3374 ± 285 Reflexion 57.3 ± 5.07 0.299 ± 0.021 0.66 ± 0.035 56.3% ± 4.2% 75.0% ± 3.8% 4390 ± 312 CycleVLA 35.5 ± 2.91 0.400 ± 0.026 0.72 ± 0.047 22.2% ± 2.8% 47.5% ± 4.1% 4688 ± 420 CLARE 21.6 ± 1.68 0.344 ± 0.016 0.72 ± 0.052 28.4% ± 3.5% 46.0% ± 4.5% 4249 ± 360 SayCan 38.5 ± 2.49 0.283 ± 0.013 0.78 ± 0.046 57.0% ± 4.1% 72.6% ± 3.7% 4682 ± 380 AgentEvolver 19.2 ± 1.24 0.264 ± 0.012 0.21 ± 0.015 48.8% ± 3.9% 66.8% ± 4.2% 3084 ± 250 InnerMono 43.5 ± 2.69 0.149 ± 0.007 0.53 ± 0.026 49.5% ± 4.0% 67.5% ± 4.3% 3670 ± 290 CoPAL 11.9 ± 1.02 0.302 ± 0.017 0.85 ± 0.078 55.8% ± 4.5% 72.6% ± 3.9% 4846 ± 410 SMART-LLM 28.2 ± 2.20 0.339 ± 0.018 0.40 ± 0.036 55.7% ± 3.8% 74.3% ± 4.1% 5085 ± 395 Baseline 25.0 ± 2.03 0.200 ± 0.009 0.79 ± 0.042 53.0% ± 3.6% 69.7% ± 3.5% 3671 ± 275 This benchmark also shows that no current method dominates all resilience aspects. CoPAL achieves a favorable rebound–extensibility trade-off, with low recovery cost (Crec=11.9C_rec=11.9) and the highest stress capacity (λ∗=0.85λ^*=0.85), but it requires long simulated execution steps. InnerMono has the best stability profile (β=0.149β=0.149), yet its recovery cost remains high (Crec=43.5C_rec=43.5). AgentEvolver has relatively low recovery cost (Crec=19.2C_rec=19.2), yet its stress capacity is the weakest among all methods (λ∗=0.21λ^*=0.21), showing limited graceful extensibility. These results demonstrate that resilience should be interpreted as a multi-dimensional execution profile rather than a single scalar ranking. EAS Methods Resilience Signatures Under the same benchmark protocol, we analyze distinct behaviors (performance, simulation steps, rebound, stability, and graceful extensibility) among the methods in Fig. 5. The shape of the radar plot serves as a visual fingerprint for the EAS resilience profile, and exhibits different resilience profiles, strength, and weakness. Some methods continue execution through costly recovery, some maintain stable execution, some fluctuate during planning, and some show graceful extensibility under stress. These profiles provide a resilience overview of current EAS methods and serve as a diagnostic handbook for optimization experiments. We conclude resilience benchmark findings as below: Figure 5: The Resilience Diagnosis. Visualizing the distinct behavioral archetypes of EASs. Resilience Evaluation Benchmark Findings 1) Outcome-equivalent executions can be resilience-inequivalent. Success rate and completion describe whether an EAS finishes the task, but they do not reveal how much recovery, replanning instability, or stress margin is consumed during execution. Resilience metrics expose these hidden process differences. 2) There is no current EAS method dominates all resilience aspects. Some methods recover efficiently, some remain stable under semantic variation, and some tolerate stress, but no method dominates all dimensions. This indicates that EAS deployment should be matched to the target failure regime: difficult tasks require stronger Graceful Extensibility, while sensitive tasks require stronger Stability. 3) Resilience metrics provide actionable diagnosis. Abnormal metric patterns can be traced back to concrete execution events, such as repeated recovery loops, action oscillation, or stress-boundary collapse. This makes the evaluation layer not only a benchmark tool, but also a diagnostic basis for the metric-guided optimizations in Sec. 4. 4 EAS Optimization: Metric-Guided Resilience Diagnostics Based on the resilience metrics and the EAS resilience profiles, we introduce a unified metric-guided resilience optimization to the EAS execution. The goal is not to replace the original planner, but to use our resilience metrics as diagnostic signals that indicate where the execution loop should be minimally repaired. We implement three targeted optimizations for Rebound, Stability, and Graceful Extensibility, with implementation details provided in our Code Repository and Project Website. We compare the Original and Optimized variants under identical evaluation conditions. Table 3 shows that the resilience metrics are not only descriptive, but can also guide practical resilience improvements. What’s more, targeted optimization of a single resilient aspect often leads to compromises in other resilience aspects (optimizing GE results in high recovery cost CrecC_rec). This confirms our findings: resilience is a multi-dimensional structural balance, and our resilient evaluation layer could capture this trade-off. Table 3: Comparative results of metric-guided targeted improvements. Target Methods Rebound CrecC_rec ↓ Rec. Win. ↓ βstep _step ↓ β ↓ GE. Comp. ↑ Formal GE Baseline 59.11 3.13 0.2881 0.6834 0.887 – Rebound Optimized 33.73 ↓ 42.94% 2.38 ↓ 23.96% 0.2079 ↓ 27.84% 0.6870 ↑ 0.53% 0.819 ↓ 7.67% – Baseline 51.01 2.63 0.3185 0.6788 0.903 – Stability Optimized 43.01 ↓ 15.68% 3.00 ↑ 14.07% 0.2552 ↓ 19.87% 0.6772 ↓ 0.24% 0.750 ↓ 16.94% – Baseline 36.70 1.33 1.4594 0.6917 0.833 Incomplete GE Optimized 99.59 ↑ 171.36% 2.25 ↑ 69.17% 0.7215 ↓ 50.56% 0.6371 ↓ 7.89% 0.917 ↑ 10.08% Complete† † Formal GE is reported only for GE stress-tests, since it requires a stress-response curve over λ grid. Opt1 Recovery: Feedback-Guided Recovery Implementation High recovery cost CrecC_rec and long recovery windows indicate that the agent often spends extra reasoning or physical actions after local failures. To address this failure mode, we add a feedback-guided recovery layer on top of the centralized LLM planner. The layer monitors World-Graph evidence, recent execution feedback, and task-progress stagnation, and diagnoses recoverable faults such as missing graph nodes, stale object locations, repeated tool failures, or invalid object references. Instead of overriding the planner with a separate recovery policy, we get Rebound Guidance Πt+1 _t+1 into the next planning context, including perception refresh, reflection, state summarization, or belief rollback. Then we formalize it as Habitat skill tools for better use. This keeps the original action space unchanged while converting low-level execution feedback ℱtF_t into planner recovery hints. As shown in Table 3, this optimization reduces CrecC_rec from 59.1159.11 to 33.7333.73 and the recovery window from 3.133.13 to 2.382.38, showing that the metric-guided feedback loop directly reduces redundant recovery effort. Opt2 Stability: Consistency State Record High β values reveal unstable replanning behavior, where the agent adapts to new feedback but may oscillate between similar actions, repeat completed subtasks, or drift away from the current execution ctc_t. We therefore introduce a consistency record for replanning. At each step, it constructs a read-only phase guidance from the task goal, current phase, World-Graph state, current room and holding state, completed and pending subtasks, and recent action-response history. This design improves consistency without making the policy brittle. The optimized variant reduces βstep _step from 0.31850.3185 to 0.25520.2552 and improves β from 0.67880.6788 to 0.67720.6772, indicating that the stability metric captures and guides reductions in local replanning variance. Opt3 GE: Boundary Long-tail Control for Difficulties Graceful Extensibility focuses on whether performance degrades smoothly as stress increases, rather than collapsing or entering unbounded long-tail behavior. In high-stress cases, we observed that the planner may repeatedly wait or receive near-duplicate execution feedback without making progress. Such behavior inflates execution cost and makes the stress-response curve insensitive to true adaptive capacity. To control this boundary effect, we add a long-tail monitor that detects dual-wait patterns, semantically repeated feedback, and stagnant task progress. When a boundary stall BtB_t is detected, the system injects a bounded cognitive reset that discourages further non-progressive waiting and requests either a concrete productive action or termination. This converts uncontrolled long-tail execution into measurable boundary execution. 5 Conclusion and Future Work We presented the first Resilience Evaluation Framework for EASs. By designing novel metrics: Rebound, Stability, and Graceful Extensibility, our framework can expose resilience hidden pathologies, such as high-cost recovery and catastrophic forgetting, which remain invisible to outcome-centric metrics. Empirical analysis across representative baselines identified distinct Resilience Archetypes, confirming that architectural choices involve quantifiable trade-offs between execution, stability, and adaptability. The proposed optimization framework can help EASs developers to diagnose root causes and enforce non-degradation contracts to build a resilient and trustworthy EAS. Future Work Future work can extend resilience evaluation along three directions: Sim2Real validation of whether simulation-derived resilience profiles transfer to physical robots, theoretical analysis of trade-offs among resilience aspects, and deployment-aware optimization that uses benchmarked resilience signatures to improve EASs for target application conditions. These directions can move resilience evaluation toward more principled, deployable, and balanced EAS design. References [1] A. F. Agarap (2018) Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375. Cited by: §3.1. [2] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. (2022) Do as i can, not as i say: grounding language in robotic affordances. arXiv preprint arXiv:2204.01691. Cited by: §2. [3] A. Avizienis, J. Laprie, B. Randell, and C. Landwehr (2004) Basic concepts and taxonomy of dependable and secure computing. IEEE transactions on dependable and secure computing 1 (1), p. 11–33. Cited by: §1. [4] R. Buyya et al. (2026) Agentic artificial intelligence (ai): architectures, taxonomies, and evaluation of large language model agents. arXiv preprint arXiv:2601.12560. Cited by: §1. [5] M. Cavorsi, L. Sabattini, and S. Gil (2023) Multirobot adversarial resilience using control barrier functions. IEEE Transactions on Robotics 40, p. 797–815. Cited by: §2. [6] Z. Chen, Y. Gao, L. Wang, Y. Zhao, Y. Liu, J. Li, X. Zheng, Z. Wu, C. Wang, X. Ma, et al. (2026) HazardArena: evaluating semantic safety in vision-language-action models. arXiv preprint arXiv:2604.12447. Cited by: §2. [7] R. I. Cook and B. A. Long (2021) Building and revising adaptive capacity sharing for technical incident response: a case of resilience engineering. Applied ergonomics 90, p. 103240. Cited by: §2. [8] P. Fung, Y. Bachrach, A. Celikyilmaz, K. Chaudhuri, D. Chen, W. Chung, E. Dupoux, H. Gong, H. Jégou, A. Lazaric, et al. (2025) Embodied ai agents: modeling the world. arXiv preprint arXiv:2506.22355. Cited by: §1. [9] C. Gao, X. Hu, S. Gao, X. Xia, and Z. Jin (2025) The current challenges of software engineering in the era of large language models. ACM Transactions on Software Engineering and Methodology 34 (5), p. 1–30. Cited by: §1. [10] Y. Guo, Y. Wang, L. Zha, and J. Chen (2024) Doremi: grounding language model by detecting and recovering from plan-execution misalignment. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 12124–12131. Cited by: §2. [11] C. S. Holling et al. (1973) Resilience and stability of ecological systems. International Institute for Applied Systems Analysis Laxenburg. Cited by: §1. [12] E. Hollnagel (2018) Safety-i and safety-i: the past and future of safety management. CRC press. Cited by: §2. [13] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. (2022) Inner monologue: embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608. Cited by: §2. [14] F. Joublin, A. Ceravola, P. Smirnov, F. Ocker, J. Deigmoeller, A. Belardinelli, C. Wang, S. Hasler, D. Tanneberg, and M. Gienger (2024) CoPAL: corrective planning of robot actions with large language models. In 2024 ieee international conference on robotics and automation (ICRA), p. 8664–8670. Cited by: §2. [15] P. Koopman and M. Wagner (2016) Challenges in autonomous vehicle testing and validation. SAE International Journal of Transportation Safety 4 (1), p. 15–24. Cited by: §2. [16] M. Li, S. Zhao, Q. Wang, K. Wang, Y. Zhou, S. Srivastava, C. Gokmen, T. Lee, E. L. Li, R. Zhang, et al. (2024) Embodied agent interface: benchmarking llms for embodied decision making. Advances in Neural Information Processing Systems 37, p. 100428–100534. Cited by: §1, §2. [17] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng (2022) Code as policies: language model programs for embodied control. arXiv preprint arXiv:2209.07753. Cited by: §2. [18] Y. Liu, W. Chen, Y. Bai, X. Liang, G. Li, W. Gao, and L. Lin (2025) Aligning cyber space with physical world: a comprehensive survey on embodied ai. IEEE/ASME Transactions on Mechatronics. Cited by: §1, §2. [19] X. Lu, Z. Chen, X. Hu, Y. Zhou, W. Zhang, D. Liu, L. Sheng, and J. Shao (2025) IS-bench: evaluating interactive safety of vlm-driven embodied agents in daily household tasks. arXiv preprint arXiv:2506.16402. Cited by: §1, §2. [20] C. Ma, G. Yang, K. Lu, S. Xu, B. Byrne, N. Trigoni, and A. Markham (2026) Cyclevla: proactive self-correcting vision-language-action models via subtask backtracking and minimum bayes risk decoding. arXiv preprint arXiv:2601.02295. Cited by: §2. [21] P. C. Mahalanobis (2018) On the generalized distance in statistics. Sankhyā: The Indian Journal of Statistics, Series A (2008-) 80, p. S1–S7. Cited by: §3.1. [22] V. Moskalenko, V. Kharchenko, A. Moskalenko, and B. Kuzikov (2023) Resilience and resilient systems of artificial intelligence: taxonomy, models and methods. Algorithms 16 (3), p. 165. Cited by: §1. [23] F. Ni, M. Zhang, P. Li, Y. Yuan, L. Zhang, Y. Liu, P. Han, L. Kou, S. Ma, J. Qiao, D. G. A. Bravo, Y. Wang, X. Hu, Z. Zhang, X. Yao, Y. Li, Z. Zhang, Y. Wen, Y. Chen, X. Liang, L. Lin, B. He, H. Bou-Ammar, H. Wang, H. Xu, J. Deng, S. Luo, S. Jiang, W. Pan, Y. Gao, S. Zafeiriou, J. Peters, Y. Zhuang, Y. Zhang, Y. Zheng, H. Tang, and J. Hao (2025) Embodied arena: a comprehensive, unified, and evolving evaluation platform for embodied ai. External Links: 2509.15273, Link Cited by: §1. [24] B. Or (2025) MTTR-a: measuring cognitive recovery latency in multi-agent systems. arXiv preprint arXiv:2511.20663. Cited by: §2, §2. [25] A. Prorok, M. Malencia, L. Carlone, G. S. Sukhatme, B. M. Sadler, and V. Kumar (2021) Beyond robustness: a taxonomy of approaches towards resilient multi-robot systems. arXiv preprint arXiv:2109.12343. Cited by: §2. [26] X. Puig, E. Undersander, A. Szot, M. Dallaire Cote, T. Yang, R. Partsey, R. Desai, A. Clegg, M. Hlavac, S. Y. Min, et al. (2024) Habitat 3.0: a co-habitat for humans, avatars, and robots. In International Conference on Learning Representations, Vol. 2024, p. 15306–15336. Cited by: §3.2.1. [27] N. Sharma, A. Tabandeh, and P. Gardoni (2018) Resilience analysis: a mathematical formulation to model resilience of engineering systems. Sustainable and Resilient Infrastructure 3 (2), p. 49–67. Cited by: §2. [28] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. Advances in neural information processing systems 36, p. 8634–8652. Cited by: §2. [29] M. Skreta, Z. Zhou, J. L. Yuan, K. Darvish, A. Aspuru-Guzik, and A. Garg (2024) Replan: robotic replanning with perception and language models. arXiv preprint arXiv:2401.04157. Cited by: §2. [30] X. Tan, B. Liu, Y. Bao, Q. /. i. Tian, Z. Gao, X. Wu, Z. Luo, S. Wang, Y. Zhang, X. Wang, et al. (2025) Towards safe and trustworthy embodied ai: foundations, status, and prospects. Shanghai AI Lab., Shanghai, China, Tech. Rep.. Cited by: §1, §2. [31] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar (2023) Voyager: an open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291. Cited by: §2. [32] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al. (2024) A survey on large language model based autonomous agents. Frontiers of Computer Science 18 (6), p. 186345. Cited by: §1. [33] D. D. Woods (2015) Four concepts for resilience and the implications for the future of resilience engineering. Reliability engineering & system safety 141, p. 5–9. Cited by: §1, §2. [34] R. Yang, H. Chen, J. Zhang, M. Zhao, C. Qian, K. Wang, Q. Wang, T. V. Koripella, M. Movahedi, M. Li, et al. (2025) Embodiedbench: comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint arXiv:2502.09560. Cited by: §2. [35] S. Yin, X. Pang, Y. Ding, M. Chen, Y. Bi, Y. Xiong, W. Huang, Z. Xiang, J. Shao, and S. Chen (2024) Safeagentbench: a benchmark for safe task planning of embodied llm agents. arXiv preprint arXiv:2412.13178. Cited by: §1, §2. [36] N. Yokoyama, S. Ha, and D. Batra (2021) Success weighted by completion time: a dynamics-aware evaluation criteria for embodied navigation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 1562–1569. Cited by: §1, §2. [37] T. Yuan, Z. He, L. Dong, Y. Wang, R. Zhao, T. Xia, L. Xu, B. Zhou, F. Li, Z. Zhang, et al. (2024) R-judge: benchmarking safety risk awareness for llm agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 1467–1490. Cited by: §2. [38] Y. Zhai, S. Tao, C. Chen, A. Zou, Z. Chen, Q. Fu, S. Mai, L. Yu, J. Deng, Z. Cao, et al. (2025) AgentEvolver: towards efficient self-evolving agent system. arXiv preprint arXiv:2511.10395. Cited by: §2. [39] S. Zhang, Z. Xu, P. Liu, X. Yu, Y. Li, Q. Gao, Z. Fei, Z. Yin, Z. Wu, Y. Jiang, et al. (2025) Vlabench: a large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 11142–11152. Cited by: §2. [40] T. Zhang, K. Qu, Z. Li, J. Wu, M. Hutter, M. Li, and F. Shi (2026) Using large language models for embodied planning introduces systematic safety risks. arXiv preprint arXiv:2604.18463. Cited by: §2. [41] Z. Zhu, B. Wu, Z. Zhang, L. Han, Q. Liu, and B. Wu (2024) Earbench: towards evaluating physical risk awareness for task planning of foundation model-based embodied ai agents. arXiv preprint arXiv:2408.04449. Cited by: §2.