Paper deep dive
CIVA: Critic-Induced Value-Subspace Attacks on Visual World-Model Agents
Jiancheng Wang, Mingli Zhu, Tong Zhang, Jiaqi Ruan, Wei Wang, Siyuan Liang, Dacheng Tao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/24/2026, 6:12:46 AM
Summary
The paper introduces CIVA (Critic-Induced Value-Subspace Attacks), a white-box, causal, online adversarial attack method targeting visual world-model agents like DreamerV3. CIVA exploits the observation that critic-guided perturbations concentrate in a low-dimensional subspace. It extracts this subspace offline using SVD on critic-guided PGD perturbations and performs online optimization within this subspace, using exponential moving average (EMA) for temporal coherence. Experiments on DMC walker walk, Atari Pong, and Crafter demonstrate that CIVA outperforms five recent methods, achieving significant reward drops with low temporal variation.
Entities (13)
Relation Signals (11)
CIVA → evaluatedon → DMC walker walk
confidence 95% · Extensive experiments on DMC walker walk, Atari Pong, and Crafter show that CIVA consistently outperforms five recent methods
CIVA → evaluatedon → Crafter
confidence 95% · Extensive experiments on DMC walker walk, Atari Pong, and Crafter show that CIVA consistently outperforms five recent methods
CIVA → evaluatedon → Atari Pong
confidence 95% · Extensive experiments on DMC walker walk, Atari Pong, and Crafter show that CIVA consistently outperforms five recent methods
CIVA → targets → DreamerV3
confidence 95% · We study white-box, causal, online attacks on such agents [DreamerV3] and propose Critic-Induced Value-Subspace Attacks (CIVA).
CIVA → uses → EMA
confidence 90% · At test time, it optimizes only the subspace coefficients, smooths them with an exponential moving average (EMA)
CIVA → uses → SVD
confidence 90% · CIVA first probes the frozen victim offline with critic-guided PGD and extracts a low-rank value-subspace by SVD.
CIVA → outperforms → Illusory
confidence 85% · Extensive experiments on DMC walker walk, Atari Pong, and Crafter show that CIVA consistently outperforms five recent methods
CIVA → outperforms →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Visual world-model agents such as DreamerV3 act through a recurrent latent state rather than a single observation, which weakens frame-wise observation attacks and makes their perturbations vary sharply over time under a strict per-frame perturbation constraint. We study white-box, causal, online attacks on such agents and propose Critic-Induced Value-Subspace Attacks (\textbf{CIVA}). Our key observation is that, along a rollout, critic-guided perturbations concentrate in a low-dimensional subspace induced by the victim's own critic. Based on this observation, CIVA first probes the frozen victim offline with critic-guided PGD and extracts a low-rank value-subspace by SVD. At test time, it optimizes only the subspace coefficients, smooths them with an exponential moving average (EMA), and maps them back to pixels. This design attacks value-sensitive recurrent dynamics while keeping the online optimization cheap and temporally coherent. Extensive experiments on DMC walker walk, Atari Pong, and Crafter show that CIVA consistently outperforms five recent methods; on DMC walker walk, it achieves the largest reward drop of 26.07\% while keeping temporal variation low, with TempAbs of 0.646.
Tags
Links
- Source: https://arxiv.org/abs/2608.21114v1
- Canonical: https://arxiv.org/abs/2608.21114v1
Trouble viewing inline? Open PDF directly →
Full Text
101,630 characters extracted from source content.
Expand or collapse full text
CIVA: Critic-Induced Value-Subspace Attacks on Visual World-Model Agents Jiancheng WangMingli ZhuTong ZhangJiaqi RuanWei Wang Siyuan LiangDacheng Tao[4pt] Corresponding author:pandaliang521@gmail.com Abstract Visual world-model agents such as DreamerV3 act through a recurrent latent state rather than a single observation, which weakens frame-wise observation attacks and makes their perturbations vary sharply over time under a strict per-frame perturbation constraint. We study white-box, causal, online attacks on such agents and propose Critic-Induced Value-Subspace Attacks (CIVA). Our key observation is that, along a rollout, critic-guided perturbations concentrate in a low-dimensional subspace induced by the victim’s own critic. Based on this observation, CIVA first probes the frozen victim offline with critic-guided PGD and extracts a low-rank value-subspace by SVD. At test time, it optimizes only the subspace coefficients, smooths them with an exponential moving average (EMA), and maps them back to pixels. This design attacks value-sensitive recurrent dynamics while keeping the online optimization cheap and temporally coherent. Extensive experiments on DMC walker walk, Atari Pong, and Crafter show that CIVA consistently outperforms five recent methods; on DMC walker walk, it achieves the largest reward drop of 26.07% while keeping temporal variation low, with TempAbs of 0.646. 1 Introduction Visual world models are becoming a mainstream paradigm for visual decision-making. Rather than reacting to the current frame alone, Dreamer-style agents write each observation into a recurrent latent state and act through learned dynamics in that internal space [14, 15, 16, 17]. Related lines on planning with learned models [52], transformer- [42, 51, 71] and diffusion-based world models [1], large-scale generative environments [4], scalable model-based control [19], and robot deployment [65] further reinforce recurrent latent dynamics as a core primitive in visual RL [6, 13, 66]. As these agents move closer to safety-critical use, understanding how to attack them becomes increasingly important. Existing adversarial attacks on RL already show that deep policies are highly sensitive to small perturbations in visual or state observations [54, 11, 47, 45, 5, 41, 21, 26, 35, 48, 10, 31, 63, 30, 32, 29, 36, 58, 59]. Stronger variants have since been developed from several angles, including robust state-observation formulations [69, 68], real-time and universal observation attacks [56, 44, 61, 37], policy-aware attackers [53], detectability-aware stealthy attacks [9], and policy-distribution-based methods [7, 25, 2]. However, most of these methods still optimize perturbations directly in pixel space and treat each frame as an essentially independent target. That assumption is mismatched with visual world-model agents: what matters is not only whether one frame is perturbed, but whether the perturbation keeps steering the recurrent latent state under a realistic per-frame compute budget. We study this setting in a white-box, causal, online attack regime against a frozen DreamerV3 victim. The attacker may modify only the current image observation and must produce the perturbation before the next environment step. Across DMC walker walk (DeepMind Control Suite [55]), Atari Pong, and Crafter, we observe two consistent phenomena. First, naive per-frame full-pixel projected gradient descent (PGD) [41] is a weak primitive here: recurrent latent dynamics dilute isolated high-frequency perturbations, so greedy frame-wise attacks struggle to accumulate influence over long horizons. Second, effective attack directions do not spread uniformly across the full pixel space. When we stack critic-guided perturbations across time, their energy concentrates in a surprisingly low-dimensional subspace. This echoes subspace and low-frequency findings from supervised learning [67, 12, 44, 40, 23], but the structure here is induced by the victim’s own value landscape rather than by input statistics. These observations suggest a different attack strategy: identify value-sensitive directions once, then reuse them causally over time. We instantiate this idea as CIVA. CIVA first collects critic-guided perturbations offline and applies singular value decomposition (SVD) to extract a low-dimensional, victim-aware attack subspace. At deployment time, it no longer searches the full pixel space at every frame; instead, it optimizes only the subspace coefficients and smooths them with an exponential moving average before lifting them back to pixels. This directly addresses the core difficulty of attacking world-model agents under a tight ℓ∞ _∞ budget: the perturbation must remain strong enough to influence recurrent latent dynamics while also staying computationally cheap, temporally coherent, and visually subtle [20, 70, 60, 24, 72]. Across continuous control, discrete control, and open-ended tasks, CIVA yields stronger and more stable degradation than representative observation-space baselines [69, 56, 53, 9, 7], while preserving favorable temporal smoothness and perceptual similarity. Ablations further show that the gains come from the combination of a critic-induced subspace, online optimization constrained to that subspace, and temporal smoothing on its coefficients. Our contributions are threefold: • We identify a key mismatch between conventional frame-wise observation attacks and visual world-model agents: because decisions depend on recurrent latent state, successful attacks must shape the victim’s internal representation over time rather than only degrade individual frames [38, 39]. • We propose CIVA, a critic-induced value-subspace attack that extracts a low-rank victim-aware perturbation basis offline and performs efficient causal online optimization with temporal smoothing inside that subspace. • We evaluate CIVA on DMC walker walk, Atari Pong, and Crafter across attack effectiveness, action-distribution shift, temporal variation, and perceptual similarity, showing that low-dimensional critic-aligned perturbations are more effective for world-model agents than direct full-pixel frame-wise optimization. 2 Related Work World-model reinforcement learning. World models build policies around a recurrent latent dynamics model that summarizes past observations and supports multi-step prediction [14]. The Dreamer line demonstrates that this recipe scales across continuous control, Atari, and open-ended visual domains under a largely unified configuration [15, 16, 17]. Parallel efforts extend the same general direction to planning with learned models [52], transformer- [42, 51, 71] and diffusion-based world models [1], large-scale generative environments [4], scalable model-based control [19], and robotic deployment [65]; recent surveys summarize this broader trend [6, 13, 66]. For our purposes, the important point is not only that these agents are powerful, but that their decision process is mediated by a recurrent latent state. That makes their adversarial surface fundamentally temporal rather than frame-local. Adversarial attacks on deep policies. Adversarial vulnerability in supervised learning [54, 11, 47, 45, 5, 41] carries over directly to RL, where small perturbations in observations can cause severe behavioral degradation [21, 26, 35, 48]. Follow-up work has strengthened this line through robust state-observation formulations [69, 68], real-time and universal perturbations [56, 61, 37], policy-aware attackers [53], detectability-aware attacks [9, 20, 24], and policy-distribution-based objectives [7, 70, 60]. Other work studies adversarial agents or robustness notions more broadly in multi-agent and deep-RL settings [10, 25, 2, 38, 39]. These methods provide strong attack baselines, but most still optimize directly in pixel space and effectively target each frame separately. Our setting differs because the victim is a visual world-model agent whose behavior depends on how perturbations accumulate inside recurrent latent dynamics. Robust RL and defenses. From the defense side, robust RL has been studied via adversarial training with a learned attacker [49, 68], regularization against worst-case observation perturbations [46], worst-case-aware training without explicit adversarial rollouts [33], and game-theoretic formulations for temporally coupled disturbances [34]. This literature mainly asks how to train agents that remain reliable under perturbation. Our question is different: before defending visual world-model agents, we need to understand what strong causal observation attacks look like against them in the first place, and whether their recurrent latent structure induces a different geometry of effective perturbations. Structured, low-dimensional, and temporally coherent attacks. Another relevant thread asks whether adversarial perturbations really need the full pixel space. In supervised learning, universal perturbations [44], low-frequency attacks [12], and subspace attacks [67, 40, 23] all show that effective directions often lie in much lower-dimensional or smoother spaces than raw pixels. In video and streaming settings, temporally coordinated attacks can be more effective and less perceptible than independent frame-wise perturbations [64, 28]. RL work such as real-time universal perturbations and policy-aware attackers similarly introduces cross-frame structure to improve efficiency or attack strength [56, 53, 7]. CIVA is related in spirit but differs in the origin of the subspace and in how it is deployed. Our attack basis is not hand-crafted, random, or derived from observation statistics; it is extracted directly from the victim critic’s attack gradients collected on rollouts of the target agent. The resulting basis is then reused causally across frames, with temporal coherence enforced by smoothing the subspace coefficients. This ties low-dimensional attack structure directly to the value landscape of a recurrent world-model agent. 3 Threat Model 3.1 Victim Model We attack visual world-model reinforcement learning agents. The victim is a pretrained DreamerV3 policy [17]. At each step the agent receives an RGB image ot∈[0,255]H×W×Co_t∈[0,255]^H× W× C with H=W=64H=W=64 and C=3C=3. The world model updates a recurrent latent state from this image. The actor head then outputs an action, and the critic head outputs a categorical value distribution. We use the same victim setup on three tasks that cover different decision regimes: DMC walker walk (continuous control), Atari Pong (discrete control), and Crafter (open-ended world). The victim weights and the environment dynamics are frozen during the attack. The attacker does not retrain or fine-tune the victim. 3.2 Attacker’s Capability The attacker can only modify the image observation that is fed into the agent. It cannot touch the actions, rewards, environment state, or the recurrent latent. We use a white-box online setting. The attacker has full access to the policy and value heads and can backpropagate through them. The attacker is also causal: at frame t it can only use the current and past observations, and it must produce δt _t before the next environment step. To match the comparison protocol in Section 5, this causal constraint applies at deployment time to all methods, including baselines. Methods that require one-time preprocessing (e.g., universal perturbation fitting or subspace extraction) are allowed to use clean rollouts from the same frozen victim before deployment, but the resulting parameters (e.g., the value-subspace basis VrV_r used by CIVA) are then frozen for the entire evaluation; no method may access future test observations, alter the environment. Each perturbation is bounded by an ℓ∞ _∞ pixel budget, ∥δt∥∞≤ε _t _∞≤ . We use ε=24/255 =24/255 throughout this paper. A larger budget makes the perturbation visually obvious, while a much smaller budget barely degrades the agent. We find 24/25524/255 to be a good trade-off between attack strength and visual stealth. The attacker also has only a small per-frame compute budget, i.e. a few gradient steps per observation. A practical attack must therefore be both parameter-efficient and step-efficient. 3.3 Attacker’s Goal Let T be the episode horizon and tstartt_start the attack onset. The attacker minimizes the cumulative episodic return: min∑t=tstartTδtt=tstartTrts.t.∥δt∥∞≤ε,ot+δt∈[0,255]H×W×C. _\ _t\_t=t_start^T\; _t=t_start^Tr_t .t. _t _∞≤ ,\ \ o_t+ _t∈[0,255]^H× W× C. (1) The reward is not differentiable through the environment. We therefore optimize victim-exposed differentiable surrogates, such as value-based, action-based, or reward-head objectives. Baselines are evaluated under the same threat model and keep their method-specific optimization forms when applicable, but all optimization signals are restricted to victim-exposed differentiable quantities and never use true environment reward gradients. Beyond reducing return, an effective attack should also be stealthy: temporally coherent, spatially structured, and visually close to the clean observation. 3.4 Challenges The above setting raises three coupled challenges that motivate our design. (C1) Recurrent dilution. A world-model agent integrates observations through a recurrent latent state. This latent state smooths out isolated high-frequency pixel noise. As a result, single-frame greedy attacks struggle to accumulate effect over a long horizon. (C2) Cost of full-pixel per-frame PGD. Running multi-step white-box PGD in the full D=H×W×3D=H\!×\!W\!×\!3 pixel space at every frame is expensive. It also produces unstructured and temporally jittery perturbations. This is incompatible with the small per-frame compute budget stated in Section 3.2. (C3) Budget–effectiveness–stealth trilemma. Under a tight ℓ∞ _∞ budget, we want strong return suppression, temporal smoothness, and visual naturalness at the same time. Naive PGD usually sacrifices at least one of the three. These three challenges point to the same need: a low-dimensional, data-driven perturbation parameterization that is shared across frames and explicitly enforces temporal coherence. We develop such a design in Section 4. 4 Method Motivated by the challenges in Section 3.4, we propose CIVA (Critic-Induced Value-Subspace Attacks). Unlike prior subspace attacks that build the search space from input statistics (e.g., PCA over observations or hand-crafted DCT bases) [67, 12], CIVA constructs its low-dimensional subspace directly from the victim critic’s own attack gradients. As shown in Figure 1, CIVA proceeds in two stages. Stage A (offline) probes the victim with critic-guided per-frame PGD on a clean rollout, collects the resulting pixel perturbations, and extracts a low-dimensional value-subspace via SVD. Stage B (online) performs per-frame PGD only inside this subspace, and uses an exponential moving average (EMA) on the subspace coefficients to enforce temporal coherence. We detail the two stages in Section 4.1 and Section 4.2. Figure 1: CIVA framework. Stage A extracts a critic-induced value-subspace from offline PGD probes; Stage B performs online subspace-PGD with EMA smoothing. 4.1 Offline Value-Subspace Discovery Critic-guided per-frame PGD probing. The strong baseline implied by Section 3.2 is per-frame full-pixel PGD: at every frame the attacker searches the full D=H×W×CD=H\!×\!W\!×\!C pixel space with multiple gradient steps [41, 56]. The resulting perturbations are independent across frames, temporally jittery, and sit in a D=12288D=12288 dimensional space. A natural question is whether all D pixel directions are truly needed, or whether the effective attack directions concentrate in a much lower-dimensional subspace. To answer this, we first collect a set of critic-guided pixel perturbations on a clean rollout. We start the collection only after a short 200200-step warm-up so that the recurrent latent state has stabilised. The DreamerV3 critic outputs a categorical value distribution p(k∣ot)p(k o_t) over K symlog-spaced bins [17]. We do not need to decode this distribution back into a calibrated value: any quantity that is monotone in the agent’s predicted return suffices as an attack signal. We therefore use the expected bin index v(ot)=∑k=1Kk⋅softmax(ℓk(ot)),v(o_t)\;=\; _k=1^Kk·softmax( _k(o_t)), (2) as a cheap differentiable surrogate (since the bins are sorted by value, lowering v(ot)v(o_t) shifts probability mass toward lower-value bins), and run L steps of ℓ∞ _∞ PGD per frame: δ(ℓ+1)=Πℬε(δ(ℓ)−η⋅normalize(∇δv(ot+δ(ℓ)))),δ^( +1)\;=\; _B_ \! (δ^( )-η·normalize ( _δv(o_t+δ^( )) ) ), (3) where normalize(g):=g/[g2]normalize(g):=g/ E[g^2] is RMS-normalization of the (sub)gradient [57] and ℬε=δ:∥δ∥∞≤εB_ =\δ: δ _∞≤ \ is composed with the pixel-range projection ot+δ∈[0,255]H×W×Co_t+δ∈[0,255]^H× W× C. We denote the final perturbation as δi:=δ(L) _i:=δ^(L) and repeat this for M frames. Stacking the flattened perturbations gives the attack-direction matrix Δ=[δ1,…,δM]⊤∈ℝM×D. \;=\;[ _1,…, _M] ^M× D. (4) Each row of Δ is, by construction, a pixel direction that empirically drives the victim critic toward lower predicted value. It is neither an observation, nor a hand-crafted low-frequency prior, but the victim’s own “where to push pixels” signal. Constructing the critic-induced value-subspace. Given Δ , we take a truncated SVD [8] Δ=UΣV⊤,Vr:=V[:,:r]∈ℝD×r. \;=\;U V , V_r\;:=\;V[:,:r] ^D× r. (5) The columns of VrV_r are the top-r right singular vectors of Δ and form an orthonormal basis of an r-dimensional subspace of ℝDR^D; we call its column span r:=col(Vr)S_r:=col(V_r) the Critic-Induced Value-Subspace. CIVA confines all online perturbations to rS_r. The basis VrV_r is computed once offline and remains frozen at evaluation time. With r=32≪D=12288r=32 D=12288, sharing this victim-aware, spatially structured basis across frames directly addresses challenges: it shrinks the per-frame search space, avoids the high-frequency artifacts of independent pixel-level PGD, and keeps consecutive perturbations on a common direction set that the EMA in Section 4.2 can smooth without losing critic-effectiveness. Relation to existing subspace attacks. Prior subspace attacks typically derive their search spaces from input statistics, such as PCA or hand-crafted DCT bases, or from random/query-saving projections in black-box settings. CIVA instead extracts its subspace directly from the victim critic’s gradients, making VrV_r victim-aware by construction. This distinction matters in our online recurrent setting, where temporal coherence is central and is evaluated explicitly in Section 5 through comparison with a random-basis ablation. 4.2 Online Subspace Attack Given VrV_r, we now describe how CIVA produces the per-frame perturbation δt _t during deployment. Value-based attack objective. The true reward rtr_t is non-differentiable through the environment (see Section 3.2), so we minimize the same critic-based surrogate used in Stage A. Given the current observation oto_t and a perturbation δt _t, we define the per-frame loss ℒval(δt,ot)=v(ot+δt),L_val( _t;o_t)\;=\;v(o_t+ _t), (6) where v(⋅)v(·) is the same expected value as in Eq. (2). Minimizing ℒvalL_val makes the agent believe its future return is lower than it actually is, and indirectly induces suboptimal actions. Using the same surrogate in Stage A and Stage B keeps the directions in VrV_r semantically aligned with the online optimization target. Subspace reparameterization. We reparameterize the per-frame perturbation as a low-dimensional coefficient [22] αt∈ℝr _t ^r: δt=Vrαt. _t\;=\;V_r\, _t. (7) Online PGD then operates in ℝrR^r rather than ℝDR^D. At each step we first take a pixel-space gradient gδ=∇δℒvalg_δ= _δL_val, project it into the subspace, and apply a normalized gradient step: gα=Vr⊤gδ,αt(ℓ+1)=αt(ℓ)−η⋅normalize(gα).g_α\;=\;V_r \,g_δ, _t^( +1)\;=\; _t^( )-η·normalize(g_α). (8) The ℓ∞ _∞ budget and the pixel-range constraint are still enforced in pixel space: after each update we lift αt _t to δt _t, project onto ℬεB_ and onto [0,255]H×W×C[0,255]^H× W× C, and re-project back to αt _t. This approximately enforces the subspace constraint, the ℓ∞ _∞ budget, and pixel validity at the same time; in practice the residual reprojection error is below 10−310^-3 in pixel scale and does not affect the reported results. Temporal coherence via subspace EMA. C3 in Section 3.2 requires the perturbation sequence to be temporally coherent rather than just frame-wise strong. We enforce this directly on the subspace coefficients via an exponential moving average [50]: αt=ρ⋅αt−1+(1−ρ)⋅αt⋆, _t\;=\;ρ· _t-1\;+\;(1-ρ)· _t , (9) where αt⋆ _t is the per-frame PGD optimum from Eq. (8), and ρ∈[0,1]ρ∈[0,1] controls the smoothing strength. With ρ→1ρ→ 1 the perturbation barely changes across frames (very smooth, slightly weaker); with ρ→0ρ→ 0 CIVA reduces to independent per-frame attack (strongest, but most jittery). We stress that the EMA is meaningful precisely because it is applied in the subspace. A pixel-space EMA on δt _t would act as a temporal low-pass filter and could smooth the perturbation off the critic-effective direction set. Inside rS_r, every αt _t is by construction a critic-effective direction, so temporal smoothing does not trade away attack strength. This is the direct payoff of sharing VrV_r across frames, as anticipated in Section 4.1. Joint online procedure. Putting Eq. (6)–(9) together, CIVA produces δt _t at frame t as follows. (i) Warm-start from the previous (post-EMA) coefficient αt−1 _t-1 and run a small number of subspace-PGD steps on ℒvalL_val (Eq. (8)) to obtain αt⋆ _t . (i) Smooth via Eq. (9) to get αt _t. (i) Lift to δt=Vrαt _t=V_r\, _t and project onto the ℓ∞ _∞ ball and the pixel range. (iv) Feed ot+δto_t+ _t to the victim and step the environment. The procedure is causal, online, parameter-efficient (only r scalars per frame), and step-efficient (a few PGD steps per frame), satisfying every constraint of Section 3.4. 5 Experiment 5.1 Experimental Setup Environments and victim. We evaluate on three visual control benchmarks covering distinct action regimes: DMC walker walk [55] (continuous), Atari Pong [3, 43] (discrete), and Crafter [18] (open-ended survival). All environments deliver ×364\!×\!64\!×\!3 RGB observations (D=12,288D=12,288) with a 1,000-step episode horizon. The victim is DreamerV3 [17] trained per task with the official configuration; the resulting checkpoint is frozen for all attacks. Baselines and attack budget. We compare CIVA against five representative observation-space attackers: MAD [69], UAP-RL [56], PA-AD [53], Illusory [9], and DAPGD [7]. All methods share an ℓ∞ _∞ per-frame budget ε=24/255 =24/255 and identical white-box access to the frozen victim. CIVA hyperparameters. Stage A (offline) collects M=600M=600 critic-guided per-frame PGD perturbations with L=8L=8 inner steps and step size η=ε/4η= /4, after a 200-step warm-up that lets the recurrent latent state stabilise. SVD on Δ∈ℝ600×12288 ^600× 12288 yields the top r=32r=32 right singular vectors that form the columns of Vr∈ℝ12288×32V_r ^12288× 32. Stage B (online) reuses L=8L=8 and η=ε/4η= /4 but optimises only the coefficient vector αt∈ℝr _t ^r, smoothed by an EMA with momentum ρ=0.75ρ=0.75 before being lifted back to pixels. The basis VrV_r is computed once and frozen at evaluation time. Evaluation protocol. Each method is evaluated over 55 independent random seeds (one episode per seed) and we report the across-seed means of episode reward, the reward drop ratio (Drop%) relative to the natural policy following the convention of [48, 69], the Kullback–Leibler divergence [27] between clean and attacked action distributions (Action-KL) as used in policy-distribution attacks [53, 7], the mean absolute frame-to-frame change of the perturbation (TempAbs) inspired by temporal-coherence measures for video adversarial attacks [64, 28], and SSIM [62] between clean and perturbed observations. All experiments run on a single NVIDIA RTX PRO 6000 Blackwell GPU (96 GB). 5.2 Main Results Tables 1 and 2 report main attack results on DMC walker walk (continuous control) and Atari Pong (discrete control). Crafter results are deferred to Appendix A.2. Table 1: Attack results on DMC walker walk. “Natural” is the unattacked baseline; arrows show the favorable direction for the attacker. Bold = best, underline = second best. Method Venue / Year Episode Reward ↓ Drop% ↑ Action-KL ↑ TempAbs ↓ SSIM ↑ Natural (no attack) — 952.25 — — — 1.000 MAD [69] NeurIPS 2020 934.71 1.84% 0.753 9.126 0.748 UAP-RL [56] ESORICS 2022 730.86 23.25% 1.000 0.00 0.656 PA-AD [53] ICLR 2022 786.75 17.38% 1.35 19.97 0.35 Illusory [9] ICLR 2024 795.13 16.50% 1.018 10.97 0.445 DAPGD [7] ICASSP 2025 773.33 18.79% 0.775 12.31 0.501 CIVA (Ours) 2026 703.97 26.07% 1.130 0.646 0.710 Table 2: Attack results on Atari Pong. Method Venue / Year Episode Reward ↓ Drop% ↑ Action-KL ↑ TempAbs ↓ SSIM ↑ Natural (no attack) — 21.0 — — — 1.000 MAD [69] NeurIPS 2020 7.0 66.67% 0.155 4.357 0.864 UAP-RL [56] ESORICS 2022 18.0 14.29% 0.098 0.000 0.344 PA-AD [53] ICLR 2022 17.0 19.05% 0.069 4.029 0.697 Illusory [9] ICLR 2024 16.0 23.81% 0.138 9.570 0.599 DAPGD [7] ICASSP 2025 19.0 9.52% 0.080 17.194 0.719 CIVA (Ours) 2026 3.0 85.71% 0.184 0.394 0.566 Figure 2: Attack dynamics over time. Cumulative reward (left pair) and Action-KL (right pair) on DMC walker walk and Atari Pong. The dashed line marks the attack start at t=200t=200. Attack effectiveness. Under the same ℓ∞=24/255 _∞=24/255 budget, Tables 1 and 2 show that CIVA achieves the strongest attack effectiveness on both tasks. On DMC walker walk, it attains the lowest attacked reward (703.97703.97) and the highest Drop% (26.07%26.07\%), outperforming the strongest baseline UAP-RL (23.25%23.25\%). On Atari Pong, the margin is larger: CIVA reaches 85.71%85.71\% Drop%, whereas the runner-up MAD reaches 66.67%66.67\% and all other baselines remain below 25%25\%. Figure 2 is consistent with the table results: after the attack starts at t=200t=200, CIVA yields the slowest cumulative reward growth on both tasks, finishing lowest on DMC and plateauing near reward 33 on Pong. Overall, CIVA not only produces the best final reward suppression, but also degrades performance more persistently over time. Behavioural deviation. The Action-KL column and the right half of Figure 2 show that CIVA produces the strongest behavioural deviation, reaching 1.1301.130 on DMC and 0.1840.184 on Pong. DMC also shows that KL alone is insufficient: PA-AD attains the largest Action-KL (1.351.35) but achieves only a 17.38%17.38\% reward drop. By constraining perturbations to the critic-relevant subspace, CIVA induces policy shifts that are better aligned with long-horizon value degradation. Temporal coherence and visual stealth. Per-frame baselines pay a heavy temporal cost: PA-AD, Illusory and DAPGD report TempAbs of 1010–2020 on DMC and 44–1717 on Pong, one to two orders of magnitude above CIVA (0.6460.646 and 0.3940.394). UAP-RL has lower TempAbs only because it injects a static pattern, which caps its Pong drop at 14.29%14.29\%. CIVA delivers the strongest reward drop while maintaining very low temporal variation, improving TempAbs by roughly 10×10× over per-frame baselines while keeping SSIM within 0.040.04–0.300.30 of the most stealth-oriented baseline (MAD). On the same GPU, the online subspace optimization further reduces per-frame attack latency by roughly an order of magnitude relative to full-pixel per-frame PGD (V2 in Section 5.4), since the inner PGD operates over r=32r=32 scalars instead of D=12,288D=12,288 pixels. 5.3 Analysis of Policy Decision Shifts To complement the aggregate reward and Action-KL metrics, Figure 3 compares the victim’s action logits under the natural policy, Illusory, and CIVA at three representative attack-time steps. Illusory changes the logit landscape but can leave the selected action unchanged, as at step 300. In contrast, CIVA consistently makes a different action dominant in these examples, showing that the critic-induced subspace translates distributional shifts into concrete decision changes. This also explains why CIVA’s large Action-KL in Tables 1 and 2 is coupled with larger reward degradation: the shifted distributions cross the policy’s action-selection boundary at decision-critical frames. Figure 3: Policy-logit shifts across attacks on Atari Pong. Dark bars mark the selected actions. 5.4 Ablation Study We ablate each component of CIVA on DMC walker walk; Tables 3 and 4 report effectiveness and stealth. From the full model (V0), each variant changes one component: V1 replaces the critic-induced subspace with a random orthonormal basis of the same rank; V2 removes the subspace and uses full-pixel PGD; V3 removes the EMA (ρ=0ρ=0); V4 replaces the critic-value surrogate with the victim’s reward-prediction-head gradient. Table 3: Ablation: reward drop on DMC walker walk. The Online and EMA columns indicate whether subspace-restricted online PGD and temporal EMA are enabled. ID Variant (A) Signal (B) Subspace Online EMA EpReward ↓ Drop% ↑ V0 CIVA (full) critic critic-SVD ✓ ✓ 703.97 26.07% V1 w/o critic-induced subspace critic random ✓ ✓ 886.54 6.90% V2 w/o subspace constraint (full-pixel) critic — ✗ ✗ 832.21 12.61% V3 w/o temporal EMA critic critic-SVD ✓ ✗ 696.66 26.84% V4 w/o critic (reward-grad subspace) reward reward-SVD ✓ ✓ 906.13 4.84% Table 4: Ablation: behavioural deviation and stealth (DMC walker walk). Companion to Table 3. ID Variant (A) Signal (B) Subspace Online EMA Action-KL ↑ TempAbs ↓ SSIM ↑ V0 CIVA (full) critic critic-SVD ✓ ✓ 1.130 0.646 0.710 V1 w/o critic-induced subspace critic random ✓ ✓ 0.058 0.8439 0.612 V2 w/o subspace constraint (full-pixel) critic — ✗ ✗ 0.730 3.2350 0.908 V3 w/o temporal EMA critic critic-SVD ✓ ✗ 0.798 1.7496 0.675 V4 w/o critic (reward-grad subspace) reward reward-SVD ✓ ✓ 1.054 0.542 0.833 Critic-induced subspace. A random basis of the same rank (V1) drops effectiveness from 26.07%26.07\% to 6.90%6.90\% and KL from 1.1301.130 to 0.0580.058. Full-pixel critic-PGD (V2) is also weak (12.61%12.61\% drop) and inflates TempAbs to 3.2353.235, the largest in the table. Low rank alone is not enough, and critic guidance alone is not enough either; the gain comes from aligning a low-rank basis with critic-sensitive directions. Temporal EMA. Disabling the EMA (V3) gives a marginal gain in effectiveness (+0.77+0.77 p) but nearly triples TempAbs (1.74961.7496 vs. 0.6460.646) and lowers SSIM by 3.53.5 points. Since CIVA targets the joint optimum of effectiveness and stealth, we keep the EMA in V0: the small reward-drop loss is outweighed by the much smoother and more imperceptible perturbation sequence. Critic value vs. reward head. Replacing the critic-value gradient with the reward-head gradient (V4) drops effectiveness to 4.84%4.84\%, the weakest variant, and Action-KL also falls below V0 (1.0541.054 vs. 1.1301.130). Although V4’s stealth indicators (TempAbs 0.5420.542, SSIM 0.8330.833) look favourable, this is precisely because the reward-head signal points along directions that move the world model’s one-step predicted reward without affecting the long-horizon return that drives the policy: the perturbations are easy to make smooth and visually subtle, but no longer functionally adversarial. Taking effectiveness, behavioural deviation, and stealth together, V0 (full CIVA) gives the best overall trade-off among all variants. 6 Conclusion This paper proposes CIVA, a critic-induced value-subspace attack for visual world-model agents under tight online and ℓ∞ _∞ constraints. CIVA extracts a low-dimensional attack subspace from the frozen DreamerV3 critic and optimizes only subspace coefficients online with EMA smoothing. Across diverse visual control tasks, CIVA consistently yields stronger performance degradation than prior attacks and highlights the value landscape as a realistic adversarial surface of recurrent world-model agents. Limitations. CIVA targets the white-box online setting and needs access to the victim critic and clean rollouts; our evaluation uses a single backbone (DreamerV3). Full discussion is in Appendix A.5. Acknowledgments and Disclosure of Funding References [1] E. Alonso, A. Jelley, V. Micheli, A. Kanervisto, A. Storkey, T. Pearce, and F. Fleuret (2024) Diffusion for world modeling: visual details matter in Atari. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2. [2] F. Bai, R. Liu, Y. Du, Y. Wen, and Y. Yang (2025) Rat: adversarial attacks on deep reinforcement agents for targeted behaviors. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 15453–15461. Cited by: §1, §2. [3] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling (2013) The arcade learning environment: an evaluation platform for general agents. Journal of Artificial Intelligence Research 47, p. 253–279. Cited by: NeurIPS Paper Checklist, §5.1. [4] J. Bruce, M. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, Y. Aytar, S. Bechtle, F. Behbahani, S. Chan, N. Heess, L. Gonzalez, S. Osindero, S. Ozair, S. Reed, J. Zhang, K. Zolna, J. Clune, N. de Freitas, S. Singh, and T. Rocktäschel (2024) Genie: generative interactive environments. In International Conference on Machine Learning (ICML), Cited by: §1, §2. [5] N. Carlini and D. Wagner (2017) Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy (SP), Cited by: §1, §2. [6] J. Ding, Y. Zhang, Y. Shang, Y. Zhang, Z. Zong, J. Feng, Y. Yuan, H. Su, N. Li, N. Sukiennik, F. Xu, and Y. Li (2024) Understanding world or predicting future? a comprehensive survey of world models. arXiv preprint arXiv:2411.14499. Cited by: §1, §2. [7] T. Duan, Z. Zhang, Z. Lin, Y. Gao, L. Xiong, Y. Cui, H. Liang, X. Chen, H. Cui, and D. Huang (2025) Rethinking adversarial attacks in reinforcement learning from policy distribution perspective. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 1–5. Cited by: §A.4.5, Table 5, §1, §1, §2, §2, §5.1, §5.1, Table 1, Table 2. [8] C. Eckart and G. Young (1936) The approximation of one matrix by another of lower rank. Psychometrika 1 (3), p. 211–218. Cited by: §4.1. [9] T. Franzmeyer, S. McAleer, J. F. Henriques, J. N. Foerster, P. H. S. Torr, P. Kohli, and P. Maini (2024) Illusory attacks: information-theoretic detectability matters in adversarial attacks. In International Conference on Learning Representations (ICLR), Cited by: §A.4.4, Table 5, §1, §1, §2, §5.1, Table 1, Table 2. [10] A. Gleave, M. Dennis, C. Wild, N. Kant, S. Levine, and S. Russell (2020) Adversarial policies: attacking deep reinforcement learning. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. [11] I. J. Goodfellow, J. Shlens, and C. Szegedy (2015) Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. [12] C. Guo, J. S. Frank, and K. Q. Weinberger (2019) Low frequency adversarial perturbation. In Conference on Uncertainty in Artificial Intelligence (UAI), Cited by: §1, §2, §4. [13] Z. Guo, S. Liang, A. Balogh, N. Lunberry, R. Tu, M. Jelasity, and D. Tao (2026) When world models dream wrong: physical-conditioned adversarial attacks against world models. arXiv preprint arXiv:2602.18739. Cited by: §1, §2. [14] D. Ha and J. Schmidhuber (2018) Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2. [15] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi (2019) Dream to control: learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603. Cited by: §1, §2. [16] D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba (2020) Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193. Cited by: §1, §2. [17] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap (2023) Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104. Cited by: §A.4.6, NeurIPS Paper Checklist, §1, §2, §3.1, §4.1, §5.1. [18] D. Hafner (2021) Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.06780. Cited by: §A.2, NeurIPS Paper Checklist, §5.1. [19] N. Hansen, H. Su, and X. Wang (2024) TD-MPC2: scalable, robust world models for continuous control. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. [20] B. He, X. Jia, S. Liang, T. Lou, Y. Liu, and X. Cao (2023) SA-attack: improving adversarial transferability of vision-language pre-training models via self-augmentation. arXiv preprint arXiv:2312.04913. Cited by: §1, §2. [21] S. Huang, N. Papernot, I. Goodfellow, Y. Duan, and P. Abbeel (2017) Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02284. Cited by: §1, §2. [22] A. Ilyas, L. Engstrom, A. Athalye, and J. Lin (2018) Black-box adversarial attacks with limited queries and information. In International Conference on Machine Learning (ICML), Cited by: §4.2. [23] Y. Jing, S. Hong, and D. Tao (2024) Deep graph mating. Advances in Neural Information Processing Systems 37, p. 9753–9772. Cited by: §1, §2. [24] D. Kong, S. Liang, X. Zhu, Y. Zhong, and W. Ren (2024) Patch is enough: naturalistic adversarial patch against vision-language pre-training models. Visual Intelligence 2 (1), p. 1–10. Cited by: §1, §2. [25] E. Korkmaz (2023) Adversarial robust deep reinforcement learning requires redefining robustness. In Proceedings of the AAAI Conference on Artificial Intelligence, Cited by: §1, §2. [26] J. Kos and D. Song (2017) Delving into adversarial attacks on deep policies. In International Conference on Learning Representations (ICLR) Workshop, Cited by: §1, §2. [27] S. Kullback and R. A. Leibler (1951) On information and sufficiency. Annals of Mathematical Statistics 22 (1), p. 79–86. Cited by: 3rd item, §5.1. [28] S. Li, A. Neupane, S. Paul, C. Song, S. V. Krishnamurthy, A. K. Roy-Chowdhury, and A. Swami (2019) Adversarial perturbations against real-time video classification systems. In Network and Distributed System Security Symposium (NDSS), Cited by: 4th item, §2, §5.1. [29] S. Liang, L. Li, Y. Fan, X. Jia, J. Li, B. Wu, and X. Cao (2022) A large-scale multiple-objective method for black-box attack against object detection. In European Conference on Computer Vision, Cited by: §1. [30] S. Liang, X. Wei, and X. Cao (2021) Generate more imperceptible adversarial examples for object detection. In ICML 2021 Workshop on Adversarial Machine Learning, Cited by: §1. [31] S. Liang, X. Wei, S. Yao, and X. Cao (2020) Efficient adversarial attacks for visual object tracking. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVI 16, Cited by: §1. [32] S. Liang, B. Wu, Y. Fan, X. Wei, and X. Cao (2022) Parallel rectangle flip attack: a query-based black-box attack against object detection. arXiv preprint arXiv:2201.08970. Cited by: §1. [33] Y. Liang, Y. Sun, R. Zheng, and F. Huang (2022) Efficient adversarial training without attacking: worst-case-aware robust reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. [34] Y. Liang, Y. Sun, R. Zheng, X. Liu, B. Eysenbach, T. Sandholm, F. Huang, and S. McAleer (2024) Game-theoretic robust reinforcement learning handles temporally-coupled perturbations. In International Conference on Learning Representations (ICLR), Cited by: §2. [35] Y. Lin, Z. Hong, Y. Liao, M. Shih, M. Liu, and M. Sun (2017) Tactics of adversarial attack on deep reinforcement learning agents. In International Joint Conference on Artificial Intelligence (IJCAI), Cited by: §1, §2. [36] A. Liu, J. Guo, J. Wang, S. Liang, R. Tao, W. Zhou, C. Liu, X. Liu, and D. Tao (2023) X-Adv: physical adversarial object attacks against x-ray prohibited item detection. In 32nd USENIX Security Symposium (USENIX Security 23), Cited by: §1. [37] J. Liu, S. Zhu, S. Liang, J. Zhang, H. Fang, W. Zhang, and E. Chang (2023) Improving adversarial transferability by stable diffusion. arXiv preprint arXiv:2311.11017. Cited by: §1, §2. [38] S. Liu, J. Song, Y. Zhou, N. Yu, K. Chen, Z. Feng, and M. Song (2024) Interaction pattern disentangling for multi-agent reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (12), p. 8157–8172. Cited by: 1st item, §2. [39] S. Liu, Y. Zhou, J. Song, T. Zheng, K. Chen, T. Zhu, Z. Feng, and M. Song (2023) Contrastive identity-aware learning for multi-agent value decomposition. In Proceedings of the AAAI conference on artificial intelligence, Vol. 37, p. 11595–11603. Cited by: 1st item, §2. [40] T. Lou, X. Jia, J. Gu, L. Liu, S. Liang, B. He, and X. Cao (2024) Hide in thicket: generating imperceptible and rational adversarial perturbations on 3d point clouds. arXiv preprint arXiv:2403.05247. Cited by: §1, §2. [41] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu (2018) Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), Cited by: §1, §1, §2, §4.1. [42] V. Micheli, E. Alonso, and F. Fleuret (2023) Transformers are sample-efficient world models. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. [43] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis (2015) Human-level control through deep reinforcement learning. Nature 518 (7540), p. 529–533. Cited by: §5.1. [44] S. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard (2017) Universal adversarial perturbations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §1, §2. [45] S. Moosavi-Dezfooli, A. Fawzi, and P. Frossard (2016) DeepFool: a simple and accurate method to fool deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §2. [46] T. Oikarinen, W. Zhang, A. Megretski, L. Daniel, and T. Weng (2021) Robust deep reinforcement learning through adversarial loss. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. [47] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami (2016) The limitations of deep learning in adversarial settings. In IEEE European Symposium on Security and Privacy (EuroS&P), Cited by: §1, §2. [48] A. Pattanaik, Z. Tang, S. Liu, G. Bommannan, and G. Chowdhary (2018) Robust deep reinforcement learning with adversarial attacks. In International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), Cited by: 2nd item, §1, §2, §5.1. [49] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta (2017) Robust adversarial reinforcement learning. In International Conference on Machine Learning (ICML), Cited by: §2. [50] B. T. Polyak and A. B. Juditsky (1992) Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization 30 (4), p. 838–855. Cited by: §4.2. [51] J. Robine, M. Hoftmann, T. Uelwer, and S. Harmeling (2023) Transformer-based world models are happy with 100k interactions. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. [52] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al. (2020) Mastering Atari, Go, chess and shogi by planning with a learned model. Nature 588 (7839), p. 604–609. Cited by: §1, §2. [53] Y. Sun, R. Zheng, Y. Liang, and F. Huang (2022) Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL. In International Conference on Learning Representations (ICLR), Cited by: §A.4.3, Table 5, §1, §1, §2, §2, §5.1, §5.1, Table 1, Table 2. [54] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus (2014) Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), Cited by: §1, §2. [55] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. (2018) Deepmind control suite. arXiv preprint arXiv:1801.00690. Cited by: NeurIPS Paper Checklist, §1, §5.1. [56] B. G. Tekgul, S. Wang, S. Marchal, and N. Asokan (2022) Real-time adversarial perturbations against deep reinforcement learning policies: attacks and defenses. In European Symposium on Research in Computer Security, p. 384–404. Cited by: §A.4.2, Table 5, §1, §1, §2, §2, §4.1, §5.1, Table 1, Table 2. [57] T. Tieleman and G. Hinton (2012) Lecture 6.5—RMSProp: divide the gradient by a running average of its recent magnitude. Note: COURSERA: Neural Networks for Machine Learning Cited by: §4.1. [58] J. Wang, H. Qiu, Y. Rong, H. Ye, Q. Li, Z. Li, and C. Zhang (2022) BET: black-box efficient testing for convolutional neural networks. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, p. 164–175. Cited by: §1. [59] J. Wang, Y. Wu, W. Xu, Y. Huang, C. Zhang, Z. Li, M. Xu, and Z. Liang (2025) Your scale factors are my weapon: targeted bit-flip attacks on vision transformers via scale factor manipulation. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 20103–20112. Cited by: §1. [60] L. Wang, T. Zhang, Y. Qu, S. Liang, Y. Chen, A. Liu, X. Liu, and D. Tao (2025) Black-box adversarial attack on vision language models for autonomous driving. arXiv preprint arXiv:2501.13563. Cited by: §1, §2. [61] Z. Wang, Z. Zhang, S. Liang, and X. Wang (2023) Diversifying the high-level features for better adversarial transferability. arXiv preprint arXiv:2304.10136. Cited by: §1, §2. [62] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli (2004) Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13 (4), p. 600–612. Cited by: 5th item, §5.1. [63] X. Wei, S. Liang, N. Chen, and X. Cao (2018) Transferable adversarial attacks for image and video object detection. arXiv preprint arXiv:1811.12641. Cited by: §1. [64] X. Wei, J. Zhu, S. Yuan, and H. Su (2019) Sparse adversarial perturbations for videos. In Proceedings of the AAAI Conference on Artificial Intelligence, Cited by: 4th item, §2, §5.1. [65] P. Wu, A. Escontrela, D. Hafner, K. Goldberg, and P. Abbeel (2022) DayDreamer: world models for physical robot learning. In Conference on Robot Learning (CoRL), Cited by: §1, §2. [66] S. Xu, S. Liang, H. Zheng, Y. Luo, H. Hu, L. Zhang, and D. Tao (2026) CtrlAttack: a unified attack on world-model control in diffusion models. arXiv preprint arXiv:2603.13435. Cited by: §1, §2. [67] Z. Yan, Y. Guo, and C. Zhang (2019) Subspace attack: exploiting promising subspaces for query-efficient black-box attacks. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2, §4. [68] H. Zhang, H. Chen, D. Boning, and C. Hsieh (2021) Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint arXiv:2101.08452. Cited by: §1, §2, §2. [69] H. Zhang, H. Chen, C. Xiao, B. Li, M. Liu, D. Boning, and C. Hsieh (2020) Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in neural information processing systems 33, p. 21024–21037. Cited by: 2nd item, §A.4.1, Table 5, §1, §1, §2, §5.1, §5.1, Table 1, Table 2. [70] T. Zhang, L. Wang, X. Zhang, Y. Zhang, B. Jia, S. Liang, S. Hu, Q. Fu, A. Liu, and X. Liu (2024) Visual adversarial attack on vision-language models for autonomous driving. arXiv preprint arXiv:2411.18275. Cited by: §1, §2. [71] W. Zhang, G. Wang, J. Sun, Y. Yuan, and G. Huang (2023) STORM: efficient stochastic transformer based world models for reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §2. [72] H. Zhao, W. Zhou, D. Chen, T. Wei, W. Zhang, and N. Yu (2021) Multi-attentional deepfake detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 2185–2194. Cited by: §1. Appendix A Supplementary Material A.1 Detailed Experimental Setup This appendix expands the compressed description of the experimental pipeline in Section 5.1 into the level of detail required for full reproducibility, covering (i) the software and hardware environment, (i) how the DreamerV3 victim checkpoint is trained for each task, and (i) the attack-evaluation pipeline together with the formal definition of every metric reported in the main paper. Software and hardware environment. All DreamerV3 training and CIVA attack experiments are run on a single NVIDIA RTX PRO 6000 Blackwell GPU with 96 GB of memory. Because the online attack only optimises a low-dimensional coefficient vector αt∈ℝr _t ^r with r=32r=32, its memory footprint is substantially smaller than that of training and can co-reside with the frozen victim on the same device. The DreamerV3 training stack is built on JAX, Flax and Optax, while the attack scripts additionally use NumPy and scikit-image for metric computation. DreamerV3 victim training. CIVA attacks a DreamerV3 agent whose parameters are fully frozen at evaluation time, so a victim checkpoint must first be trained independently for each task. We reuse the official DreamerV3 training implementation without modifying the model architecture, the loss formulation, or any hyperparameter; different tasks simply use different training configurations: • DMC walker walk (continuous control): 1.1×1061.1×10^6 training steps, train_ratio=256, vision-only 64×64×364×64×3 observations with proprioception disabled. • Atari Pong (discrete control): 5.1×1075.1×10^7 training steps, train_ratio=32, with the default DreamerV3 image preprocessing (frame skipping, sticky actions, etc.). • Crafter (open-ended survival): 1.1×1061.1×10^6 training steps, train_ratio=512, envs=1, 64×64×364×64×3 observations. The trained weights, configuration, and training curves are saved for each task. The attack scripts load the checkpoint and keep all victim parameters frozen during evaluation; only critic forward passes and gradients are used to optimise the perturbation. Attack evaluation pipeline. Given a frozen victim, CIVA proceeds in two stages, exactly as in Section 5.1, but the per-stage operations are described in more detail here: 1. Stage A — offline subspace discovery. The frozen victim is rolled out on the target task with a 200200-step warm-up that lets the RSSM latent state stabilise. Critic-induced PGD then produces M=600M=600 per-frame perturbations; the flattened matrix Δ∈ℝM×D ^M× D with D=12,288D=12,288 is subjected to a truncated SVD, whose top r=32r=32 right singular vectors form the basis matrix Vr∈ℝD×rV_r ^D× r. The basis is saved to disk and kept fixed throughout evaluation. 2. Stage B — online low-dimensional PGD attack. A fresh episode is launched and the attack starts at t=t0=200t=t_0=200. At every attacked step the attacker performs L=8L=8 PGD updates on αt∈ℝr _t ^r with step size η=ε/4η= /4, smooths the result with an EMA of momentum ρ=0.75ρ=0.75, lifts it back to pixel space through VrV_r, and projects onto both ‖δt‖∞≤ε=24/255\| _t\|_∞\!≤\! =24/255 and the valid pixel range [0,255][0,255]. Each task is evaluated on five independent random seeds (one episode per seed); CIVA and all baselines share the same victim checkpoint, the same ε , and the same evaluation script to ensure a strict apples-to-apples comparison. Metric definitions. Let T be the length of an evaluation episode and t0=200t_0=200 the attack-start step. Denote the clean and attacked episodic returns by Rnat=∑t=1Trt(ot)R^nat= _t=1^Tr_t(o_t) and Ratk=∑t=1Trt(o~t)R^atk= _t=1^Tr_t( o_t), where ot∈[0,255]H×W×3o_t∈[0,255]^H× W× 3 is the clean observation, δt _t the per-frame perturbation, and o~t=clip(ot+δt,0,255) o_t=clip(o_t+ _t,0,255) the corresponding attacked observation. Each metric below is computed per seed and then averaged over the five seeds: • Episode Reward. The attacked episodic return, Ratk=∑t=1Trt(o~t)R^atk\,=\, _t=1^Tr_t( o_t). • Drop%. The relative reward drop with respect to the natural policy, following [48, 69]: Drop%=Rnat−RatkRnat×100%.Drop\%= R^nat-R^atkR^nat× 100\%. (10) • Action-KL. The mean Kullback–Leibler divergence [27] between the victim’s clean and attacked action distributions, averaged over the attacked window: Action-KL=1T−t0∑t=t0+1TKL(π(⋅∣ot)∥π(⋅∣o~t)).Action-KL= 1T-t_0 _t=t_0+1^TKL\! (π(· o_t)\,\|\,π(· o_t) ). (11) For DMC’s continuous policy we use the closed-form diagonal-Gaussian KL, and for the discrete policies on Pong / Crafter we apply the discrete KL on softmax-normalised logits. • TempAbs. The mean absolute frame-to-frame change of the perturbation, used to quantify temporal coherence (analogous to temporal-coherence measures for video adversarial attacks [64, 28]): TempAbs=1(T−t0−1)HWC∑t=t0+1T−1‖δt+1−δt‖1.TempAbs= 1(T-t_0-1)\,HWC _t=t_0+1^T-1 \| _t+1- _t \|_1. (12) Smaller values indicate that consecutive perturbations are closer, making temporal flicker harder to detect. • SSIM. The structural similarity index [62] between clean and attacked observations, averaged over the attacked window: SSIM=1T−t0∑t=t0+1TSSIM(ot,o~t),SSIM(x,y)=(2μxμy+c1)(2σxy+c2)(μx2+μy2+c1)(σx2+σy2+c2).SSIM= 1T-t_0 _t=t_0+1^TSSIM(o_t, o_t), (x,y)= (2 _x _y+c_1)(2 _xy+c_2)( _x^2+ _y^2+c_1)( _x^2+ _y^2+c_2). (13) We use the default scikit-image implementation with a data range of 255, computed per channel and averaged. A.2 Additional Results on Crafter Why we defer Crafter to the appendix. Crafter’s reward signal differs structurally from the dense per-step rewards used in DMC walker walk and Atari Pong. Crafter [18] returns reward only when the agent unlocks one of a fixed set of achievements, so the episodic return is a sparse, integer-valued count rather than a smooth per-step quantity. As a result, the same metrics used in the main paper carry slightly different weight here: Drop% reflects how many achievement events the attacker prevents rather than a continuous performance gap, and Action-KL measures policy drift relative to a victim that already exhibits diverse exploratory behaviour on this benchmark. The visual-stealth metrics (TempAbs, SSIM) are computed exactly as in the main experiments and remain directly comparable across tasks. The full experimental setup, including the DreamerV3 victim training and the precise definition of every metric, is described in Appendix A.1. Table 5: Attack results on Crafter. Method Venue / Year Episode Reward ↓ Drop% ↑ Action-KL ↑ TempAbs ↓ SSIM ↑ Natural (no attack) — 10.10 — — — 1.000 MAD [69] NeurIPS 2020 8.10 19.80% 1.837 6.25 0.874 UAP-RL [56] ESORICS 2022 7.10 29.70% 0.310 0.00 0.527 PA-AD [53] ICLR 2022 8.10 19.80% 0.913 8.12 0.478 Illusory [9] ICLR 2024 8.10 19.80% 1.562 19.99 0.495 DAPGD [7] ICASSP 2025 9.10 9.90% 1.960 33.64 0.529 CIVA (Ours) 2026 7.10 29.70% 0.326 0.27 0.970 Attack effectiveness. Under the same budget, CIVA matches the strongest baseline in reward suppression: the episode reward drops from the natural 10.1010.10 to 7.107.10, yielding a Drop% of 29.70%29.70\% that ties with UAP-RL and is clearly above MAD, PA-AD, and Illusory (19.80%19.80\%) and far above DAPGD (9.90%9.90\%). Given that Crafter rewards are discrete and sparse, suppressing roughly 30%30\% of the achievement events is already a substantial attack effect. Behavioural deviation. CIVA reaches an Action-KL of 0.3260.326 on Crafter, which is not the largest in the table (DAPGD attains 1.9601.960 and MAD 1.8371.837). This does not contradict the trend on DMC and Pong where larger Action-KL coincides with stronger attacks: DAPGD pushes the action distribution the furthest yet only achieves a 9.90%9.90\% reward drop, indicating that its perturbations are injected into directions that are largely orthogonal to long-horizon value. By construction, CIVA confines its perturbations to the critic-induced subspace and therefore concentrates the policy shift on directions that are aligned with long-horizon return. Even though the resulting distributional change appears milder, it is more effective at suppressing achievement unlocks, reaffirming the observation that “larger KL does not imply a stronger attack” also holds in the sparse-reward regime. Temporal coherence and visual stealth. CIVA dominates both stealth metrics. Its TempAbs of 0.270.27 is the second lowest among all methods. UAP-RL achieves a TempAbs of 0.000.00 because it uses a fixed universal perturbation that never changes across frames. CIVA follows closely while remaining fully adaptive per frame, and improves over PA-AD (8.128.12), Illusory (19.9919.99), and DAPGD (33.6433.64) by one to two orders of magnitude. On SSIM, CIVA reaches 0.9700.970, which is at least 0.090.09 higher than every baseline and far above UAP-RL (0.5270.527) and MAD (0.8740.874). Combined with the perturbation visualisation in Appendix A.3 (Figure 4), this confirms that CIVA’s perturbations are spatially concentrated and visually close to the clean observations. A.3 Perturbation Visualization Figure 4 visualizes the perturbation magnitude maps on six representative Crafter frames for CIVA and three baselines. Each row corresponds to one attack method; brighter colors indicate larger perturbation intensity. The contrast is consistent with the quantitative results in Table 5: compared with MAD, PA-AD, and Illusory, CIVA concentrates its perturbation on a smaller set of task-relevant regions, producing more spatially focused patterns with visibly less scattered high-intensity responses across the image. This spatial concentration helps explain why CIVA maintains the strongest visual stealth on Crafter while matching the largest reward drop. Figure 4: Perturbation visualization on Crafter. A.4 From Full-Space Attacks to Value-Aligned Subspaces This appendix collects the theoretical formulation of the five baselines and our method. We first state a unified threat model. We then derive each baseline’s attack objective and present its algorithm. Finally, we give the formulation of CIVA. Unified threat model. Let πθ(⋅∣st) _θ(· s_t) be the victim policy and vψ(st)v_ψ(s_t) be its critic. The latent state sts_t is produced by the recurrent encoder from past observations. At each step t, the attacker observes the clean frame ot∈[0,255]H×W×Co_t∈[0,255]^H× W× C and outputs a perturbation δt _t under the ℓ∞ _∞ budget ‖δt‖∞≤ε\| _t\|_∞≤ . The victim then receives o~t=clip(ot+δt,0,255) o_t=clip(o_t+ _t,0,255). We write Πℬε _B_ for the projection onto the ℓ∞ _∞ ball of radius ε and clip[0,255]clip_[0,255] for the pixel-range projection. All baselines and CIVA share this setting. A.4.1 MAD: Maximum-Action-Deviation Attack MAD [69] maximises the divergence between the clean and the attacked action distributions. The per-frame objective is ℒMAD(δ;ot)=KL(πθ(⋅∣ot)∥πθ(⋅∣ot+δ)).L_MAD(δ;o_t)\;=\;KL\! ( _θ(· o_t)\,\|\, _θ(· o_t+δ) ). (14) For a discrete policy this is the categorical KL on the softmax outputs. For a Gaussian policy it has the closed-form KL((μ1,σ1)∥(μ2,σ2))=12[logσ22σ12+σ12+(μ1−μ2)2σ22−1].KL(N( _1, _1)\,\|\,N( _2, _2))\;=\; 12\! [\, _2^2 _1^2+ _1^2+( _1- _2)^2 _2^2-1 ]. (15) The attacker applies L steps of ℓ∞ _∞ PGD on Eq. (14) per frame. Algorithm 1 MAD attack 1: victim policy πθ _θ, frame oto_t, budget ε , steps L, step size η 2: Initialise δ←0δ← 0 3: for ℓ=1,…,L =1,…,L do 4: g←∇δKL(πθ(⋅∣ot)∥πθ(⋅∣ot+δ))g← _δ\,KL\! ( _θ(· o_t)\,\|\, _θ(· o_t+δ) ) 5: δ←Πℬε(δ+η⋅sign(g))δ← _B_ \! (δ+η·sign(g) ) 6: end for 7: return δt←clip[0,255](ot+δ)−ot _t _[0,255](o_t+δ)-o_t A.4.2 UAP-RL: Universal Adversarial Perturbation for RL UAP-RL [56] learns a single perturbation δu∈ℝDδ^u ^D that is reused on every frame. The perturbation is trained offline on a buffer D of clean observations: δu=argmax‖δ‖∞≤εo∼[ℒ(δ,o)],δ^u\;=\; *arg\,max_\|δ\|_∞≤ \;E_o \! [\,L(δ;o) ], (16) where ℒL is a policy-deviation loss such as the KL in Eq. (14). At test time the same δuδ^u is applied to every frame, so the temporal variation is zero by construction. Algorithm 2 UAP-RL training 1: buffer D, budget ε , epochs E, batch size B, step size η 2: Initialise δu←0δ^u← 0 3: for e=1,…,Ee=1,…,E do 4: for each minibatch oii=1B⊂\o_i\_i=1^B do 5: g←1B∑i=1B∇δℒ(δu,oi)g← 1B _i=1^B _δ\,L(δ^u;o_i) 6: δu←Πℬε(δu+η⋅sign(g))δ^u← _B_ \! (δ^u+η·sign(g) ) 7: end for 8: end for 9: return δuδ^u A.4.3 PA-AD: Policy-Aware Adversarial Attack PA-AD [53] treats the attacker as a learned policy πϕadv(δ∣ot) _φ^adv(δ o_t). The attacker is trained by RL to minimise the victim’s return. Let τ=(o1,δ1,…,oT,δT)τ=(o_1, _1,…,o_T, _T) be the joint rollout and rtr_t be the victim reward. The attacker’s objective is maxϕτ∼πϕadv,πθ[−∑t=1Trt]s.t.∥δt∥∞≤ε. _φ\;E_τ _φ^adv, _θ\! [\,- _t=1^Tr_t ] .t. \| _t\|_∞≤ . (17) A standard policy-gradient estimator gives ∇ϕ(ϕ)=[∑t=1T∇ϕlogπϕadv(δt∣ot)⋅A^t], _φJ(φ)\;=\;E\! [ _t=1^T _φ _φ^adv( _t o_t)· A_t ], (18) where A^t A_t is an advantage estimate based on the negated victim reward. Algorithm 3 PA-AD attacker training 1: victim πθ _θ, attacker πϕadv _φ^adv, budget ε , iterations I 2: for i=1,…,Ii=1,…,I do 3: Roll out joint trajectory τ with δt∼πϕadv(⋅∣ot) _t _φ^adv(· o_t), projected to ℬεB_ 4: Compute attacker reward rtadv=−rtr^adv_t=-r_t and advantages A^t A_t 5: Update ϕφ by Eq. (18) with PPO/A2C 6: end for 7: return πϕadv _φ^adv A.4.4 Illusory: Detectability-Aware Attack Illusory [9] adds a stealth term so that the attacked observation stays close to a plausible clean prediction o^t o_t (e.g. produced by a learned dynamics model). The objective is ℒIll(δ;ot)=−a∼πθ(⋅∣ot+δ)[Q(ot,a)]+λ⋅(ot+δ,o^t),L_Ill(δ;o_t)\;=\;-E_a _θ(· o_t+δ)[\,Q(o_t,a)\,]\;+\;λ·D(o_t+δ, o_t), (19) where the first term degrades the action value and the second term penalises detectable deviation. D is a perceptual distance (e.g. ℓ2 _2 or LPIPS) and λ>0λ>0 trades off attack strength against stealth. Algorithm 4 Illusory attack 1: victim πθ,Q _θ,Q, frame oto_t, predictor o^t o_t, budget ε , steps L, step size η, weight λ 2: Initialise δ←0δ← 0 3: for ℓ=1,…,L =1,…,L do 4: g←∇δℒIll(δ,ot)g← _δL_Ill(δ;o_t) ⊳ Eq. (19) 5: δ←Πℬε(δ−η⋅sign(g))δ← _B_ \! (δ-η·sign(g) ) 6: end for 7: return δt←clip[0,255](ot+δ)−ot _t _[0,255](o_t+δ)-o_t A.4.5 DAPGD: Distribution-Aware Projected Gradient Descent DAPGD [7] replaces the point-wise KL with a distance between the full action distributions. For a discrete policy with logits ℓ(o)∈ℝ|| (o) ^|A|, DAPGD uses the Bhattacharyya distance ℒDAPGD(δ;ot)=−log∑a∈pθ(a∣ot)⋅pθ(a∣ot+δ),L_DAPGD(δ;o_t)\;=\;- \! _a \! p_θ(a o_t)· p_θ(a o_t+δ), (20) where pθ(a∣o)=softmax(ℓ(o))ap_θ(a o)=softmax( (o))_a. Maximising Eq. (20) pushes the two action distributions apart in a distributional sense rather than along a single mode. Algorithm 5 DAPGD attack 1: victim πθ _θ, frame oto_t, budget ε , steps L, step size η 2: Initialise δ←0δ← 0 3: for ℓ=1,…,L =1,…,L do 4: g←∇δℒDAPGD(δ,ot)g← _δ\,L_DAPGD(δ;o_t) ⊳ Eq. (20) 5: δ←Πℬε(δ+η⋅sign(g))δ← _B_ \! (δ+η·sign(g) ) 6: end for 7: return δt←clip[0,255](ot+δ)−ot _t _[0,255](o_t+δ)-o_t The five baselines all act in the full pixel space and treat each frame independently (UAP-RL is the only exception, using one shared perturbation). None of them exploits the victim critic’s gradient structure across frames. They therefore either lose temporal coherence (MAD, PA-AD, Illusory, DAPGD) or lose per-frame adaptivity (UAP-RL). CIVA addresses both issues at once by sharing a critic-induced subspace across frames and smoothing its coefficients in time. A.4.6 CIVA: Critic-Induced Value-Subspace Attack The main paper presents the algorithmic form of CIVA. Here we expand on the parts that the method section only states briefly. We discuss (i) why the categorical-bin surrogate is a valid attack signal, (i) why the critic-gradient matrix Δ is approximately low-rank, (i) what we lose by restricting the attack to rS_r, (iv) how the pixel-range and ℓ∞ _∞ projections interact with the subspace constraint, (v) a frequency-domain view of the EMA, and (vi) a recipe for choosing the rank r. (i) Validity of the categorical-bin surrogate. DreamerV3’s critic returns a categorical distribution p(k∣o)=softmax(ℓk(o))p(k o)=softmax( _k(o)) over K symlog-spaced bins [17]. The attacker does not need a calibrated value; any monotone surrogate of the predicted return suffices. Let bkb_k be the value of bin k (sorted so that b1<⋯<bKb_1<·s<b_K). The calibrated value is V(o)=∑kbkp(k∣o)V(o)= _kb_k\,p(k o), and our surrogate v(o)=∑kp(k∣o)v(o)= _kk\,p(k o) replaces bkb_k by k. Both are linear in p(⋅∣o)p(· o) with positive, increasing coefficients, so they share the same monotonicity in distributional sense: if p′p first-order stochastically dominates p, then V(p′)≥V(p)V(p )≥ V(p) and v(p′)≥v(p)v(p )≥ v(p). Therefore lowering v shifts probability mass toward lower-value bins, which is exactly the attacker’s goal. Using v instead of V avoids the symlog inverse and removes the bin-spacing constants from the gradient, which makes step sizes more stable across tasks with different return scales. (i) Why Δ is approximately low-rank. A natural concern is whether the SVD of Δ can really capture the attack directions when each row is computed independently. Two structural reasons explain the observed low rank. First, the rows of Δ are produced by gradients of the same critic network. Let f=∇δv(o+δ)|δ=0f= _δv(o+δ) |_δ=0 be the first-order critic gradient at frame o. By backpropagation, f=Jenc(o)⊤∇sv(s)f=J_enc(o) \, _sv(s), where JencJ_enc is the Jacobian of the visual encoder and ∇sv(s) _sv(s) is the gradient of the critic with respect to the latent state s∈ℝdss ^d_s. Since ds≪Dd_s D, the column space of Jenc⊤J_enc has rank at most dsd_s. The first-order critic gradient at any frame therefore lies in a dsd_s-dimensional subspace of ℝDR^D. Second, the rollout visits a slow time-varying region of observation space. The Jacobian Jenc(o)J_enc(o) changes smoothly with o, so neighbouring frames produce highly correlated gradients. PGD adds high-frequency components on top of f, but those components are bounded by the budget and do not span new directions on average. Empirically, the top r=32r=32 singular values explain over 95%95\% of the spectral energy of Δ on every task we tested, which matches the latent dimension dsd_s used by DreamerV3. Two conclusions follow: • the rank of Δ is governed by the latent dimension dsd_s, not the pixel dimension D; and • the SVD basis VrV_r is essentially a basis of the encoder’s pull-back of the critic gradient. This explains why a critic-induced subspace generalises across frames in a way that a PCA basis built from raw observations cannot: PCA captures appearance variance, while VrV_r captures value sensitivity. (i) Suboptimality bound of the subspace attack. Let δ⋆δ be the optimum of the per-frame full-pixel attack and δr⋆=VrVr⊤δ⋆δ _r=V_rV_r δ its projection onto rS_r. Assume v(o+⋅)v(o+·) is LvL_v-smooth in δ on the budget ball. By smoothness, v(o+δr⋆)−v(o+δ⋆)≤⟨∇δv(o+δ⋆),δr⋆−δ⋆⟩+Lv2‖δr⋆−δ⋆‖22.v(o+δ _r)-v(o+δ )\;≤\; _δv(o+δ ),\;δ _r-δ + L_v2\,\|δ _r-δ \|_2^2. (21) At a stationary point of the inner ℓ∞ _∞ problem the first-order term vanishes on active coordinates, and the residual is quadratic in the projection error ξ:=‖δ⋆−VrVr⊤δ⋆‖2ξ:=\|δ -V_rV_r δ \|_2. By Eckart–Young, ξ is bounded by the tail singular energy of Δ : ξ≤σr+1(Δ)+‖δ⋆−δ¯‖2,ξ\;≤\; _r+1( )\;+\;\|δ - δ\|_2, (22) where δ¯ δ is the closest row of Δ to δ⋆δ . So when the singular spectrum of Δ decays fast and the test frame is not too far from the probed distribution, the suboptimality of the subspace attack is O(Lvσr+1(Δ)2)O(L_v\, _r+1( )^2). The fast spectral decay observed empirically is therefore the formal reason that CIVA does not lose much per-frame attack strength. (iv) Pixel-range and ℓ∞ _∞ projection inside the subspace. The reparameterisation δ=Vrαδ=V_rα does not automatically respect the ℓ∞ _∞ ball or the pixel range. The exact constrained problem minα∈ℝrv(o+Vrα)s.t.‖Vrα‖∞≤ε,o+Vrα∈[0,255]D, _α ^r\;v(o+V_rα) .t. \|V_rα\|_∞≤ ,\;\;o+V_rα∈[0,255]^D, (23) has a non-axis-aligned feasible set in ℝrR^r, so its projection has no closed form. We use a cheap two-step alternation: lift α to δ=Vrαδ=V_rα, apply the closed-form projections Πℬε _B_ and clip[0,255]clip_[0,255] in pixel space, then push back via α←Vr⊤δα← V_r δ. This is a single iteration of Dykstra’s projection between the affine set rS_r and the box ℬε∩[0,255]B_ ∩[0,255]. The remaining residual δ−VrVr⊤δ-V_rV_r δ measures by how much the pixel-space constraints push the perturbation off rS_r. Empirically this residual is below 10−310^-3 in pixel scale at ε=24/255 =24/255, because the budget is small and the box almost never binds at training time. (v) Frequency-domain view of the EMA. The EMA in Eq. (9) is a first-order infinite-impulse filter on the per-frame solutions αt⋆\ _t \. Its discrete-time transfer function is H(z)=1−ρ1−ρz−1,H(z)\;=\; 1-ρ1-ρ z^-1, (24) which is low-pass with cutoff ωc≈−logρ _c≈- ρ (in radians per frame). At ρ=0.75ρ=0.75 the cutoff is roughly 0.290.29 rad/frame, so the EMA strongly suppresses frequency components above ∼5 5 frames. This is exactly the temporal scale of high-frequency PGD jitter that hurts the TempAbs metric. Two consequences follow. • The EMA only attenuates fast variations in the coefficients αt⋆ _t . Slow drifts—which carry the actual value-degrading signal—are passed through with unit gain in the limit. • Because VrV_r is shared across frames, filtering in coefficient space corresponds to filtering each pixel direction in rS_r with the same H(z)H(z). The pixel-space perturbation δt=Vrαt _t=V_r _t inherits the same low-pass property without leaving rS_r. Combined with the bound ‖δt−δt−1‖2≤(1−ρ)‖Vrαt⋆−δt−1‖2\| _t- _t-1\|_2≤(1-ρ)\|V_r _t - _t-1\|_2 already discussed in the main paper, this gives a clean way to trade TempAbs against per-frame attack strength by tuning a single scalar ρ. (vi) Choosing the rank r. A practical recipe is to pick r from the cumulative spectral energy of Δ : r⋆(τ)=minr:∑i=1rσi2(Δ)∑i=1Mσi2(Δ)≥τ,r (τ)\;=\; \! \r\;:\; _i=1^r _i^2( ) _i=1^M _i^2( )\;≥\;τ \, (25) with τ∈[0.9,0.99]τ∈[0.9,0.99]. On all three tasks, τ=0.95τ=0.95 gives r⋆∈[28,40]r ∈[28,40], which justifies the fixed choice r=32r=32 used in the main experiments. Two practical notes: • Increasing r beyond the knee of the spectrum brings near-zero improvement in attack strength but slightly increases the per-frame optimisation cost and the spatial complexity of the perturbation, which weakens SSIM and TempAbs. • Decreasing r below dsd_s is cheap but quickly truncates critic-aligned directions, which costs reward suppression. The spectrum-based rule above sits naturally between these two regimes and removes the need to tune r per task. Putting it together. Each design choice in CIVA matches one structural fact above. The categorical-bin surrogate (i) provides a stable attack signal with the same monotonicity as the calibrated value. The latent-state geometry (i) makes Δ approximately low-rank, so the SVD basis VrV_r captures most of the value-degrading directions. The smoothness bound (i) shows that restricting to rS_r costs only O(Lvσr+12)O(L_v _r+1^2) in attack strength. The two-step projection (iv) handles the pixel-range and ℓ∞ _∞ constraints with negligible residual. The low-pass property of the EMA (v) suppresses temporal jitter without filtering out the attack signal. The spectrum-based rank rule (vi) removes a hyperparameter without sacrificing performance. A.5 Limitations and Future Directions Our method assumes a white-box online attacker with access to the victim critic and its gradients, and therefore does not directly apply to strict black-box settings where only actions or rewards are observable. The offline subspace discovery stage further requires clean rollouts from the target agent, and the learned subspace may need to be refreshed if the deployment environment changes substantially. Our evaluation also covers a single world-model backbone, DreamerV3, and three representative visual control tasks spanning continuous control, discrete control, and open-ended world interaction; broader validation on additional world-model architectures, larger observation resolutions, and more diverse environments is left to future work. Along the same line, future work may study query-efficient or transfer-based variants of the proposed attack, adaptive subspace updates under distribution shift, and dedicated defenses against critic-aligned temporal perturbations. Broader impact. CIVA exposes a previously under-studied attack surface of visual world-model agents and could in principle be misused against deployed agents (e.g., visual policies for robotics or simulated control). On balance we believe the positive impact outweighs this risk: CIVA requires white-box access to the victim critic, which limits realistic misuse, and surfacing such failure modes is a prerequisite for designing robust world-model agents and certified defenses. We will release our code so that defense research can be reproduced on the same footing as the attack. NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and Section 1 list three contributions—a critic-induced value subspace, a two-stage offline-probe + online-subspace-PGD attack with temporal EMA, and a joint effectiveness/behaviour/stealth evaluation.The claims made match the theoretical and experimental results. Guidelines: • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: A dedicated Limitations paragraph in Section 6 states that CIVA assumes white-box access to the victim critic and clean rollouts, is evaluated on three visual benchmarks (DMC walker_walk, Atari Pong, Crafter) with a single victim backbone (DreamerV3), and a fuller discussion is given in Appendix A.5. Guidelines: • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate “Limitations” section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: The assumptions in the paper are clearly stated . Guidelines: • The answer [N/A] means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Section 5.1 fully specifies the victim (DreamerV3 with official per-task configuration) and our framework, and provide parameters. Guidelines: • The answer [N/A] means that the paper does not include experiments. • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We submit the code and data, and introduce the data processing method. . Guidelines: • The answer [N/A] means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://neurips.c/public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.c/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results? Answer: [Yes] Justification: Section 5.1 reports the victim configuration (DreamerV3 official per-task setting, frozen), the three benchmarks, all CIVA hyperparameters (ε , η, L, M, r, ρ, warm-up), the optimizer choice (per-frame projected gradient descent), and the 5-seed evaluation protocol. The ablations in Section 5.4 document how the key design choices (subspace, EMA, critic vs. reward head) were selected. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: As stated in Section 5.1, every reported number in Table is the mean across 55 independent random evaluation seeds (one rollout per seed); the seed-to-seed variation is much smaller than the gap between CIVA and the strongest baseline (e.g., 1919 percentage points of Drop% on Pong), so we report means in the main tables for readability. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates). • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Section 5.1 states that all attack and evaluation runs use a single NVIDIA RTX PRO 6000 Blackwell GPU (96 GB); a full evaluation of one method on one task completes within a few hours, and the offline subspace probing (Stage A) takes under one hour per task. Pilot tuning of CIVA hyperparameters used the same single GPU and consumed an additional compute amount comparable to the reported main experiments. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.c/public/EthicsGuidelines? Answer: [Yes] Justification:We comply with the NeurIPS Code of Ethics. Guidelines: • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer [No] , they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: A Broader impact paragraph in Appendix A.5 acknowledges that CIVA exposes a new attack surface of visual world-model agents and could be misused, but its white-box requirement limits realistic misuse; releasing the attack and the accompanying code on the same footing is intended to enable robust world-model agents and certified defenses. Guidelines: • The answer [N/A] means that there is no societal impact of the work performed. • If the authors answer [N/A] or [No] , they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [N/A] Justification: We did not use data or models that have a high risk of misuse. Guidelines: • The answer [N/A] means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All existing assets used in this work are publicly released open-source artefacts and are cited at first use: DreamerV3 [17] (MIT License), the DeepMind Control Suite [55] (Apache 2.0), the Arcade Learning Environment [3] (GPL-2.0), and Crafter [18] (MIT License); we use them in accordance with their respective licenses. Guidelines: • The answer [N/A] means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., C-BY 4.0) should be included for each asset. • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [N/A] Justification: The paper does not release new assets. Guidelines: • The answer [N/A] means that the paper does not release new assets. • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: The paper does not involve crowdsourcing or research with human subjects. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: The paper does not involve research with human subjects. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required. Answer: [N/A] Justification: LLMs are not used as any important, original, or non-standard component of the core method; their use, if any, is limited to incidental writing and editing assistance. Guidelines: • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.