Paper deep dive
Predictive Semantic Safety: From Visual Physical Reasoning to Safety-Critical Control
Taekyung Kim, Salem Fradi, Yanning Dai, Mateusz Ostaszewski, Jürgen Schmidhuber
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Physical interactions can create future hazards that are not apparent from the robot's current geometric surroundings. We present a framework termed Predictive Semantic Safety (PSS), which connects visual physical reasoning to backup-based safety filtering. A vision-language model (VLM) predicts physical events and their timing or directly predicts object displacements. An explicit motion model converts event hypotheses into object trajectories. Split conformal prediction calibrates position errors jointly across specified objects, observation times, and future times; geometric shape bounds convert the resulting position regions into predicted object occupancy. PSS evaluates a prescribed backup maneuver against this occupancy and derives input-affine constraints for minimally modifying the nominal input while preserving backup feasibility under the robot dynamics and input limits. MuJoCo experiments with a Unitree Go1 consider falling fixtures, impact-driven support loss, and contact propagation. PSS achieves a safe episode rate of 99.3%, compared with 43.3% for a Backup Control Barrier Function baseline that only uses current obstacle geometry.
Tags
Links
- Source: https://arxiv.org/abs/2609.34356v1
- Canonical: https://arxiv.org/abs/2609.34356v1
Trouble viewing inline? Open PDF directly →
Full Text
46,959 characters extracted from source content.
Expand or collapse full text
mysize Predictive Semantic Safety: From Visual Physical Reasoning to Safety-Critical Control Taekyung Kim Affiliation: Department of Robotics, University of Michigan, Ann Arbor, MI 48109, USA taekyung@umich.edu Salem Fradi Affiliation: Center of Excellence in Generative AI, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia. Yanning Dai Affiliation: Center of Excellence in Generative AI, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia. Mateusz Ostaszewski Affiliation: Center of Excellence in Generative AI, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia. Jürgen Schmidhuber †thanks: *These authors contributed equally to this work Affiliation: Center of Excellence in Generative AI, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia. Affiliation: Dalle Molle Institute for Artificial Intelligence Research (IDSIA), Switzerland. Affiliation: Università della Svizzera italiana (USI), Switzerland. Affiliation: Scuola universitaria professionale della Svizzera italiana (SUPSI), Switzerland. Abstract Physical interactions can create future hazards that are not apparent from the robot’s current geometric surroundings. We present a framework termed Predictive Semantic Safety (PSS), which connects visual physical reasoning to backup-based safety filtering. A vision-language model (VLM) predicts physical events and their timing or directly predicts object displacements. An explicit motion model converts event hypotheses into object trajectories. Split conformal prediction calibrates position errors jointly across specified objects, observation times, and future times; geometric shape bounds convert the resulting position regions into predicted object occupancy. PSS evaluates a prescribed backup maneuver against this occupancy and derives input-affine constraints for minimally modifying the nominal input while preserving backup feasibility under the robot dynamics and input limits. MuJoCo experiments with a Unitree Go1 consider falling fixtures, impact-driven support loss, and contact propagation. PSS achieves a safe episode rate of 99.3%, compared with 43.3% for a Backup Control Barrier Function baseline that only uses current obstacle geometry. [Project Page]11 1 Project page: https://w.taekyung.me/pss [Code] [Video] I INTRODUCTION Robots navigating homes, warehouses, and industrial facilities must account for hazards that develop as their surroundings change. Water spreading across a corridor can make the floor unsafe to traverse, a growing fire can block an exit, and an unstable stack of boxes can collapse into an otherwise clear passage. Assessing these situations requires reasoning beyond current obstacle geometry to anticipate how physical interactions may change the safety of the surrounding space. Consider the partially detached ceiling fixture in Fig. 1. Although the floor beneath it remains clear, an approaching robot may need to brake before the fixture falls. Waiting until the hazard enters its path may leave insufficient control authority to avoid collision. Safety therefore depends on anticipating physical changes and responding while a safe maneuver remains feasible. Visual physical reasoning provides predictions of how observed objects may move and interact. Physical-interaction benchmarks and learned dynamics models study this capability across a range of scenes and motion types [1, 2, 3, 4]. More recently, vision-language models (VLMs) have shown increasing capability to reason about physical events directly from visual observations. Code-as-World, for example, uses executable world representations to train VLMs for quantitative physical reasoning [5]. These advances make it possible to anticipate how a physical event may change the scene, but they do not directly specify how the robot should respond. A safety-critical controller must additionally account for the predicted object’s spatial extent, the timing and uncertainty of its predicted motion, and the robot’s dynamics and input limits [6]. Fig. 1: Motivating example of predictive semantic safety. A partially detached ceiling fixture remains above an apparently clear route, but its anticipated fall may leave the robot with insufficient control authority to avoid collision. To address this problem, we introduce a framework termed Predictive Semantic Safety (PSS), which converts predicted physical events into time-varying unsafe occupancy for backup-based safety filtering [7]. Semantic reasoning identifies physical mechanisms that may change object motion, such as support loss, impact, or contact propagation. Using Code-as-World-VL [5], we obtain object-motion predictions either directly or through an explicit motion model conditioned on an event hypothesis and its timing. Split conformal prediction calibrates the resulting position errors, and geometric shape bounds convert the position regions into predicted object occupancy (Fig. 2). Occupancy coverage alone does not establish that the robot retains a feasible response. Building on Backup Control Barrier Functions (CBFs) [8] and OcclusionCBF [9], PSS evaluates a prescribed backup maneuver against the predicted occupancy and requires its terminal state to admit a safe continuation. The resulting filter minimally modifies the nominal input subject to input-affine constraints that preserve backup feasibility under the robot dynamics and input limits. The filter can therefore intervene before predicted object motion obstructs the route, without imposing a particular nominal navigation policy. Fig. 2: Overview of PSS, illustrated for the ceiling fixture. RGB-D observations and measured motion are used to predict a physical event and its timing with a VLM. The event hypothesis and an explicit motion model determine future object motion, which is combined with geometric shape bounds and calibrated uncertainty to obtain unsafe occupancy. A Backup CBF safety filter evaluates a prescribed backup maneuver against this occupancy and modifies the nominal command when necessary. The VLM panel shows an excerpt from a recorded model output. MuJoCo trials are shown in Fig. 4. I-A Related Work Physical reasoning and semantic safety: Visual physical reasoning aims to predict how a scene will evolve from what a robot observes. Benchmarks test whether models can anticipate physical interactions and understand physical behavior [1, 4], while learned dynamics models predict how objects move and influence one another [2, 3]. Other approaches learn structured representations of the physical world from video for prediction and planning, including V-JEPA 2 [10], which realizes ideas introduced earlier in Predictability Maximization (PMAX) [11]. Code-as-World uses executable scene descriptions to train physical reasoning [5]. These advances provide a basis for anticipating hazards, but a forecast alone does not determine how the robot should respond safely. FORTRESS uses multimodal reasoning to anticipate failure modes and fallback goals, then plans a dynamics-aware fallback under inferred semantic constraints [12]. In contrast, PSS represents predicted object motion as time-varying occupancy sets and constrains the robot’s input to preserve the feasibility of a prescribed backup maneuver against that occupancy. Prediction uncertainty: Conformal prediction has been used to account for trajectory-prediction uncertainty in planning and MPC [13, 14, 15]. Most closely related, Stamouli et al. [16] construct simultaneous prediction regions across multiple agents, observation times, and future times for shrinking-horizon MPC. PSS adopts this calibration principle for object motion predicted from visually inferred physical events and combines the resulting position regions with geometric shape bounds to obtain predicted object occupancy for safety filtering. This calibration accounts for prediction uncertainty, while the backup formulation separately addresses whether the robot retains a feasible response. Predictive safety filtering: Standard CBFs construct safety constraints from the current state [17], while Backup CBFs construct them from a prescribed backup policy and its induced recoverable set [8]. Recent extensions evaluate multiple fallback policies through parallel finite-horizon rollouts [18]. Closest to our control formulation, OcclusionCBF evaluates a prescribed backup rollout against reachable occupancy of potentially hidden dynamic obstacles and a terminal condition [9]. PSS builds on this construction, but addresses changes in the motion of observed objects caused by visually inferred physical events. It calibrates the resulting position errors and evaluates the backup against the corresponding time-varying occupancy. I-B Contributions Our contributions are summarized as follows. • We introduce PSS, which converts visually inferred physical events and their timing into time-varying unsafe occupancy sets for safety-critical control. • We combine conformal calibration with backup-based safety filtering to derive input-affine occupancy and terminal constraints that preserve backup feasibility under the robot dynamics and input limits. • We evaluate PSS in MuJoCo quadruped navigation, demonstrating higher safe episode rates than both the CBF baselines using current obstacle geometry and the OmniVLA navigation policy. I PRELIMINARIES I-A Robot Dynamics Consider a robot with control-affine dynamics ˙=f()+g(),∈,∈, x=f( x)+g( x) u, x , u , (1) where ⊆ℝnX ^n is an open state domain, ⊂ℝmU ^m is compact and convex, and f and g are continuously differentiable. Let ⊆ℝdW ^d be the workspace. A continuously differentiable map P:→P:X gives the position =P() p=P( x) of a fixed robot reference point. A nominal controller nom:×ℝ≥0→ u_ nom:X×R_≥ 0 commands the robot toward its task objective but is not assumed to satisfy the safety constraints. It may be any task-achieving policy, including a reinforcement-learning or vision-language-action (VLA) policy. I-B Control Barrier Functions Let h:→ℝh:X be continuously differentiable and define =∈∣h()≥0C=\ x h( x)≥ 0\. The function h is a control barrier function if there exists a locally Lipschitz extended class-K function α:ℝ→ℝα:R such that [19] sup∈∇h()⊤(f()+g())≥−α(h()),∀∈. _ u ∇ h( x) (f( x)+g( x) u )≥-α (h( x) ), ∀ x . A locally Lipschitz controller satisfying the corresponding inequality renders C forward invariant for as long as the solution exists in X. However, h()≥0h( x)≥ 0 alone does not ensure feasibility of the barrier condition under the input bounds. Backup CBFs address this issue using a prescribed admissible backup policy. I-C Backup Control Barrier Functions For a time-invariant safe set C, let 0⊆S_0 be a terminal set that is forward invariant under a continuously differentiable state-feedback policy πb:→ _ b:X . For a backup horizon T>0T>0, the recoverable set induced by πb _ b contains states whose backup trajectories remain in C over [0,T][0,T] and reach 0S_0 at T [8, 7]. Its size depends on the choice of backup policy. The closed-loop backup dynamics are fb()=f()+g()πb()f_ b( x)=f( x)+g( x) _ b( x). Their flow φb(,s) _ b( x,s) satisfies dsφb(,s)=fb(φb(,s)),φb(,0)=, dds _ b( x,s)=f_ b ( _ b( x,s) ), _ b( x,0)= x, (2) where s≥0s≥ 0 is the rollout time. The sensitivity Jacobian Φb(,s):=∂φb(,s)/∂ _ b( x,s):=∂ _ b( x,s)/∂ x satisfies dsΦb(,s)=∂fb∂(φb(,s))Φb(,s),Φb(,0)=I. dds _ b( x,s)= ∂ f_ b∂ x ( _ b( x,s) ) _ b( x,s), _ b( x,0)=I. Since the backup dynamics are autonomous, Φb(,s)fb()=fb(φb(,s)). _ b( x,s)f_ b( x)=f_ b ( _ b( x,s) ). (3) The Backup CBF filter imposes separate safety conditions along the backup trajectory and at its terminal state. In the predictive setting considered here, the backup trajectory must be evaluated against obstacle occupancy at the corresponding future times, together with a terminal condition that ensures a safe continuation beyond the prediction horizon. I PROBLEM FORMULATION Consider planar navigation in ⊆ℝ2W ^2 for the robot in (1). Let J denote the finite set of objects included in the safety model, j(τ)⊆O_j(τ) the region occupied by object j at time τ, and j(τ)∈ p_j(τ) its reference position. Let ℛ()⊆R( x) denote the robot’s occupied geometry. RGB-D observations provide object geometry and motion history used for physical-event prediction. Assumption 1 (Object inventory and geometric bounds). The object set J is specified independently of the prediction output. For each j∈j , a known radius rj>0r_j>0 bounds the object’s planar extent about j(τ) p_j(τ) for all considered orientations. Assumption 2 (Exogenous object motion). Over the prediction horizon, the future motion of each j∈j is independent of the candidate robot input evaluated by the safety filter. The safety claim is limited to the objects in J; detecting previously unmodeled hazards is outside the scope of the formulation. Likewise, robot-induced interactions are not covered by Assumption 2 and would require action-conditioned predictions or conservative bounds on their effects. Problem 1 (Safety filtering with visual physical reasoning). Given the robot dynamics in (1), the nominal controller nom u_ nom, the prescribed backup policy introduced in Section I-C, and physical-event predictions for the objects in J, construct a safety filter that minimally modifies the nominal input while accounting for prediction uncertainty and preserving feasibility of the backup maneuver. Under Assumptions 1 and 2, the desired safety condition is dist(ℛ((τ)),j(τ))≥ϵ,∀j∈,∀τ,dist (R( x(τ)),O_j(τ) )≥ε, ∀ j ,\ ∀τ, (4) where ϵ>0ε>0 is the prescribed clearance. IV PREDICTIVE SEMANTIC SAFETY We address Problem 1 by predicting object motion, constructing calibrated occupancy sets, and evaluating the prescribed backup maneuver against them (Fig. 2). The filter modifies the nominal input to preserve backup feasibility. Let tkt_k denote the observation time and τ≥tkτ≥ t_k an absolute future time. We suppress the fixed tkt_k except when comparing predictions from different observation times. IV-A Visual Physical Reasoning Physical interactions can alter an object’s motion before the resulting hazard is apparent from current geometry. Loss of support can cause a suspended object to fall, while impact can set a stationary object in motion. We therefore use visual observations to infer how an object’s motion may change and when that change may occur, rather than predicting a semantic risk label alone. We use Code-as-World-VL [5], a pretrained VLM, for visual physical reasoning. Given a recent RGB clip and object motion history measured from RGB-D, a first query interprets the observed physical interaction. A second structured query uses the same observations and the first response to identify the object, its attachment state, the predicted motion mechanism, and its timing. For the ceiling fixture, the timing is represented by a detachment-time interval. We then use a motion model to convert this event hypothesis into metric object trajectories, representing the fixture as supported before detachment, falling after detachment, and stationary after ground contact. The predicted detachment-time interval is treated as uncertain. After verifying that the structured VLM output contains the required fields and valid numerical values, we sample candidate detachment times within the interval and generate the corresponding trajectories with the motion model. For the stack scenarios, the VLM instead predicts object displacement directly, without a separate detachment-time model. In both cases, the resulting motion predictions are calibrated in Section IV-C. IV-B Event-Conditioned Object Motion For object j∈j , let mjm_j be the predicted motion mechanism and Θj _j a nonempty finite set of motion hypotheses. Each θ∈Θjθ∈ _j specifies an event realization, such as a detachment time relative to tkt_k. Given the measured position and linear velocity j∈ℝ6 z_j ^6 at tkt_k, the motion model predicts ^j(tk+s,θ)=Ψmj(j,s,θ),θ∈Θj, q_j(t_k+s;θ)= _m_j( z_j,s;θ), θ∈ _j, (5) where s≥0s≥ 0 is the look-ahead time and ^j(tk+s,θ)∈ℝ3 q_j(t_k+s;θ) ^3. Let j(τ)∈ℝ3 q_j(τ) ^3 be the true object reference position and Π:ℝ3→ℝ2 :R^3 ^2 the planar projection, with j(τ)=Π(j(τ)) p_j(τ)= ( q_j(τ)). At τ=tk+sτ=t_k+s, define the mean planar prediction and maximum deviation across hypotheses as ^j(τ) p_j(τ) =1|Θj|∑θ∈ΘjΠ(^j(τ,θ)), = 1| _j| _θ∈ _j ( q_j(τ;θ) ), (6) dj(τ) d_j(τ) =maxθ∈Θj‖Π(^j(τ,θ))−^j(τ)‖2. = _θ∈ _j \| ( q_j(τ;θ) )- p_j(τ) \|_2. By Assumption 1, the disk centered at ^j(τ) p_j(τ) with radius rj+dj(τ)r_j+d_j(τ) contains the shape disk of each motion hypothesis. The spread djd_j does not account for shared prediction errors or unrepresented event realizations. For |Θj|=1| _j|=1, dj(τ)=0d_j(τ)=0. IV-C Conformal Occupancy Prediction We use split conformal prediction to calibrate the complete observation-to-motion prediction procedure [20, 16]. We restore the observation-time argument as ^j(τ,tk) p_j(τ;t_k) and dj(τ,tk)d_j(τ;t_k). Let ℐI be a fixed finite set of triples (j,k,τ)(j,k,τ), with j∈j and τ>tkτ>t_k, specifying the objects, observation times, and future times to be covered. The set ℐI is fixed independently of prediction outcomes. We split the data by complete trajectories into disjoint training and calibration sets. Each calibration example includes all observations and object motions over ℐI. The prediction procedure and positive finite scales σj(τ,tk) _j(τ;t_k) are fixed independently of the calibration data and remain unchanged during calibration and testing. The scales are computed from training data. For each calibration trajectory c=1,…,Ncalc=1,…,N_ cal, define the normalized nonconformity score Rc=max(j,k,τ)∈ℐ‖j(c)(τ)−^j(c)(τ,tk)‖2σj(τ,tk).R_c= _(j,k,τ) \| p_j^(c)(τ)- p_j^(c)(τ;t_k) \|_2 _j(τ;t_k). (7) The normalization accounts for differences in prediction-error scale across objects and horizons [21]. A missing or invalid required prediction gives Rc=+∞R_c=+∞. Fix the miscoverage level δ∈(0,1)δ∈(0,1) and suppress dependence on it. Define the conformal quantile q^≔Quantile1−δ(R1,…,RNcal,+∞), q _1-δ (R_1,…,R_N_ cal,+∞ ), (8) where Quantile1−δQuantile_1-δ denotes the ⌈(Ncal+1)(1−δ)⌉ (N_ cal+1)(1-δ) -th smallest element of its arguments. Let ℬ(r)=∈ℝ2∣‖2≤rB(r)=\ e ^2 \| e\|_2≤ r\ be the closed planar ball. With calibrated radius ρj(τ,tk)≔q^σj(τ,tk) _j(τ;t_k) q\, _j(τ;t_k), define j(τ,tk)≔^j(τ,tk)+ℬ(ρj(τ,tk)).T_j(τ;t_k) p_j(τ;t_k)+B ( _j(τ;t_k) ). (9) If the test prediction is invalid or q^=+∞ q=+∞, set j(τ,tk)=ℝ2T_j(τ;t_k)=R^2. By Assumption 1, j(τ)⊆j(τ)+ℬ(rj)O_j(τ) p_j(τ)+B(r_j). Combining this shape bound with the conformal position region and hypothesis spread gives the predicted object occupancy ^j(τ,tk)≔j(τ,tk)⊕ℬ(rj+dj(τ,tk)), O_j(τ;t_k) _j(τ;t_k) (r_j+d_j(τ;t_k) ), (10) where ⊕ denotes the Minkowski sum. If the required prediction is invalid or no motion hypothesis is available, we set ^j(τ,tk)=ℝ2 O_j(τ;t_k)=R^2. Proposition 1 (Simultaneous occupancy coverage). Suppose the prediction procedure and normalization scales are fixed independently of the calibration data, the calibration and test trajectories, including their observations, are exchangeable, and Assumption 1 holds over ℐI. Then ℙ(j(τ)⊆^j(τ;tk),∀(j,k,τ)∈ℐ)≥1−δ.P\! (O_j(τ) O_j(τ;t_k), ∀(j,k,τ) )≥ 1-δ. (11) Proof. The split-conformal rank argument applied to (7) gives j(τ)∈j(τ,tk) p_j(τ) _j(τ;t_k) simultaneously over ℐI with probability at least 1−δ1-δ. On this event, Assumption 1 and dj(τ,tk)≥0d_j(τ;t_k)≥ 0 imply j(τ)⊆j(τ,tk)⊕ℬ(rj)⊆^j(τ,tk)O_j(τ) _j(τ;t_k) (r_j) O_j(τ;t_k). ∎ The guarantee is marginal over exchangeable calibration and test trajectories, not conditional on a scene or an accepted update. Since (7) measures error from the mean prediction, djd_j does not justify reducing ρj _j. Moreover, Proposition 1 provides coverage only for the objects and times included in ℐI; it does not imply coverage between those times or feasibility of the robot’s response. IV-D Backup Rollout and Terminal Set Following OcclusionCBF [9], we require the prescribed backup to avoid the predicted occupancy and reach a terminal set that admits a safe continuation. Between accepted updates, fix the prediction and suppress tkt_k. For current time t and look-ahead time s≥0s≥ 0, let =φb(,s) y= _ b( x,s). Choose rR>0r_R>0 such that ℛ()⊆P()+ℬ(rR)R( y) P( y)+B(r_R) for all relevant states. Define the collision-inflated predicted occupancy ℋ^(j)(t,s)≔^j(t+s)⊕ℬ(rR+ϵ). H^(j)(t,s) O_j(t+s) (r_R+ε). (12) For a valid prediction with finite ρj _j, the disk representation in (10) gives the separation margin hjC(,t,s)=‖P()−^j(t+s)‖2−rR−rj−dj(t+s)−ρj(t+s)−ϵ. splith_j^C( y,t,s)=& \|P( y)- p_j(t+s) \|_2-r_R-r_j\\ &-d_j(t+s)- _j(t+s)-ε. split (13) Nonnegativity of hjCh_j^C places P()P( y) outside the interior of ℋ^(j)(t,s) H^(j)(t,s) and ensures clearance ϵε between ℛ()R( y) and ^j(t+s) O_j(t+s). The margin is differentiable in y when P()≠^j(t+s)P( y)≠ p_j(t+s). Its time dependence is through t+st+s, so ∂hjC/∂t=∂hjC/∂s∂ h_j^C/∂ t=∂ h_j^C/∂ s wherever the prediction is differentiable. Choose tft_f within the common prediction interval and define the remaining backup horizon T(t)≔tf−tT(t) t_f-t for t≤tft≤ t_f. The terminal time and backup policy are fixed between accepted updates. Let 0(τ)≔∈∣hjS(,τ)≥0,∀j∈S_0(τ) \ y h_j^S( y,τ)≥ 0,\ ∀ j \ for continuously differentiable terminal margins hjSh_j^S. Membership in 0(tf)S_0(t_f) must ensure separation at tft_f and a safe continuation under πb _ b thereafter. For each absolute future time τ∈[t,tf]τ∈[t,t_f], define the occupancy and terminal margins evaluated along the backup flow as ξjC(,t,τ) _j^C( x,t;τ) ≔hjC(φb(,τ−t),t,τ−t), h_j^C ( _ b( x,τ-t),t,τ-t ), (14) ηj(,t) _j( x,t) ≔hjS(φb(,T(t)),tf). h_j^S ( _ b( x,T(t)),t_f ). The resulting recoverable set is (t)≔∈|ξjC(,t;τ)≥0,∀j∈,∀τ∈[t,tf],ηj(,t)≥0,∀j∈. splitS(t) \ x \ |\ & _j^C( x,t;τ)≥ 0, ∀ j ,\ ∀τ∈[t,t_f],\\ & _j( x,t)≥ 0, ∀ j \. split (15) The set (t)S(t) contains states whose backup avoids the predicted occupancy through tft_f and reaches 0(tf)S_0(t_f). In Fig. 3, Semantic-Off evaluates the backup against current geometry, whereas Semantic-On uses predicted future occupancy to intervene before the backup becomes infeasible. Fig. 3: Backup feasibility with and without physical-event prediction. (a) Semantic-Off evaluates the backup using current obstacle geometry, so the resulting motion may continue toward the object’s future occupancy. (b) Semantic-On evaluates the same backup against predicted future occupancy and modifies the nominal motion while the backup remains feasible. Predicted occupancy is shown in both panels but is used only by Semantic-On. Generated conceptual illustration; geometry, trajectories, and timing are schematic. IV-E Input-Affine Safety Filtering We impose separate CBF conditions on the occupancy and terminal margins. For fixed τ, let s=τ−ts=τ-t, =φb(,s) y= _ b( x,s), and f=φb(,T(t)) y_f= _ b( x,T(t)). Define AjC A_j^C ≔(∇hjC)⊤Φb(,s)g(), ( _ yh_j^C ) _ b( x,s)g( x), (16) AjS A_j^S ≔(∇hjS)⊤Φb(,T(t))g(), ( _ yh_j^S ) _ b( x,T(t))g( x), where the gradients are evaluated at (,t,s)( y,t,s) and (f,tf)( y_f,t_f), respectively. Since τ and tft_f are fixed, ds/dt=T˙=−1ds/dt= T=-1. Using ∂thjC=∂shjC _th_j^C= _sh_j^C and (3), differentiation of (14) gives ξ˙jC=AjC(−πb()) ξ_j^C=A_j^C( u- _ b( x)) and η˙j=AjS(−πb()) η_j=A_j^S( u- _ b( x)). We suppress the arguments of AjCA_j^C, AjSA_j^S, ξjC _j^C, and ηj _j when their evaluation points are clear. The safety filter is then ⋆(,t)= u ( x,t)= argmin∈12‖−nom(,t)‖W2 _ u 12 \| u- u_ nom( x,t) \|_W^2 (17) s.t. s.t. AjC(−πb())≥−αC(ξjC),∀j,τ, A_j^C ( u- _ b( x) )≥- _C( _j^C), ∀ j,τ, AjS(−πb())≥−αS(ηj),∀j, A_j^S ( u- _ b( x) )≥- _S( _j), ∀ j, where j∈j , τ∈[t,tf]τ∈[t,t_f], W≻0W 0, and αC,αS _C, _S are locally Lipschitz extended class-K functions. The predictive constraints are affine in u. Such QP-based safety filters can be solved at millisecond-level runtime in robotic systems [18]. With finitely many enforced times, affine static-obstacle constraints, and polyhedral U, (17) is a QP; for a general compact convex U, it remains a convex optimization problem. For ∈(t) x (t), =πb() u= _ b( x) satisfies both predictive constraint families. Under continuous enforcement and suitable regularity, scalar comparison preserves initially nonnegative margins between updates. Additional static-obstacle CBF constraints must remain jointly feasible; higher-order CBFs may be used when required by relative degree [22]. IV-F Prediction Updates and Practical Implementation Predictions retain their observation times, so delays shorten the remaining horizon without shifting the predicted event. We accept a joint update for all objects only after checking object identities, the prediction interval, current-state membership in the updated (t)S(t), and feasibility of the full filter, including static-obstacle constraints. After rejection, the previous prediction remains active only while its own object scope, time interval, and filter feasibility remain valid. A missing prediction is not treated as an absent obstacle. If no valid prediction remains, the controller executes an emergency response, which does not itself certify safety outside (t)S(t). In implementation, the rollout constraints are enforced at finitely many times in [t,tf][t,t_f]. Fig. 4: Selected MuJoCo trials with physical-event prediction disabled (Semantic-Off) and enabled (Semantic-On). (a) Ceiling fixture: Semantic-Off continues beneath the falling fixture and collides, whereas Semantic-On brakes before full detachment and later navigates around the debris. The red region shows the unsafe occupancy used by the filter at 7.0 s. (b) Impact-induced loss of support: Semantic-Off continues as the stack begins to fall and is struck by a block, whereas Semantic-On brakes and subsequently reaches the goal. Insets show synchronized robot-view images. These qualitative trials are separate from the aggregate benchmark. V RESULTS We evaluate PSS in quadruped navigation with hazards arising from support loss, impact, and contact propagation, where the resulting future unsafe occupancy is not evident from current scene geometry alone. The experiments examine whether anticipating changes in object motion improves collision avoidance over filters that rely on current obstacle geometry. We report safe episode rates and use selected trials to examine when the robot brakes and how it resumes navigation after the event. V-A Experimental Setup Robot and controller setup: We simulate a Unitree Go1 in MuJoCo. PSS uses onboard RGB-D observations for visual physical reasoning and metric motion estimation. The safety filter uses the planar double-integrator model =[⊤,⊤]⊤ x=[ p , v ] , ˙= p= v, and ˙= v= u, where u is planar acceleration. The prescribed backup policy is πb()=satamax(−kb), _ b( x)=sat_a_ (-k_ b v ), where kb=2.5s−1k_ b=2.5\,s^-1 and saturation is applied componentwise at amax=2.0m/s2a_ =2.0\,m/s^2. The implementation uses rR=0.403r_R=0.403 m, ϵ=0.05ε=0.05 m, αC(h)=αS(h)=3h _C(h)= _S(h)=3h, and W=IW=I. The QP is solved using OSQP without slack variables. The nominal acceleration is proportional to the difference between a goal-directed velocity reference and the previously commanded velocity, with gain 2.5s−12.5\,s^-1 and componentwise limits of 1.5m/s21.5\,m/s^2. The filtered acceleration is integrated to obtain the velocity command supplied to the locomotion policy. The reduced-order model does not by itself certify full-body tracking. Learned-model inference is performed on one NVIDIA A100 GPU. Prediction and sampling: We use pretrained Code-as-World-VL-9B without further fine-tuning. The safety filter and locomotion policy update every 0.02 s, and RGB-D acquisition is scheduled every 0.05 s. The prediction runners request updates at 1 s intervals. For the ceiling fixture, trajectories are computed over 8 s at 0.1 s intervals using up to 17 evenly spaced detachment-time hypotheses. For the stack and domino scenarios, the VLM predicts five future positions at 0.5 s intervals over 2.5 s. The constraint grid includes the prediction nodes and subdivides intervals to keep their spacing at most 0.15 s. For conformal calibration, we set δ=0.05δ=0.05 and use Ncal=1000N_ cal=1000. Following [16], σj(τ,tk) _j(τ;t_k) is set to the maximum training-set position error at the corresponding prediction time. Scenarios: We consider a ceiling fixture that loses support, a rolling ball that destabilizes a stack, and contact propagation through falling dominoes. The ceiling fixture requires reasoning about a possible fall while the space beneath it remains clear. The stack and domino scenarios involve motion induced by interactions among objects and are inspired by Physion [1]. These cases test whether the robot responds to the predicted consequences of a physical interaction rather than only to obstacles already in its path. Figure 4 shows selected ceiling-fixture and stack trials. V-B Evaluation Protocol and Compared Methods The benchmark contains 30 physical cases, with 10 per scenario type and five variations of the initial robot position and yaw. This gives 150 configurations per method and 750 episodes across five methods, including hazardous events and benign controls. The primary metric is the safe episode rate: an episode is counted as safe if the robot avoids collisions with hazards and fixed obstacles throughout the evaluation horizon. Stopping without collision counts as a safe outcome, even if the robot does not reach the goal. We compare Plain CBF, OmniVLA, Backup CBF (Semantic-Off), PSS without the conformal margin, and PSS. Plain CBF and Backup CBF use current simulator-measured geometry at 50 Hz. An object enters the obstacle map when its lowest measured point reaches a height of 0.55 m and remains tracked afterward. Backup CBF uses the same braking policy, flow sensitivities, and numerical terminal condition as PSS, but holds measured obstacle geometry fixed during each backup rollout. PSS instead evaluates the backup against predicted future occupancy. In the qualitative comparisons, Semantic-On denotes PSS with physical-event prediction enabled. OmniVLA [23] uses the unmodified NHirose/omnivla-original checkpoint with the action head and pose projector from training step 120,000. Its inputs are the current forward RGB image, the goal pose relative to the robot, and the instruction “move toward the specified goal while avoiding obstacles.” The model predicts eight waypoints; the fifth is converted to forward-speed and yaw-rate commands using the official decoder, with limits of 0.3 m/s and 0.3 rad/s, respectively. Inference is requested at a nominal rate of 3 Hz in simulation time, and the latest command is held between updates. These commands are tracked by the shared Go1 locomotion policy without an additional CBF filter. V-C Safety Performance Figure 5 reports the safe episode rate across all three scenario types. PSS achieves 99.3%, compared with 20.0% for Plain CBF, 37.3% for OmniVLA, and 43.3% for Backup CBF. The improvement over Backup CBF is 56.0 percentage points. Although Backup CBF improves on Plain CBF, evaluating a braking maneuver against current geometry does not account for the changes in object motion caused by the physical event. The comparison supports evaluating the backup against predicted future occupancy, rather than relying on the availability of a braking maneuver alone. PSS without the conformal margin achieves a safe episode rate of 98.7%, corresponding to 148 safe episodes out of 150, compared with 149 for PSS. Both variants substantially outperform the CBF baselines that use current geometry, while their difference amounts to one safe episode in this benchmark. The empirical safe episode rate measures collision avoidance; it is distinct from the occupancy-coverage probability in Proposition 1. Fig. 5: Safe episode rates over 750 episodes, with 150 episodes per method. Error bars show 95% bootstrap confidence intervals obtained by resampling physical cases while grouping the five pose variations of each case. Stopping without collision counts as a safe outcome; goal completion is not required. V-D Intervention Timing and Navigation Ceiling fixture: Figure 4(a) illustrates the distinction between current obstacle geometry and predicted unsafe occupancy. With physical-event prediction disabled, the robot approaches the partially detached fixture, continues beneath it as it falls, and collides at 6.3 s. PSS instead begins braking while the fixture is still suspended. The robot remains outside the unsafe occupancy shown at 7.0 s and later navigates around the fallen fixture, as shown at 20.0 s. Thus, the robot responds to the anticipated fall before the object obstructs its route, then continues navigation around the resulting debris. Fig. 6: Planar speed during the ceiling-fixture trial. Backup braking begins at 2.88 s, before full fixture detachment at 5.80 s. The robot remains nearly stationary through detachment and resumes motion during debris avoidance, which begins at 7.20 s. Annotations mark the three events. Impact-induced loss of support: In Fig. 4(b), a rolling ball destabilizes the stack. At 2.8 s, Semantic-Off continues as the stack begins to fall and is struck by a block shortly afterward. Semantic-On brakes as the stack begins to fall, avoids the falling blocks, and reaches the goal by the 19.5 s frame. Here, the hazard is not only the approaching ball but also the motion it induces in the initially stationary stack. The comparison illustrates the role of predicting the consequences of the impact when evaluating the robot’s backup maneuver. These selected trials are separate from the aggregate benchmark. Braking before detachment: Figure 6 shows the planar speed during the ceiling-fixture trial. Backup braking begins at 2.88 s, which is 2.92 s before full detachment at 5.80 s. The speed decreases to nearly zero by approximately 4.0 s, and the robot remains nearly stationary through detachment. Motion resumes during debris avoidance, which begins at 7.20 s. The speed trace therefore confirms that braking precedes full detachment, while the subsequent acceleration and the trajectory in Fig. 4(a) show continued navigation after the event. VI CONCLUSION We presented PSS, a framework that connects visual physical reasoning to safety-critical control. PSS converts predicted physical events into time-varying unsafe occupancy and combines conformal calibration with backup-based safety filtering. By evaluating a prescribed backup maneuver against the predicted occupancy, the filter minimally modifies the nominal input to preserve backup feasibility under the robot dynamics and input limits. MuJoCo quadruped experiments involving support loss, impact, and contact propagation demonstrate improved safe episode rates over filters that rely on current obstacle geometry. Selected trials show braking before full fixture detachment and subsequent navigation around the debris, illustrating the importance of anticipating physical events while a safe response remains feasible. Future work will extend PSS to robot-induced interactions through action-conditioned object-motion prediction. We will also incorporate bounds on locomotion tracking errors into backup evaluation and validate the framework on physical robots. References [1] D. M. Bear, E. Wang, D. Mrowca, F. Binder, H. F. Tung, R. T. Pramod, C. Holdaway, S. Tao, K. Smith, F. Sun, F. Li, N. Kanwisher, J. B. Tenenbaum, D. L. K. Yamins, and J. Fan (2021) Physion: evaluating physical prediction from vision in humans and machines. In Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, Cited by: §I-A, §I, §V-A. [2] J. Han, W. Huang, H. Ma, J. Li, J. B. Tenenbaum, and C. Gan (2022) Learning physical dynamics with subequivariant graph neural networks. In Neural Information Processing Systems (NeurIPS), p. 26256–26268. External Links: Document Cited by: §I-A, §I. [3] J. Yuan, G. Sun, Z. Xiao, H. Zhou, X. Luo, J. Luo, Y. Zhao, W. Ju, and M. Zhang (2024) EGODE: an event-attended graph ODE framework for modeling rigid dynamics. In Neural Information Processing Systems (NeurIPS), p. 59093–59118. External Links: Document Cited by: §I-A, §I. [4] W. Chow, J. Mao, B. Li, D. Seita, V. Guizilini, and Y. Wang (2025) PhysBench: benchmarking and enhancing vision-language models for physical world understanding. In International Conference on Learning Representations (ICLR), p. 97959–98108. Cited by: §I-A, §I. [5] H. Wang, Y. Cai, W. Chen, J. Chi, H. Sun, Q. Dai, Y. Hung, X. Guo, J. Ren, R. Yao, Z. Liu, M. Long, Y. Duan, J. Gao, J. Lyu, F. Liu, and J. Wu (2026) Code as Worlds: agentic discovery of executable world representations for physical reasoning. Note: arXiv preprint arXiv:2608.27549 External Links: Document, 2608.27549 Cited by: §I-A, §I, §I, §IV-A. [6] T. Kim (2026) Is Your Safe Controller Actually Safe? A Critical Review of CBF Tautologies and Hidden Assumptions. In arXiv preprint arXiv:2603.06954, (en). Cited by: §I. [7] T. Kim, A. D. Menon, A. Trivedi, and D. Panagou (2026) Backup-Based Safety Filters: A Comparative Review of Backup CBF, Model Predictive Shielding, and gatekeeper. In IEEE Conference on Decision and Control (CDC), External Links: Document Cited by: §I, §I-C. [8] Y. Chen, M. Jankovic, M. Santillo, and A. D. Ames (2021) Backup control barrier functions: formulation and comparative study. In IEEE Conference on Decision and Control (CDC), p. 6835–6841. External Links: Document Cited by: §I-A, §I, §I-C. [9] T. Kim, H. K. Park, R. Wada, N. Atanasov, S. Koga, and D. Panagou (2026) OcclusionCBF: backup control barrier functions for safe navigation among hidden dynamic obstacles. Note: arXiv preprint arXiv:2609.06342 External Links: Document, 2609.06342 Cited by: §I-A, §I, §IV-D. [10] M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Komeili, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, S. Arnaud, A. Gejji, A. Martin, F. R. Hogan, D. Dugas, P. Bojanowski, V. Khalidov, P. Labatut, F. Massa, M. Szafraniec, K. Krishnakumar, Y. Li, X. Ma, S. Chandar, F. Meier, Y. LeCun, M. Rabbat, and N. Ballas (2025) V-JEPA 2: self-supervised video models enable understanding, prediction and planning. Note: arXiv preprint arXiv:2506.09985 External Links: Document, 2506.09985 Cited by: §I-A. [11] J. Schmidhuber and D. Prelinger (1993) Discovering predictable classifications. Neural Computation 5 (4), p. 625–635. Note: Originally Technical Report CU-CS-626-92, University of Colorado, 1992 External Links: Document Cited by: §I-A. [12] M. Ganai, R. Sinha, C. Agia, D. Morton, L. Di Lillo, and M. Pavone (2025) Real-time out-of-distribution failure prevention via multi-modal reasoning. In Conference on Robot Learning (CoRL), p. 283–308. Cited by: §I-A. [13] L. Lindemann, M. Cleaveland, G. Shim, and G. J. Pappas (2023) Safe planning in dynamic environments using conformal prediction. IEEE Robotics and Automation Letters 8 (8), p. 5116–5123. External Links: Document Cited by: §I-A. [14] A. Dixit, L. Lindemann, S. X. Wei, M. Cleaveland, G. J. Pappas, and J. W. Burdick (2023) Adaptive conformal prediction for motion planning among dynamic agents. In Learning for Dynamics and Control Conference (L4DC), p. 300–314. Cited by: §I-A. [15] T. Kim, R. I. Kee, and D. Panagou (2026) Learning to Adapt Control Barrier Functions Under Epistemic and Aleatoric Uncertainty. In arXiv Preprint arXiv:2504.03038, External Links: 2504.03038 Cited by: §I-A. [16] C. Stamouli, L. Lindemann, and G. J. Pappas (2024) Recursively feasible shrinking-horizon MPC in dynamic environments with conformal prediction guarantees. In Learning for Dynamics and Control Conference (L4DC), p. 1330–1342. Cited by: §I-A, §IV-C, §V-A. [17] A. D. Ames, X. Xu, J. W. Grizzle, and P. Tabuada (2017) Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control 62 (8), p. 3861–3876. External Links: Document Cited by: §I-A. [18] T. Kim, H. Okamoto, B. Hoxha, G. Fainekos, and D. Panagou (2026) Policy library cbf: finite-horizon safety at runtime via parallel rollouts. arXiv preprint arXiv:2605.16588. Cited by: §I-A, §IV-E. [19] A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada (2019) Control barrier functions: theory and applications. In European Control Conference (ECC), p. 3420–3431. External Links: Document Cited by: §I-B. [20] A. N. Angelopoulos and S. Bates (2023) Conformal prediction: a gentle introduction. Foundations and Trends in Machine Learning 16 (4), p. 494–591. External Links: Document Cited by: §IV-C. [21] X. Yu, Y. Zhao, X. Yin, and L. Lindemann (2026) Signal temporal logic control synthesis among uncontrollable dynamic agents with conformal prediction. Automatica 183, p. 112616. External Links: Document Cited by: §IV-C. [22] W. Xiao and C. Belta (2019) Control barrier functions for systems with high relative degree. In IEEE Conference on Decision and Control (CDC), p. 474–479. External Links: Document Cited by: §IV-E. [23] N. Hirose, C. Glossop, D. Shah, and S. Levine (2026) OmniVLA: an omni-modal vision-language-action model for robot navigation. In IEEE International Conference on Robotics and Automation (ICRA), Cited by: §V-B.