Paper deep dive
PV-WM: A Heterogeneous Micro-Macro World Model for Articulated Pedestrian-Vehicle Co-Rollout
Haozhuang Chi, Jingsong Liang, Ziying Song, Lei Yang, Shihao Li, Haoruo Zhang, Chen Lv
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/13/2026, 5:13:43 AM
Summary
The paper introduces PV-WM, a heterogeneous micro-macro world model for articulated pedestrian-vehicle co-rollout. It jointly predicts pedestrian root motion, 15-joint articulation, and rigid vehicle states in a synchronized recurrent framework. PV-WM outperforms modular specialists and one-shot predictors in accuracy (Root ADE, MPJPE, P-V distance error) while significantly reducing parameters, FLOPs, and latency.
Entities (9)
Relation Signals (6)
PV-WM → developedby → Nanyang Technological University
confidence 95% · Authors affiliated with Nanyang Technological University, Singapore.
PV-WM → outperforms → Modular Specialist
confidence 95% · PV-WM reduces Root ADE by 5.2%, MPJPE by 7.6%... relative to a validation-selected Modular Specialist.
PV-WM → predicts → pedestrian articulation
confidence 95% · It recurrently advances pedestrian root motion, 15-joint articulation, and learned vehicle states
PV-WM → predicts → vehicle states
confidence 95% · It recurrently advances pedestrian root motion, 15-joint articulation, and learned vehicle states
PV-WM → usesdatafrom → Waymo Open Motion Dataset
confidence 95% · We align the 15-joint pedestrian tracks of Waymo-3DSkelMo with vehicle tracks and oriented boxes from the Waymo Open Dataset
PV-WM → hasbetterefficiencythan → Modular Specialist
confidence 90% · The single-network model uses 57.1% fewer parameters, 96.5% lower average FLOPs... and 25.5% lower measured p95 latency.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Local pedestrian-vehicle forecasting spans heterogeneous physical scales: pedestrians combine root locomotion with articulated motion, whereas vehicles are rigid bodies described by kinematic state and oriented extent. Existing road-agent forecasters typically omit pedestrian articulation, while pose forecasters leave vehicle futures outside the learned rollout. We introduce PV-WM, a history-only world model over structured post-perception tracks. It recurrently advances pedestrian root motion, 15-joint articulation, and learned vehicle states within a synchronized heterogeneous state. The generated pedestrian and vehicle chunks supply the next recurrent boundary; vehicle boxes are reconstructed from predicted center and heading with observed extent, and P-V geometry is recomputed after every transition. Relative to a matched one-shot complete-state predictor, recurrent execution reduces Root ADE by 12.7% and MPJPE by 14.8%. Feedback interventions show that later predictions depend on the content, temporal order, and pedestrian identity of generated articulation. Across 824 aligned Waymo contexts, with 797 providing valid future vehicle support, PV-WM reduces Root ADE by 5.2%, MPJPE by 7.6%, P-V distance error by 11.9%, and oriented-box closest-approach error by 5.8% relative to a validation-selected Modular Specialist. The single-network model uses 57.1% fewer parameters, 96.5% lower average FLOPs per local scene, and 25.5% lower measured p95 latency. PV-WM unifies this heterogeneous future state while preserving type-specific pedestrian and vehicle dynamics.
Tags
Links
- Source: https://arxiv.org/abs/2609.07328v1
- Canonical: https://arxiv.org/abs/2609.07328v1
Trouble viewing inline? Open PDF directly →
Full Text
52,546 characters extracted from source content.
Expand or collapse full text
PV-WM: A Heterogeneous Micro–Macro World Model for Articulated Pedestrian–Vehicle Co-Rollout Haozhuang Chi Affiliation: Nanyang Technological University, Singapore. Jingsong Liang Affiliation: Nanyang Technological University, Singapore. Ziying Song Affiliation: Nanyang Technological University, Singapore. Lei Yang Affiliation: Nanyang Technological University, Singapore. Shihao Li Affiliation: Beijing Institute of Technology, Beijing, China. Haoruo Zhang Affiliation: Nanyang Technological University, Singapore. Chen Lv †thanks: †Corresponding author, email: lyuchen@ntu.edu.sg. Affiliation: Nanyang Technological University, Singapore. Abstract Local pedestrian–vehicle forecasting spans heterogeneous physical scales: pedestrians combine root locomotion with articulated motion, whereas vehicles are rigid bodies described by kinematic state and oriented extent. Existing road-agent forecasters typically omit pedestrian articulation, while pose forecasters leave vehicle futures outside the learned rollout. We introduce PV-WM, a history-only world model over structured post-perception tracks. It recurrently advances pedestrian root motion, 15-joint articulation, and learned vehicle states within a synchronized heterogeneous state. The generated pedestrian and vehicle chunks jointly form the next recurrent boundary; vehicle boxes are reconstructed from predicted center and heading with observed extent, and P–V geometry is recomputed after every transition. Relative to a matched one-shot complete-state predictor, recurrent execution reduces Root ADE by 12.7%12.7\% and MPJPE by 14.8%14.8\%. Feedback interventions show that later predictions depend on the content, temporal order, and pedestrian identity of generated articulation. Across 824 aligned Waymo contexts, with 797 providing valid future vehicle support, PV-WM reduces Root ADE by 5.2%5.2\%, MPJPE by 7.6%7.6\%, P–V distance error by 11.9%11.9\%, and oriented-box closest-approach error by 5.8%5.8\% relative to a validation-selected Modular Specialist. The single-network model uses 57.1%57.1\% fewer parameters, 96.5%96.5\% lower average FLOPs per local scene, and 25.5%25.5\% lower measured p95 latency. PV-WM unifies this heterogeneous future state while preserving type-specific pedestrian and vehicle dynamics. Fig. 1: Motivation for heterogeneous co-rollout. Left: Agent-level forecasting omits pedestrian articulation, whereas vehicle-conditioned pose forecasting leaves vehicle futures outside the rollout. Right: PV-WM advances articulated pedestrians and learned rigid-vehicle states together. Generated P/V chunks jointly form the next boundary, P–V geometry is recomputed, and the persistent pair state is updated for the next transition. I Introduction The state and transition define the predictive scope of a world model. Local pedestrian–vehicle scenes are heterogeneous across physical type and resolution: pedestrians combine root transport with articulated body motion, vehicles carry rigid kinematic state and oriented extent, and P–V geometry evolves with both agents’ motion. Point- or box-level pedestrian states omit body configuration, whereas body-only forecasting leaves the surrounding vehicle future unresolved. Because these variables evolve on the same clock, a complete local model must advance them within one synchronized future. Existing forecasting paradigms model different subsets of this state. Multi-agent forecasters predict trajectories, centers, boxes, or motion tokens [1, 2, 3, 4, 5, 6, 7, 8], but generally retain pedestrians at agent-level resolution. Articulated forecasters preserve global locomotion and local pose [9, 10, 11], while vehicle-conditioned pose models use traffic histories without generating vehicle futures [12]. Driving world models span agent, occupancy, latent, visual, and driver-centered state spaces [13, 14, 15, 16, 17, 18], but do not expose articulated pedestrian state, learned vehicle motion, oriented extent, and synchronized P–V geometry within one recurrent local state. Composing separate pose and vehicle predictors can fill the same output fields, but does not define a shared transition in which both generated states form the next boundary and their geometry is resynchronized after every step. We introduce PV-WM, a history-only heterogeneous micro–macro world model for articulated pedestrian–vehicle co-rollout over structured post-perception tracks. It predicts pedestrian root motion and 15-joint articulation together with learned vehicle states over five recurrent chunks. The generated P/V chunks jointly form the two-frame boundary for the next transition; vehicle boxes are reconstructed and P–V geometry is recomputed after every transition. PV-WM thus evolves its own structured state across the horizon rather than decoding the future once from a fixed history representation. State completeness and pair-to-endpoint routing are independent design choices. PV-WM combines type-specific endpoint transitions with a persistent pair state while keeping the endpoint branches factorized. Routing controls inject pair context into the pedestrian branch, the vehicle branch, or both. Ablations attribute the principal architectural gains to recurrent typed execution and the persistent pair branch, while relational supervision improves closest-approach accuracy. Feedback interventions show that later transitions reuse generated articulation. Across a segment-grouped, context-disjoint five-fold evaluation of 824 Waymo contexts, including 797 with valid vehicle and pair support, PV-WM improves Root ADE, MPJPE, P–V distance error, and DCAEOBBDCAE_OBB over a validation-selected Modular Specialist by 5.2%5.2\%, 7.6%7.6\%, 11.9%11.9\%, and 5.8%5.8\%. The single-network model uses 57.1%57.1\% fewer parameters, 96.5%96.5\% lower average FLOPs per local scene, and 25.5%25.5\% lower measured p95 latency. Our contributions are threefold: • We formulate pedestrian–vehicle forecasting as recurrent prediction of a heterogeneous state spanning pedestrian transport and articulation, learned vehicle dynamics, oriented extent, and synchronized pair geometry. • We introduce a generated-state co-rollout with typed endpoint dynamics and persistent pair state, and isolate the effect of pair-to-endpoint routing under matched conditions. • We isolate the contributions of recurrent execution, typed dynamics, and the persistent pair branch, establish functional reuse of generated articulation, and demonstrate a more accurate and efficient complete-state system than a validation-selected Modular Specialist. I Related Work I-A Multi-Agent Motion Forecasting Multi-agent motion forecasting predicts road-user futures from agent histories and scene context. The Waymo Open Motion Dataset formalizes marginal and joint forecasting [1]. Trajectron++ combines heterogeneous dynamics with semantic context [2], while scene- and query-centric architectures reason over agents, time, intentions, and maps through attention [3, 4, 5, 6]. M2I decomposes interactive prediction into influencer and conditional-reactor forecasts [19], whereas FJMP factorizes joint futures over a learned directed acyclic interaction graph [20]. MotionLM autoregressively generates multi-agent futures as motion tokens and supports temporally causal conditional rollout [7]; GameFormer uses hierarchical game-theoretic refinement [8]. VP-LSTM directly combines pedestrian trajectories with vehicle oriented-box forecasting [21]. These approaches span marginal, conditional, joint, autoregressive, and graph-factorized prediction, but retain pedestrians at agent-level resolution. PV-WM instead carries articulated pedestrian state alongside learned vehicle motion on a shared recurrent timeline. I-B Articulated Human Motion Forecasting Articulated forecasting preserves the distinction between global locomotion and root-relative body motion. TBIFormer models inter- and intra-person dynamics with trajectory-aware body-part tokens [9], while T2P conditions local pose on predicted global trajectories [10]. Waymo-3DSkelMo brings temporally coherent 3D skeletons into autonomous-driving scenes [11]. VehCondPose3D further conditions future pedestrian pose on surrounding vehicle histories through vehicle encoding and pedestrian–vehicle attention [12]. These methods preserve body configuration, but surrounding vehicles remain observed conditions rather than generated future states. PV-WM advances articulated pedestrians and learned vehicles within the same future process. I-C World Models for Autonomous Driving World models learn predictive transitions for imagination, simulation, and decision making [22, 23]. Driving world models instantiate these transitions at different resolutions: TrafficBots rolls forward explicit multi-agent behavior [13], OccWorld predicts future occupancy and ego motion [14], and DriveWorld learns latent scene dynamics for downstream perception and planning [15]. Visual models such as GAIA-1, Drive-WM, and Epona generate future observations from scene, trajectory, or action conditions [24, 16, 17]. Driver-WM recurrently forecasts articulated in-cabin motion conditioned on traffic context [18]. PV-WM targets a complementary external post-perception state in which pedestrian articulation, root transport, learned rigid-vehicle state, oriented extent, and P–V geometry remain explicit and synchronized throughout rollout. I Method Fig. 2: PV-WM architecture. Typed encoders and the final two observed P/V states initialize 0=(0,ℰ0)Z_0=(H_0,E_0). Each transition decodes P/V chunks, recomputes relation geometry, reconstructs oriented boxes, updates typed memories, and refreshes ℰm+1E_m+1. Five transitions produce the structured future. PV-WM advances structured post-perception tracks through a recurrent rollout. At the macro scale, it predicts pedestrian root motion and rigid-vehicle kinematics; at the micro scale, it predicts root-relative pedestrian articulation. All variables evolve on a shared clock. The model observes To=20T_o=20 frames and predicts Tf=10T_f=10 frames at 10Hz10\,Hz through M=5M=5 transitions of c=2c=2 frames. Generated P/V chunks jointly form the next two-frame boundary; vehicle OBBs are reconstructed and P–V geometry is recomputed from the generated states after every transition. PV-WM combines typed endpoint dynamics with a persistent pair state while keeping the endpoint branches factorized. Routing controls inject pair context into the pedestrian branch, the vehicle branch, or both. I-A Heterogeneous State and Typed Encoding Indices i, k, and a∈0,…,14a∈\0,…,14\ denote pedestrians, vehicles, and joints, with a=0a=0 the pelvis root; t, m, and τ denote frame, transition, and within-chunk time. Valid components use ℳ=(P,V,R)M=( μ^P, μ^V,M^R), with μikR=maxtMik,tRμ^R_ik= _tM^R_ik,t. We translate coordinates by the final observed mean pedestrian root without rotating the scene. The typed states are i,t,a _i,t,a =i,t,0+ℓi,t,a, =p_i,t,0+ _i,t,a, ℓi,t,0 _i,t,0 =, =0, (1) k,tV ^V_k,t =(k,tV,k,tV,sinψk,t,cosψk,t,νk,t,ok,t). =(x^V_k,t,u^V_k,t, _k,t, _k,t, _k,t,o_k,t). Here i,t,0p_i,t,0 is root transport, ℓi,t,a _i,t,a is root-relative articulation, and i,tP=Πxy(i,t,0)x^P_i,t= _xy(p_i,t,0). For vehicles, k,tVx^V_k,t, k,tVu^V_k,t, ψk,t _k,t, νk,t _k,t, and ok,to_k,t denote planar center, planar velocity, heading, speed, and presence, respectively. Let t=i,t,ai,aP_t=\p_i,t,a\_i,a and t=k,tVkV_t=\s^V_k,t\_k. For each valid pair, ik,t δ_ik,t =k,tV−i,tP, =x^V_k,t-x^P_i,t, dik,t d_ik,t =‖ik,t‖2, =\| δ_ik,t\|_2, κik,t _ik,t =−ik,t⊤(k,tV−i,tP)dik,t+ϵ, =- δ_ik,t (u^V_k,t-u^P_i,t)d_ik,t+ε, ik,t _ik,t =(δx,δy,d,κ)ik,t, =(δ^x,δ^y,d,κ)_ik,t, (2) where Pu^P is a finite difference and positive κ denotes closing motion. With t=ik,ti,kR_t=\r_ik,t\_i,k and tB_t the vehicle OBB collection, the explicit state is tphys=(t,t,t,t)S_t^phys=(P_t,V_t,R_t,B_t), where t=Φ(t−1:t,t)R_t= (P_t-1:t,V_t) and t=Ψbox(t,V,obs)B_t= _box(V_t; ξ^V,obs). The structured input and physical-state output are obs ^obs =(obs,obs,obs,obs,ℳ), =(P^obs,V^obs,B^obs,R^obs,M), (3) ^phys S^phys =(^,^,^,^). =( P, V, R, B). Only P/V states are generated directly. Φ applies Eq. (2); Ψbox _box uses generated center and heading with the observed vertical center and static extent V,obs ξ^V,obs, normalizing heading only for box construction. Q denotes an auxiliary pair-risk readout optimized through ℒQL_Q. The pedestrian encoder aggregates root, anatomical-group, and root-relative motion; the vehicle GRU pools center increments, velocity, acceleration, heading, speed, and presence. It initializes ~iP h^P_i =EP(iobs), =E_P(P_i^obs), ~kV h^V_k =EV(kobs), =E_V(V_k^obs), (4) ik,0R ^R_ik,0 =FR[~iP‖~kV‖ER(ikobs,ikR)]. =F_R[ h^P_i\| h^V_k\|E_R(R^obs_ik;M^R_ik)]. Endpoint adapters initialize i,0Ph^P_i,0 and k,0Vh^V_k,0 from their corresponding typed encodings. All routing settings initialize and retain ik,0Rh^R_ik,0. Routing controls additionally inject mask-aggregated pair context into selected endpoint branches during initialization, decoding, and recurrent updates. I-B Generated-State Co-Rollout At chunk boundary m∈0,…,Mm∈\0,…,M\, m _m =(m,ℰm), =(H_m,E_m), (5) m _m =(i,mPi,k,mVk,ik,mRi,k), =(\h^P_i,m\_i,\h^V_k,m\_k,\h^R_ik,m\_i,k), ℰm _m =(mprev,mcur,mprev,mcur), =(P_m^prev,P_m^cur,V_m^prev,V_m^cur), ℰ0 _0 =Tail2(obs,obs), =Tail_2(P^obs,V^obs), 0 _0 =(0,ℰ0). =(H_0,E_0). The encoders compress the full ToT_o-frame history into 0H_0. At each boundary, mH_m carries typed context, including the persistent pair state, while ℰmE_m supplies adjacent P/V states for constant-velocity references and finite-difference geometry. The explicit relation tR_t is recomputed from P/V states, whereas ik,mRh^R_ik,m retains pair history. Parameters θ are shared across all M transitions; for m≥1m≥ 1, ℰmE_m contains generated P/V states only. For m=0,…,M−1m=0,…,M-1, the transition is (^phys,(m),^(m),m+1) ( S^phys,(m), Q^(m),Z_m+1) (6) =θ,(m;ℳ,V,obs),=(ρP,ρV)∈0,12. =T_θ, ρ(Z_m;M, ξ^V,obs), ρ=( _P, _V)∈\0,1\^2. Here ρP _P and ρV _V gate pair context to the pedestrian and vehicle endpoint branches, respectively. ℳM and V,obs ξ^V,obs remain fixed across rollout. Root and vehicle motion use constant-velocity residuals; articulation uses a gated residual from the current root-relative pose: ^i,m,τ,0 p_i,m,τ,0 =i,m,τ,0cv+Δi,m,τP, =p^cv_i,m,τ,0+ ^P_i,m,τ, ℓ^i,m,τ,a _i,m,τ,a =ℓi,m,acur+σ(γi,m,τ,a)Δℓi,m,τ,a,a=1,…,14, = ^cur_i,m,a+σ( _i,m,τ,a) _i,m,τ,a, a=1,…,14, ^k,m,τV s^V_k,m,τ =V,cvk,m,τ+ΔVk,m,τ,τ=1,…,c. =s^V,cv_k,m,τ+ ^V_k,m,τ, τ=1,…,c. (7) The full skeleton is ^i,m,τ,a=^i,m,τ,0+ℓ^i,m,τ,a p_i,m,τ,a= p_i,m,τ,0+ _i,m,τ,a. Both endpoints decode from the same pre-chunk mZ_m, without consuming each other’s new future within the chunk. Closure, recurrence, and temporal assembly are ^(m) R^(m) =Φ(ℰm,^(m),^(m)), = (E_m, P^(m), V^(m)), ^(m) B^(m) =Ψbox(^(m),V,obs), = _box( V^(m); ξ^V,obs), ^phys,(m) S^phys,(m) =(^(m),^(m),^(m),^(m)), =( P^(m), V^(m), R^(m), B^(m)), (8) ℰm+1 _m+1 =Tail2(^(m),^(m)),m+1=(m+1,ℰm+1), =Tail_2( P^(m), V^(m)), _m+1=(H_m+1,E_m+1), (9) ^phys1:Tf S^phys_1:T_f =Concatm=0M−1^phys,(m). =Concat_m=0^M-1 S^phys,(m). (10) For the first generated frame, ℰmE_m supplies the preceding pedestrian root; later relation states use adjacent generated frames. Final P/V states yield i,mPg^P_i,m and k,mVg^V_k,m; the projected final relation and masked temporal pool yield ik,mR,endg^R,end_ik,m and ik,mR,chunkg^R,chunk_ik,m. These summaries update m+1H_m+1. I-C Pair-to-Endpoint Routing Controls Let ¯n,mR,end→X g^R,end→ X_n,m be the mask-pooled final-relation summary incident to endpoint n of type X∈P,VX∈\P,V\, and define ΓρX()=ρX _ _X(x)= _Xx. 0,mXc^X_0,m is within-type, whereas 1,mXc^X_1,m additionally includes the fused P/V/R summary; mRc^R_m is the fused masked P/V/R summary used by the pair update. The recurrent updates are n,m+1X ^X_n,m+1 =n,mX+UX(n,mX,n,mX,ΓρX(¯n,mR,end→X),ρX,mX), =h^X_n,m+U_X\! (h^X_n,m,g^X_n,m, _ _X( g^R,end→ X_n,m),c^X_ _X,m ), ik,m+1R ^R_ik,m+1 =ik,mR+UR(ik,mR,ik,mR,chunk,mR). =h^R_ik,m+U_R\! (h^R_ik,m,g^R,chunk_ik,m,c^R_m ). (11) Here n=in=i for pedestrians and n=kn=k for vehicles. PV-WM uses =(0,0) ρ=(0,0), retaining the pair state for relational representation and auxiliary readout while excluding pair context from endpoint initialization, decoding, and update. The matched controls use (1,0)(1,0), (0,1)(0,1), or (1,1)(1,1), routing observed pair context during initialization and decoding and generated pair summaries during later endpoint updates. All settings share the same state, supervision, horizon, and model budget. I-D Learning Objective All routing settings minimize ℒ=ℒP+ℒV+ℒR+ℒQ+0.05ℒscene.L=L_P+L_V+L_R+L_Q+0.05L_scene. (12) ℒPL_P uses joint, root, root-relative, body-local, bone, velocity, and acceleration weights (1.0,0.50,0.25,0.10,0.05,0.05,0.02)(1.0,0.50,0.25,0.10,0.05,0.05,0.02); ℒVL_V uses state and center weights (1.0,0.50)(1.0,0.50); ℒRL_R uses R4, distance, and horizon-minimum-distance weights (0.35,0.30,0.15)(0.35,0.30,0.15); and ℒQL_Q uses pairwise and chunk-risk weights (0.10,0.10)(0.10,0.10), with targets defined as the product of smooth proximity and closing-motion scores. The scene loss regresses observed minimum P–V distance and entity counts. Since relations are closed from generated endpoints, ℒRL_R directly supervises the P/V predictors. Boxes are reconstructed without a separate regression loss. Future reference states enter training only as supervision targets, while recurrent boundaries and latent updates use generated P/V states. IV Experiments Experiments evaluate forecasting accuracy and complete-system cost, attribute gains to individual components, and test the recurrent utility of generated articulation. IV-A Experimental Setup Protocol and evaluation We align the 15-joint pedestrian tracks of Waymo-3DSkelMo [11] with vehicle tracks and oriented boxes from the Waymo Open Dataset [25], obtaining 824 driving contexts. We extract up to eight 3s3\,s windows per context at a 1s1\,s stride and form local scenes as connected components of an observation-only P–V graph. Vehicles whose minimum observed distance to any pedestrian is at most 25m25\,m are retained as candidates. A P–V edge is activated when the observed minimum distance is at most 12m12\,m, the minimum time to collision is at most 4s4\,s, or the maximum closing speed is at least 0.25ms−10.25\,m\,s^-1. Isolated pedestrians are retained, and each component is capped at eight pedestrians and eight vehicles, yielding 8,364 local-scene instances. All methods observe 2020 frames over 2s2\,s and predict 1010 frames over 1s1\,s at 10Hz10\,Hz; PV-WM executes five two-frame transitions. We use segment-grouped, context-disjoint five-fold out-of-fold evaluation: contexts, overlapping windows, and tracks from the same Waymo segment remain in the same fold, and every context is held out once. Trainable models use seeds 4242, 123123, and 456456; deterministic CV references are evaluated once. Checkpoints and Modular Specialist compositions are selected using the associated validation split only. For each seed, sufficient statistics are pooled across the five held-out folds before computing the cross-seed mean and sample standard deviation. Paired comparisons use 10,000 context-clustered bootstrap resamples with Holm correction within each prespecified family. Pedestrian metrics use all 824 contexts; vehicle and pairwise metrics draw from the 797 contexts with valid future vehicle support, with metric-specific masks applied over valid horizons. The main 824-context PV-WM models are map-free and trained from scratch. Implementation details All PV-WM routing settings use 128-dimensional hidden states and are trained from scratch with Adam, an initial learning rate of 10−410^-4, zero weight decay, and batch size 66 for at most 3030 epochs. The pedestrian encoder fuses root, body-part, root-delta, and local-motion MLP streams; vehicle history is encoded by a one-layer GRU with masked temporal pooling. Chunk heads and recurrent P/V/R updates use two-layer GELU MLPs with 128-dimensional hidden layers. Early stopping begins after epoch 1818, with patience 66 and a minimum relative improvement of 10−310^-3. Checkpoints are selected on the corresponding validation split using the equal-weight mean of six errors normalized by the corresponding errors of the CV all-joints + CV-V reference computed on the fold’s training split and fixed before validation: Root ADE, MPJPE, APE, Vehicle ADE, P–V Distance MAE, and DCAEOBBDCAE_OBB. Each routing setting is trained independently with the same optimizer, training budget, and checkpoint-selection rule. Metrics Root ADE and FDE measure pedestrian transport. MPJPE evaluates absolute 15-joint error, while APE removes root translation and evaluates local articulation. Vehicle ADE and FDE measure planar center error, and P–V Distance MAE measures framewise pedestrian-root-to-vehicle-center distance error. Following distance-of-closest-approach error [26], we report an oriented-box adaptation: DCAEOBB=1|v|∑(i,k)∈v|mint∈ikc^ik,t−mint∈ikcik,t|,DCAE_OBB= 1|P_v| _(i,k) _v | _t _ik c_ik,t- _t _ikc_ik,t |, where vP_v is the set of valid pedestrian–vehicle pairs, ikT_ik is pair (i,k)(i,k)’s valid forecast horizon, and cik,tc_ik,t is the signed BEV clearance between a 0.30m0.30\,m-radius pedestrian root disk and the reconstructed vehicle OBB. Predicted boxes use generated centers and headings with observation-derived dimensions. DCAEOBBDCAE_OBB measures error in horizon-minimum signed BEV clearance, complementing the framewise P–V Distance MAE. Cyclic BoxCorner removes arbitrary starting-corner dependence by averaging, over valid vehicle frames, the minimum mean corner distance across the four cyclic shifts of a common clockwise corner ordering [27]. The same matching rule is applied to every method. In the articulation study, fMPJPE denotes final-frame MPJPE. wAPE weights each non-root joint error by the magnitude of its reference root-relative displacement from the last observed pose, clips the weight at 0.50m0.50\,m, and normalizes by the total valid weight. Baselines and complete systems Articulated references comprise CV joint extrapolation, TBIFormer [9], VehCondPose3D [12], and T2P [10], paired with the same prescribed CV vehicle rollout. Trajectron++ [2], QCNet [6], and MTR [5] learn pedestrian-root and vehicle futures but do not predict articulated pose. Released implementations are adapted to the common temporal and evaluation protocol; multimodal methods use their highest-probability mode rather than a test-oracle minimum. The validation-selected Modular Specialist pairs TBIFormer, T2P, or VehCondPose3D with vehicle predictions from an independently trained modular recurrent model, Trajectron++, QCNet, or MTR. The recurrent model follows the same five two-frame transitions, and only its vehicle output enters the composition. Each fold selects its composition using the three-seed mean of the same validation score. PV-WM realizes the complete articulated pedestrian–vehicle state with one end-to-end network. System profiling Runtime and memory are measured directly on an NVIDIA RTX A5000 in FP32 with batch size one. Each fold uses 50 warm-up executions followed by 200 measured executions. Both Specialist networks are simultaneously resident and executed sequentially. Timing includes in-memory scene decoding, tensor construction and transfer, network execution, P–V geometry assembly, and CUDA synchronization, but excludes disk I/O. The reported p95 pools 1,000 executions across five folds, and memory is the maximum complete-execution peak across folds. FLOPs are measured in a separate pass to avoid profiler overhead and are averaged over all 8,364 local-scene instances using their observed pedestrian and vehicle counts. TABLE I: Forecasting accuracy and complete-system cost under the 2s→1s2\,s\!→\!1\,s protocol. Results are five-fold OOF mean± standard deviation over three seeds; the deterministic CV baseline is evaluated once. Errors are in millimeters. Boldface is assigned within groups with compatible output spaces; gray text marks methods with prescribed CV vehicle futures. Method Output / Construction Pedestrian Vehicle P–V Geometry System Cost Root ADE/FDE (m)↓ MPJPE/APE (m)↓ ADE/FDE (m)↓ Dist./DCAEOBBDCAE_OBB (m)↓ N/Params/Avg. FLOPs↓ p95/Mem. (ms/MiB)↓ Articulated pedestrian forecasting with prescribed vehicle motion CV all-joints + CV-V J15 + prescribed V 189.1/350.3 293.3/186.5 54.6/119.1 110.6/211.1 0/n.r./n.r. n.r. TBIFormer [9] + CV-V J15 + prescribed V 184.0± 10.5/316.8± 21.0 216.2± 9.3/87.2± 0.1 54.6/119.1 132.8± 8.1/230.0± 7.2 1/n.r./n.r. n.r. VehCondPose3D [12] + CV-V J15 + prescribed V 176.0± 5.2/304.3± 10.7 204.6± 3.1/82.5± 1.6 54.6/119.1 128.3± 4.3/225.2± 3.8 1/n.r./n.r. n.r. T2P [10] + CV-V J15 + prescribed V 389.3± 39.8/702.9± 76.2 406.2± 39.1/79.7± 1.9 54.6/119.1 289.2± 40.7/362.9± 37.7 1/n.r./n.r. n.r. Trajectory-native learned pedestrian–vehicle forecasting Trajectron++ [2] Root + learned V 179.1± 0.8/326.8± 1.1 – 64.8± 1.2/138.7± 5.2 103.0± 1.2/421.8± 7.0 1/n.r./n.r. n.r. QCNet [6] Root + learned V 143.6± 1.3/245.1± 2.5 – 68.9± 4.4/128.9± 6.0 111.7± 2.8/361.0± 2.4 1/n.r./n.r. n.r. MTR [5] Root + learned V 146.5± 1.5/251.6± 2.4 – 83.5± 5.8/155.0± 9.8 116.2± 2.7/382.4± 2.4 1/n.r./n.r. n.r. Complete articulated pedestrian–vehicle systems Modular Specialist† Val.-selected 3P×4V3P× 4V 174.2± 4.0/300.0± 7.8 204.3± 1.9/84.0± 1.6 68.1± 0.9/146.8± 2.4 128.4± 3.3/198.2± 2.0 2/8.757M/2.534G 29.46/201.09 PV-WM J15 + learned V + OBB + R 165.2± 2.3/288.4± 6.9 188.8± 1.7/83.9± 2.2 68.9± 0.4/148.2± 1.6 113.1± 0.2/186.6± 0.8 1/3.756M/89.71M 21.94/65.59 J15 denotes a predicted 15-joint skeleton; for PV-WM, OBBs are reconstructed and R is recomputed from generated P/V states; N is the number of learned networks, and n.r. denotes unavailable matched profiling. †The Modular Specialist is selected independently per fold from 3P×4V3P× 4V candidates using the six-metric validation composite (fold 0: TBIFormer + modular recurrent vehicle branch; folds 1–4: VehCondPose3D + modular recurrent vehicle branch); held-out data are never used for selection. Runtime and memory are direct complete-system measurements; FLOPs are averaged over 8,364 local-scene instances. Pedestrian metrics use 824 contexts; vehicle/P–V metrics draw from the 797 contexts with valid future support and apply metric-specific masks. Fig. 3: Complete-state forecasts on held-out contexts. Rows contain two moving and two near-interaction contexts; columns show the observation and predictions at 0.20.2, 0.60.6, and 1.0s1.0\,s. Black denotes the reference, red the Modular Specialist, and blue PV-WM; skeletons and vehicle OBBs jointly visualize articulated motion, rigid-body motion, and P–V geometry. IV-B Forecasting Accuracy and Complete-System Cost Table I groups methods by the future state they predict. Among articulated methods, PV-WM obtains the lowest Root ADE/FDE and MPJPE while learning vehicle futures; T2P has lower APE, and trajectory-native methods achieve lower root error but omit articulation. Among complete-state systems, PV-WM improves Root ADE, MPJPE, P–V Distance MAE, and DCAEOBBDCAE_OBB by 9.009.00, 15.5115.51, 15.3115.31, and 11.56m11.56\,m, or 5.2%5.2\%, 7.6%7.6\%, 11.9%11.9\%, and 5.8%5.8\%, relative to the Modular Specialist (pHolm≤0.0012p_Holm≤ 0.0012). APE is statistically indistinguishable, whereas Vehicle ADE increases by 0.85m0.85\,m. Using one rather than two networks, PV-WM reduces the parameter count by 57.1%57.1\%, average FLOPs per local scene by 96.5%96.5\%, p95 latency by 25.5%25.5\%, and peak memory by 67.4%67.4\%. Qualitative results. Figure 3 shows close agreement in body configuration and relative P–V geometry in the first three contexts; the final near-interaction case exhibits late-horizon root and box displacement. IV-C Component and Extension Ablations Table I isolates recurrent execution, typed endpoint dynamics, persistent pair state, relational supervision, and pair-to-endpoint routing under generated-state rollout. TABLE I: PV-WM component and extension ablations. The main block uses generated-state continuation; static-context extensions share the same 339 map-present contexts and are ranked separately. One-shot prediction is non-recurrent and generated-context training is training-only. The dark row is the Table I configuration. Errors are in millimeters; bold marks the lowest value within each support block. Variant Design Pedestrian Vehicle P–V Geometry Rec. Typed Pair Geo. Q R→PR\!→\!P R→VR\!→\!V Gen. Ctx. Map Rule Root ADE/FDE (m)↓ MPJPE/APE (m)↓ ADE/FDE (m)↓ Cyclic BoxCorner (m)↓ Dist./DCAEOBBDCAE_OBB (m)↓ Reference configuration PV-WM ✓ ✓ G ✓ ✓ ✗ ✗ ✗ ✗ ✗ 165.2± 2.3/288.4± 6.9 188.8± 1.7/83.9± 2.2 68.9± 0.4/148.2± 1.6 329.4± 1.8 113.1± 0.2/186.6± 0.8 Core component ablations One-shot complete-state prediction ✗ ✓ O ✓ ✓ ✗ ✗ ✗ ✗ ✗ 189.3± 0.0/350.6± 0.1 221.6± 1.5/83.2± 1.6 57.3± 0.4/123.3± 0.4 430.7± 11.4 110.7± 0.0/214.9± 2.8 Shared endpoint update ✓ ✗ G ✓ ✓ ✗ ✗ ✗ ✗ ✗ 186.9± 1.4/343.4± 4.9 206.1± 3.6/91.6± 3.8 65.5± 0.6/141.9± 1.6 349.9± 6.5 111.6± 0.2/189.4± 2.7 w/o persistent pair state ✓ ✓ – ✓ ✓ ✗ ✗ ✗ ✗ ✗ 188.1± 0.2/347.4± 0.8 208.7± 2.0/91.0± 3.3 67.4± 0.6/145.6± 1.7 377.8± 8.2 112.1± 0.1/195.9± 0.8 w/o explicit geometry losses ✓ ✓ G ✗ ✓ ✗ ✗ ✗ ✗ ✗ 168.7± 3.7/298.8± 10.3 191.4± 3.1/85.6± 2.1 70.7± 0.7/149.1± 1.4 328.8± 7.0 113.2± 0.3/187.9± 0.4 w/o Q supervision ✓ ✓ G ✓ ✗ ✗ ✗ ✗ ✗ ✗ 171.6± 3.2/304.7± 7.0 192.9± 1.8/87.3± 2.2 67.7± 0.7/145.5± 0.7 336.3± 3.2 112.0± 0.3/187.6± 0.3 w/o all relational supervision ✓ ✓ G ✗ ✗ ✗ ✗ ✗ ✗ ✗ 166.8± 1.3/293.7± 4.7 190.2± 0.9/84.7± 1.5 70.4± 0.7/147.9± 1.2 336.1± 7.8 113.4± 0.1/190.5± 1.8 Pair-to-endpoint routing controls +R→PR\!→\!P routing ✓ ✓ G ✓ ✓ ✓ ✗ ✗ ✗ ✗ 190.5± 0.6/353.6± 1.0 216.7± 1.6/84.2± 3.0 65.3± 0.2/140.2± 1.4 368.1± 10.9 111.6± 0.0/196.5± 1.8 +R→VR\!→\!V routing ✓ ✓ G ✓ ✓ ✗ ✓ ✗ ✗ ✗ 168.2± 4.1/295.0± 9.5 191.3± 3.2/84.5± 2.5 69.3± 1.2/148.8± 3.2 373.6± 11.6 112.7± 0.4/194.8± 1.6 +R→P,VR\!→\!\P,V\ routing ✓ ✓ G ✓ ✓ ✓ ✓ ✗ ✗ ✗ 190.5± 0.4/353.7± 1.4 217.4± 1.3/83.4± 1.6 66.1± 0.8/141.3± 2.6 383.6± 16.6 111.5± 0.1/200.1± 1.9 Training extension PV-WM + generated-context training ✓ ✓ G ✓ ✓ ✗ ✗ ✓ ✗ ✗ 165.4± 2.9/289.3± 7.0 189.4± 3.2/82.4± 0.5 68.6± 0.3/147.9± 0.9 346.1± 14.3 112.9± 0.2/191.6± 3.0 Static-context extensions on 339 common-support contexts PV-WM on map support ✓ ✓ G ✓ ✓ ✗ ✗ ✗ ✗ ✗ 165.3± 2.1/292.5± 6.1 190.4± 1.7/85.9± 1.8 79.2± 0.3/175.7± 1.7 358.2± 4.5 114.6± 0.4/206.9± 1.2 PV-WM + MapCtx ✓ ✓ G ✓ ✓ ✗ ✗ ✗ ✓ ✗ 170.0± 2.2/305.4± 5.2 193.3± 1.8/88.0± 1.2 79.6± 1.6/176.1± 2.7 370.4± 5.3 114.6± 0.5/210.4± 0.7 PV-WM + MapCtx + SoftRule ✓ ✓ G ✓ ✓ ✗ ✗ ✗ ✓ ✓ 176.1± 1.0/320.3± 3.9 197.8± 0.5/89.4± 1.0 112.4± 2.0/499.7± 8.8 406.7± 8.1 139.7± 1.0/222.7± 3.8 Pair is observation-fixed (O), generated (G), or removed (–). Geo. combines R4, distance-channel, and horizon-minimum-distance losses; Q is auxiliary risk supervision. Arrows indicate pair-to-endpoint routing. Gen. Ctx. denotes generated-context training, a second pass that appends a detached two-frame generated prefix to the temporally shifted observation history before predicting the remaining horizon. MapCtx adds gated static context and SoftRule its proxy objective. Pedestrian metrics use 824 contexts; vehicle/P–V metrics draw from 797 contexts with valid future support and apply metric-specific masks; map rows use 339 contexts and are ranked separately. TABLE I: Utility and content specificity of articulated state. Panel (a) separates articulated prediction from recurrent feedback; bold marks the lowest displayed error and the gain row reports Rigid minus Stateful. Panel (b) reports final-horizon changes under feedback interventions and oracle feedback. All errors are in millimeters. (a) Representation and recurrent-state utility Variant Articulated State Pedestrian Vehicle P–V Geometry Readout Feedback Root ADE/FDE↓ MPJPE/fMPJPE↓ APE/wAPE↓ ADE↓ Dist./DCAEOBBDCAE_OBB↓ Rigid state ✗ ✗ 185.17± 0.92/337.98± 3.62 218.98± 0.88/364.59± 3.48 84.91± 0.00/228.40± 0.00 66.29± 0.28 112.09± 0.24/188.25± 1.04 Articulated readout ✓ ✗ 182.11± 5.25/329.13± 14.60 211.32± 5.76/354.42± 14.43 78.94± 1.17/200.84± 5.18 66.68± 1.44 112.23± 0.53/188.31± 3.17 Stateful articulation ✓ ✓ 179.15± 3.64/321.30± 8.42 207.86± 3.57/346.19± 8.12 77.25± 0.14/185.35± 0.38 67.76± 0.57 112.61± 0.09/186.82± 1.95 Gain: Rigid −- Stateful +6.02/+16.68 +11.13/+18.40 +7.67/+43.05 −1.48-1.48 −0.52/+1.42n.s.-0.52/+1.42^n.s. (b) Feedback content specificity Intervention Tests Final-horizon error change (m) Δ Δ @1s Rigidized Content +2.79 +33.94 Time shuffle Time +2.72 +33.66 Identity shuffle† Identity +33.63 +32.54 Oracle GT Headroom -14.06 -101.51 fMPJPE is final-frame MPJPE; wAPE weights root-relative error by future-target body motion. GT denotes the reconstructed future target. For Stateful versus Rigid in panel (a), the P–V Distance MAE increase is Holm-significant; n.s. marks the nonsignificant DCAEOBBDCAE_OBB contrast. Panel (b) reports changes from factual feedback; positive values indicate degradation and negative values oracle headroom. Identity shuffling applies to 717 eligible multi-pedestrian contexts; the others retain factual feedback, yielding the same 824-context aggregation. Other interventions apply to all 824 contexts. First-chunk predictions are identical, and all displayed contrasts in panel (b) are Holm-significant. Recurrent and typed execution. Relative to the matched one-shot control, PV-WM lowers Root ADE, MPJPE, DCAEOBBDCAE_OBB, and Cyclic BoxCorner by 24.124.1, 32.832.8, 28.328.3, and 101.3m101.3\,m, respectively. The one-shot control yields lower APE, Vehicle ADE/FDE, and framewise P–V Distance MAE. Replacing typed endpoint updates with a shared transition increases Root ADE, MPJPE, DCAEOBBDCAE_OBB, and Cyclic BoxCorner by 21.721.7, 17.317.3, 2.82.8, and 20.5m20.5\,m; removing the persistent pair state increases them by 22.922.9, 19.919.9, 9.39.3, and 48.4m48.4\,m. Relational learning and routing Removing the full relational objective increases DCAEOBBDCAE_OBB by 3.9m3.9\,m after Holm correction. Removing only explicit geometry losses changes DCAEOBBDCAE_OBB by +1.3m+1.3\,m and Cyclic BoxCorner by −0.6m-0.6\,m; only the full relational objective yields a Holm-significant DCAEOBBDCAE_OBB effect. Routing pair context to both endpoint branches increases Root ADE, MPJPE, Cyclic BoxCorner, and DCAEOBBDCAE_OBB by 25.325.3, 28.628.6, 54.254.2, and 13.5m13.5\,m, while reducing APE, Vehicle ADE, and framewise P–V Distance MAE. Taken together, these controls separate three roles: generated boundaries close the recurrent loop, typed updates preserve heterogeneous dynamics, and the persistent pair state carries relational context while the endpoint branches remain factorized. Training and context extensions Generated-context training trades small local-pose and vehicle-center gains for 16.7m16.7\,m higher Cyclic BoxCorner and 5.0m5.0\,m higher DCAEOBBDCAE_OBB; PV-WM therefore uses the base training objective. On the 339 map-present contexts, neither MapCtx nor SoftRule improves the map-free reference. IV-D Utility of Articulated Pedestrian State We separate articulated output, structured supervision, and recurrent state reuse under the default factorized endpoint dynamics of PV-WM. Rigid State predicts root motion while repeating the final observed root-relative pose. Articulated Readout predicts and supervises future pose without feeding it into later chunks. Stateful Articulation additionally re-encodes generated pose into the next recurrent boundary. This independently trained suite uses the same folds and seeds, a fixed 30-epoch budget, and a common checkpoint criterion that excludes pose metrics; comparisons are therefore made within this study. On fixed Stateful Articulation checkpoints, interventions modify root-relative pose feedback at each chunk boundary before latent updates and boundary refresh. Rigidization repeats the last observed local pose; time shuffling swaps local poses between the two chunk frames; identity shuffling cyclically permutes local poses among valid pedestrians within each local scene; oracle feedback substitutes reference local poses. Each edit preserves the predicted roots and vehicle states at that boundary and leaves the decoded chunk unchanged. Stateful Articulation reduces Root ADE and FDE by 6.026.02 and 16.68m16.68\,m, MPJPE and final-frame MPJPE by 11.1311.13 and 18.40m18.40\,m, and APE and motion-weighted APE by 7.677.67 and 43.05m43.05\,m relative to Rigid State. All six effects have the same direction across the five folds and remain significant after Holm correction. Vehicle ADE and P–V Distance MAE increase by 1.481.48 and 0.52m0.52\,m, respectively; the DCAEOBBDCAE_OBB difference is not significant. Articulated Readout reduces MPJPE by 7.66m7.66\,m and wAPE by 27.56m27.56\,m relative to Rigid State; recurrent feedback provides a further 3.46m3.46\,m MPJPE and 15.49m15.49\,m wAPE improvement. Rigidizing or time-shuffling the feedback increases final-frame MPJPE by 2.792.79 and 2.72m2.72\,m, and endpoint wAPE by 33.9433.94 and 33.66m33.66\,m. Identity shuffling increases final-frame MPJPE by 33.63m33.63\,m, while oracle feedback retains 14.06m14.06\,m fMPJPE and 101.51m101.51\,m wAPE@1s headroom. Because first-chunk predictions are identical, later transitions depend on the content, temporal order, and identity of generated articulation. V Conclusion We introduced PV-WM, a heterogeneous micro–macro world model that recurrently advances articulated pedestrian and learned vehicle states, reconstructs oriented boxes, and synchronizes P–V geometry after each transition. Ablations identify recurrent typed execution and the persistent pair branch as the main architectural contributors, while relational supervision improves closest-approach accuracy. Feedback interventions show that subsequent transitions functionally reuse generated articulation. On the segment-grouped Waymo evaluation, single-network PV-WM improves pedestrian transport, full-body accuracy, and P–V geometry over a validation-selected Modular Specialist while reducing parameters by 57.1%57.1\% and average FLOPs per local scene by 96.5%96.5\%. PV-WM demonstrates that a recurrent world model can jointly advance articulated pedestrian and rigid-vehicle states while preserving type-specific dynamics and synchronized geometry. ACKNOWLEDGMENT The schematic local-scene illustration in the upper-left of Fig. 2 was generated with the image-generation capability of OpenAI ChatGPT and subsequently edited by the authors; it contains no experimental data. References [1] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp, C. R. Qi, Y. Zhou, Z. Yang, A. Chouard, P. Sun, J. Ngiam, V. Vasudevan, A. McCauley, J. Shlens, and D. Anguelov (2021) Large scale interactive motion forecasting for autonomous driving: the waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 9710–9719. Cited by: §I, §I-A. [2] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone (2020) Trajectron++: dynamically-feasible trajectory forecasting with heterogeneous data. In Computer Vision – ECCV 2020, A. Vedaldi, H. Bischof, T. Brox, and J. Frahm (Eds.), Cham, p. 683–700. External Links: ISBN 978-3-030-58523-5 Cited by: §I, §I-A, §IV-A, TABLE I. [3] J. Ngiam, V. Vasudevan, B. Caine, Z. Zhang, H. L. Chiang, J. Ling, R. Roelofs, A. Bewley, C. Liu, A. Venugopal, D. J. Weiss, B. Sapp, Z. Chen, and J. Shlens (2022) Scene transformer: a unified architecture for predicting future trajectories of multiple agents. In International Conference on Learning Representations, External Links: Link Cited by: §I, §I-A. [4] N. Nayakanti, R. Al-Rfou, A. Zhou, K. Goel, K. S. Refaat, and B. Sapp (2023) Wayformer: motion forecasting via simple & efficient attention networks. In 2023 IEEE International Conference on Robotics and Automation (ICRA), Vol. , p. 2980–2987. External Links: Document Cited by: §I, §I-A. [5] S. Shi, L. Jiang, D. Dai, and B. Schiele (2022) Motion transformer with global intention localization and local movement refinement. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 6531–6543. External Links: Link Cited by: §I, §I-A, §IV-A, TABLE I. [6] Z. Zhou, J. Wang, Y. Li, and Y. Huang (2023) Query-centric trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 17863–17873. Cited by: §I, §I-A, §IV-A, TABLE I. [7] A. Seff, B. Cera, D. Chen, M. Ng, A. Zhou, N. Nayakanti, K. S. Refaat, R. Al-Rfou, and B. Sapp (2023) MotionLM: multi-agent motion forecasting as language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 8579–8590. Cited by: §I, §I-A. [8] Z. Huang, H. Liu, and C. Lv (2023) GameFormer: game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 3903–3913. Cited by: §I, §I-A. [9] X. Peng, S. Mao, and Z. Wu (2023) Trajectory-aware body interaction transformer for multi-person pose forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 17121–17130. Cited by: §I, §I-B, §IV-A, TABLE I. [10] J. Jeong, D. Park, and K. Yoon (2024) Multi-agent long-term 3d human pose forecasting via interaction-aware trajectory conditioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 1617–1628. Cited by: §I, §I-B, §IV-A, TABLE I. [11] G. Zhu, S. Fan, H. Dai, and E. S. L. Ho (2025) Waymo-3dskelmo: a multi-agent 3d skeletal motion dataset for pedestrian interaction modeling in autonomous driving. In Proceedings of the 33rd ACM International Conference on Multimedia, M ’25, New York, NY, USA, p. 13184–13190. External Links: ISBN 9798400720352, Link, Document Cited by: §I, §I-B, §IV-A. [12] G. Zhu, X. Liu, N. Pugeault, C. Wei, and E. S. L. Ho (2026) Modeling 3d pedestrian-vehicle interactions for vehicle-conditioned pose forecasting. External Links: 2602.08962, Link Cited by: §I, §I-B, §IV-A, TABLE I. [13] Z. Zhang, A. Liniger, D. Dai, F. Yu, and L. Van Gool (2023) TrafficBots: towards world models for autonomous driving simulation and motion prediction. In 2023 IEEE International Conference on Robotics and Automation (ICRA), Vol. , p. 1522–1529. External Links: Document Cited by: §I, §I-C. [14] W. Zheng, W. Chen, Y. Huang, B. Zhang, Y. Duan, and J. Lu (2024) OccWorld: learning a 3d occupancy world model for autonomous driving. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part XIII, Berlin, Heidelberg, p. 55–72. External Links: ISBN 978-3-031-72623-1, Link, Document Cited by: §I, §I-C. [15] C. Min, D. Zhao, L. Xiao, J. Zhao, X. Xu, Z. Zhu, L. Jin, J. Li, Y. Guo, J. Xing, L. Jing, Y. Nie, and B. Dai (2024) DriveWorld: 4d pre-trained scene understanding via world models for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 15522–15533. Cited by: §I, §I-C. [16] Y. Wang, J. He, L. Fan, H. Li, Y. Chen, and Z. Zhang (2024) Driving into the future: multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 14749–14759. Cited by: §I, §I-C. [17] K. Zhang, Z. Tang, X. Hu, X. Pan, X. Guo, Y. Liu, J. Huang, L. Yuan, Q. Zhang, X. Long, X. Cao, and W. Yin (2025) Epona: autoregressive diffusion world model for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 27220–27230. Cited by: §I, §I-C. [18] H. Chi, D. Qiu, H. Su, H. Liu, Z. Li, H. Zhang, and C. Lv (2026) Driver-wm: a driver-centric traffic-conditioned latent world model for in-cabin dynamics rollout. External Links: 2605.05092, Link Cited by: §I, §I-C. [19] Q. Sun, X. Huang, J. Gu, B. C. Williams, and H. Zhao (2022) M2I: from factored marginal trajectory prediction to interactive prediction. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 6533–6542. External Links: Document Cited by: §I-A. [20] L. Rowe, M. Ethier, E. Dykhne, and K. Czarnecki (2023) FJMP: factorized joint multi-agent motion prediction over learned directed acyclic interaction graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I-A. [21] H. Bi, Z. Fang, T. Mao, Z. Wang, and Z. Deng (2019) Joint prediction for kinematic trajectories in vehicle-pedestrian-mixed scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Cited by: §I-A. [22] D. Ha and J. Schmidhuber (2018) World Models. arXiv e-prints, p. arXiv:1803.10122. External Links: Document, 1803.10122 Cited by: §I-C. [23] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap (2024) Mastering diverse domains through world models. External Links: 2301.04104, Link Cited by: §I-C. [24] A. Hu, L. Russell, H. Yeo, Z. Murez, G. Fedoseev, A. Kendall, J. Shotton, and G. Corrado (2023) GAIA-1: a generative world model for autonomous driving. External Links: 2309.17080, Link Cited by: §I-C. [25] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam, H. Zhao, A. Timofeev, S. Ettinger, M. Krivokon, A. Gao, A. Joshi, Y. Zhang, J. Shlens, Z. Chen, and D. Anguelov (2020) Scalability in Perception for Autonomous Driving: Waymo Open Dataset . In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , Los Alamitos, CA, USA, p. 2443–2451. External Links: ISSN , Document, Link Cited by: §IV-A. [26] M. Prédhumeau, L. Mancheva, J. Dugdale, and A. Spalanzani (2021) An agent-based model to predict pedestrians trajectories with an autonomous vehicle in shared spaces. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’21, Richland, SC, p. 1010–1018. External Links: ISBN 9781450383073 Cited by: §IV-A. [27] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas (2018) Frustum pointnets for 3d object detection from rgb-d data. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vol. , p. 918–927. External Links: Document Cited by: §IV-A.