Paper deep dive
RiCo: Neural Simulation of Rigid-Body Interactions via Local Contact Reasoning
Ruixiang Ouyang, Guanren Qiao, Fansen Meng, Yueci Deng, Ruixing Jin, Kui Jia, Guiliang Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/11/2026, 2:03:45 AM
Summary
The paper introduces RiCo (Rigid-body Contact Reasoning), a neural simulator for rigid-body interactions that models contact forces through sparse neighborhoods of surface points rather than dense object-level attention. RiCo constructs local contact features based on geometry, motion, and physical properties, then uses intra-object point transformers to reason about how these local contacts affect global motion. It predicts motion updates at anchor points to recover rigid transformations. Experiments on the MOVi benchmark show significant reductions in position and orientation errors compared to baselines, along with high contact fidelity and zero-shot generalization to scenes with 270 objects.
Entities (8)
Relation Signals (6)
RiCo → developedby → Chinese University of Hong Kong, Shenzhen
confidence 95% · Ruixiang Ouyang Affiliation: The Chinese University of Hong Kong, Shenzhen
RiCo → developedby → DexForce Co., Ltd.
confidence 95% · Ruixiang Ouyang Affiliation: DexForce Co., Ltd.
RiCo → evaluatedon → MOVi-benchmark
confidence 95% · Experiments on MOVi-benchmark demonstrate that RiCo reduces 100-frame position and orientation errors
MOVi-benchmark → createdby → Greff et al.
confidence 90% · We conduct extensive experiments on the MOVi benchmark (Greff et al., 2022)
RiCo → outperforms → RigidFormer
confidence 90% · RiCo reduces 100-frame position and orientation errors by 31-35% and approximately 38%, respectively, compared with baselines.
RiCo → outperforms → HOPNet
confidence 90% · RiCo reduces 100-frame position and orientation errors by 31-35% and approximately 38%, respectively, compared with baselines.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Accurate simulation of rigid-body interactions is essential for predictive physical world models. Despite recent progress in modeling object dynamics, capturing how local contacts between surfaces shape object motion remains challenging. While end-to-end world models predict interactions across entire scenes or objects, in practice, rigid-body contact is inherently local, and only nearby surfaces can directly exchange contact forces. Motivated by this observation, we introduce Rigid-body Contact Reasoning (RiCo), which represents interactions between objects through sparse neighborhoods of contact surface points. RiCo combines each point's state with the relative geometry, motion, and physical properties of nearby surfaces, then reasons across the object's points to determine how these local contacts jointly affect its motion. By confining cross-object reasoning to nearby surfaces while propagating contact information within each rigid body, RiCo retains fine-grained interaction details without the cost of modeling every pair of scene points. Such properties enable RiCo a higher accuracy and contact fidelity. Experiments on MOVi-benchmark demonstrate that RiCo reduces 100-frame position and orientation errors by 31-35% and approximately 38%, respectively, compared with baselines. Moreover, RiCo achieves high contact fidelity, with ground-truth-relative penetration-time and mean-depth differences of 11.0% and 2.22 mm, respectively. RiCo further generalizes zero-shot from small-scale training scenarios to scenes containing 270 objects. Our real-world multi-ball collision experiments further provide preliminary evidence of sim-to-real transfer.
Tags
Links
- Source: https://arxiv.org/abs/2610.12333v1
- Canonical: https://arxiv.org/abs/2610.12333v1
Trouble viewing inline? Open PDF directly →
Full Text
83,614 characters extracted from source content.
Expand or collapse full text
RiCo: Neural Simulation of Rigid-Body Interactions via Local Contact Reasoning Ruixiang Ouyang Affiliation: The Chinese University of Hong Kong, Shenzhen Affiliation: DexForce Co., Ltd. Guanren Qiao Affiliation: The Chinese University of Hong Kong, Shenzhen Fansen Meng Affiliation: South China University of Technology Yueci Deng Affiliation: The Chinese University of Hong Kong, Shenzhen Affiliation: DexForce Co., Ltd. Ruixing Jin Affiliation: The Chinese University of Hong Kong, Shenzhen Kui Jia Affiliation: The Chinese University of Hong Kong, Shenzhen Affiliation: DexForce Co., Ltd. Guiliang Liu †thanks: Corresponding author: Guiliang Liu, liuguiliang@cuhk.edu.cn Affiliation: The Chinese University of Hong Kong, Shenzhen Affiliation: Shenzhen Loop Area Institute Abstract Accurate simulation of rigid-body interactions is essential for predictive physical world models. Despite recent progress in modeling object dynamics, capturing how local contacts between surfaces shape object motion remains challenging. While end-to-end world models predict interactions across entire scenes or objects, in practice, rigid-body contact is inherently local, and only nearby surfaces can directly exchange contact forces. Motivated by this observation, we introduce Rigid-body Contact Reasoning (RiCo), which represents interactions between objects through sparse neighborhoods of contact surface points. RiCo combines each point’s state with the relative geometry, motion, and physical properties of nearby surfaces, then reasons across the object’s points to determine how these local contacts jointly affect its motion. By confining cross-object reasoning to nearby surfaces while propagating contact information within each rigid body, RiCo retains fine-grained interaction details without the cost of modeling every pair of scene points. Such properties enable RiCo a higher accuracy and contact fidelity. Experiments on MOVi-benchmark demonstrate that RiCo reduces 100-frame position and orientation errors by 31–35% and approximately 38%, respectively, compared with baselines. Moreover, RiCo achieves high contact fidelity, with ground-truth-relative penetration-time and mean-depth differences of 11.0% and 2.22 m, respectively. RiCo further generalizes zero-shot from small-scale training scenarios to scenes containing 270 objects. Our real-world multi-ball collision experiments further provide preliminary evidence of sim-to-real transfer. 1 Introduction High-fidelity modeling of the physical world has become a central milestone in embodied AI, enabling agents to reason about how environments evolve and respond to their interactions (Agarwal et al., 2025). Recent advances in world models have learned predictive representations from large-scale video observations (Bruce et al., 2024; Ye et al., 2026; Li et al., 2026). Beyond these visual patterns, however, the physical world exhibits complex dynamics involving spatial motion, object geometry, and contact interactions (Hafner et al., 2023), which remain difficult to capture with video generation backbones. To advance the modeling of dynamic objects, we study neural simulation of rigid-body motions, including translation, rotation, and interactions among multiple objects in 3D space (Chang et al., 2016; Wei and Fink, 2025). As a fundamental abstraction of physical object motion, rigid-body dynamics exclude continuous deformation of objects, but introduce a significant challenge in modelling their interactions under different geometric form and motion conditions, because collisions introduce abrupt changes in linear and angular velocities, making contact dynamics non-smooth and challenging to predict accurately (Pfaff et al., 2020; Wei and Fink, 2025). To address these challenges, prior works proposed learning neural approximations of motion dynamics from observed trajectories (Hafner et al., 2019; Brandstetter et al., 2022; Wu et al., 2023). Despite their success across various domains, modeling rigid-body dynamics remains challenging because collisions may induce abrupt changes in objects’ velocities and subsequent motion trajectories. To better capture these discontinuities, recent studies incorporate geometric priors to explicitly model contact interactions. For instance, mesh-based neural simulators exploit mesh connectivity and surface geometry to propagate information about physical interactions (Pfaff et al., 2020; Allen et al., 2023). HOPNet (Wei and Fink, 2025) further incorporates higher-order topological structures to improve the modeling of rigid-body contact dynamics. However, processing mesh connectivity and constructing high-order geometric relationships can be computationally expensive and time-consuming. Therefore, RigidFormer (Dou et al., 2026) represents individual objects using point sets and models their interactions through object-level attention, enabling rigid-body simulation without explicit mesh connectivity. While point clouds can efficiently capture surface geometry, they do not inherently encode which surface points may come into contact during object interactions. (a) Mesh-based (b) RigidFormer (c) Ours Figure 1: Comparing rigid body contact modeling. Striving for efficient and fine-grained contact modeling, we propose RiCo which identifies and models local contact relationships directly from point-cloud representations, without relying on mesh connectivity. Figure 1 shows an illustrative example. This approach reflects the local nature of rigid-body interactions: only nearby surfaces can directly exchange contact forces (Sanchez-Gonzalez et al., 2020). Accordingly, instead of modeling the dense interactions among all objects, RiCo identifies only potential contact points and models their local dynamics across interacting objects. Because the combined effects of these contacts can govern rigid-body motion, we reason over contact information by propagating the effects of local contacts across rigid body. The intra-object contact reasoning provides a principled link between fine-grained interactions and whole-object motion. This design not only supports generalization across diverse shapes but also scales efficiently as the number of surface points grows. To implement RiCo, we build a neural simulator that decomposes object-level collisions into sparse local contacts and intra-object contact reasoning. For each surface point, RiCo constructs a compact contact neighborhood containing nearby points from other objects and the environment. It then combines the source point’s state with the relative geometry, motion, and physical properties of these contact candidates to form contact-conditioned point features. Based on such features, a point transformer attends each surface point to the others on the target object, so the model can reason where contacts occur, how the contacting surfaces move, and how forces at different locations affect translation and rotation. Based on these contact-conditioned features, an anchor-based decoder predicts motion updates at a small set of surface points, from which a single rigid transformation is recovered and applied to the entire object. This design integrates fine-grained geometric information into contact-dynamics reasoning, thereby generating physically plausible rigid-body motion. We conduct extensive experiments on the MOVi benchmark (Greff et al., 2022) to evaluate RiCo in terms of prediction accuracy, generalization, contact fidelity, scalability, and real-world dynamics. Across diverse object configurations and long-horizon rollouts, RiCo reduces position and orientation errors by up to 35% and 38%, respectively. Beyond conventional trajectory metrics, we assess contact fidelity by comparing the predicted and ground-truth trajectories’ aggregate penetration-time ratios and mean penetration depths, thereby accounting for geometric overlap already present in the ground-truth trajectories. Under this metric, RiCo reduces relative contact violations to 11.0% of object-time steps, with an average relative penetration depth of 2.22 m. Moreover, RiCo generalizes effectively across unseen distributions of 270 objects. We further investigate zero-shot sim-to-real transfer through real-world multi-ball billiard collision experiments, providing preliminary evidence that the learned contact dynamics can generalize to real-world multi-object interactions without additional training. These results demonstrate that local contact reasoning enables accurate, contact-fidelity, and scalable neural simulation of complex rigid-body dynamics. 2 Related work Learned Models of Physical Dynamics. Learned world models predict future environment states from observations, ranging from latent dynamics models (Ha and Schmidhuber, 2018; Hafner et al., 2019) to feature-space predictive approaches such as I-JEPA and DINO-WM (Assran et al., 2023; Zhou et al., 2024). Recent work adopts explicit 3D representations for embodied prediction: PointWorld (Huang et al., 2026) models action-conditioned scene evolution with 3D point flows, while RoboFlow4D (Lin et al., 2026) predicts multi-frame 3D motion fields for robotic manipulation. Despite this progress, rigid-body contact remains challenging because interactions can induce abrupt changes in global object motion, resulting in discontinuous dynamics that are difficult for smooth learned predictors to capture (Allen et al., 2023). Geometric Representations for Rigid-Body Simulation. Early learned physics models introduced inductive biases for predicting physical interactions (Battaglia et al., 2016; Sanchez-Gonzalez et al., 2018), which were later extended to particle-based simulators for objects and fluids (Li et al., 2018; Sanchez-Gonzalez et al., 2020). Mesh-based approaches further exploit explicit geometric structure by performing message passing over simulation meshes (Pfaff et al., 2020). For rigid-body contact dynamics, FIGNet (Allen et al., 2023) constructs interactions between mesh faces, preserving local surface geometry around contacts. Recent mesh-based models have explored hierarchical structures for propagating collision effects, while HOPNet (Wei and Fink, 2025) explicitly represents vertices, edges, mesh triangles, contacts, and objects through higher-order topological relations. To relax the dependence on meshes, SDF-Sim (Rubanova et al., 2024) represents object geometry using learned signed-distance functions, whereas RigidFormer (Dou et al., 2026) removes mesh connectivity entirely and operates on point-cloud inputs. Our method follows the mesh-free setting, while retaining point-level geometry for sparse local contact reasoning rather than compressing interactions primarily into coarse object-level representations. 3 Method In this section, we formulate the problem of modeling rigid-body contact dynamics and introduce Rigid-body Contact Reasoning (RiCo), our neural simulator designed to solve the problem. Problem Formulation. To model the rigid-body dynamics, we consider a scene containing M rigid objects, where M may vary across scenes. At time t, object i is represented by NiN_i sampled surface points ti=[xt,1i,…,xt,Nii]⊤∈ℝNi×3x_t^i=[x_t,1^i,…,x_t,N_i^i] ^N_i× 3, with corresponding surface normals νtiν_t^i, center of mass ctic_t^i, and physical properties ϕi=[(mi)−1,μi,ei] φ^i=[(m^i)^-1,μ^i,e^i]. We denote the full scene state by t=ti,νti,ctii=1Ms_t= \ x_t^i,ν_t^i,c_t^i \_i=1^M. Given two consecutive states st−1s_t-1 and sts_t, together with the static environment ξ, our goal is to build a dynamics model fθf_θ (i.e., state transition model) to predict s^t+1=fθ(st−1,st,ϕ,ξ). s_t+1=f_θ(s_t-1,s_t, φ,ξ). Appendix A introduces more details. To effectively approximate fθf_θ, RiCo models the state update in three stages: (1) constructing a sparse representation of local contacts, (2) reasoning over contacts within each object, and (3) predicting rigid-body motion from anchors. Figure 2 provides an overview of the framework. Figure 2: Overview of RiCo. From two consecutive scene states, RiCo constructs sparse local contact neighborhoods, fuses them with point-wise object states, and performs shared intra-object reasoning with a point transformer. The resulting contact-conditioned features are decoded at a small set of anchors to recover a rigid transformation for each object, and the predicted state is recursively used for autoregressive rollout. 3.1 Sparse Contact Neighborhood Representation Contact neighborhood construction. Striving for concise representation of contact dynamics, we construct a sparse local interaction representation around each surface point, rather than modeling interactions over complete mesh-level or object-level representations (Wei and Fink, 2025; Dou et al., 2026). Specifically, we first perform an object-level broad-phase query over active object pairs using their surface-point AABBs, retaining only pairs whose minimum AABB separation is no greater than the contact radius ρ. More details can be found in appendix B.2. Candidates from all retained objects and the static environment ξ are then combined, from which the K nearest candidates within ρ are retained. Dynamic-object candidates therefore correspond to sampled surface points rather than closest points on continuous meshes, while static planar surfaces are queried analytically by projection. We denote the candidate positions by yt,n,kik=1K\y_t,n,k^i\_k=1^K11 1 If fewer than K valid candidates exist, the remaining entries are padded and masked during subsequent aggregation; when no valid candidate exists, the entire contact neighborhood is masked and contributes zero contact information.. To characterize the potential contact response, we gather each candidate’s surface normal ν~t,n,ki ν_t,n,k^i, finite-difference displacement Δyt,n,ki y_t,n,k^i, and physical properties ϕ~t,n,ki φ_t,n,k^i, comprising the inverse mass, friction coefficient, and coefficient of restitution of the corresponding external object or surface. Additionally, a binary indicator σt,n,ki _t,n,k^i distinguishes dynamic geometry (σ=0σ=0) from static objects (σ=1σ=1), with static candidates assigned zero displacement and zero inverse mass. We combine these quantities with the relative geometry and motion between the source point xt,nix_t,n^i and each external candidate yt,n,kiy_t,n,k^i to form a 14-dimensional interaction descriptor: ηt,n,ki=[ut,n,ki,δt,n,ki,ν~t,n,ki,Δyt,n,ki−Δxt,ni,ϕ~t,n,ki,σt,n,ki]∈ℝ14, _t,n,k^i= [u_t,n,k^i,\; _t,n,k^i,\; ν_t,n,k^i,\; y_t,n,k^i- x_t,n^i,\; φ_t,n,k^i,\; _t,n,k^i ] ^14, (1) whereut,n,ki=yt,n,ki−xt,ni‖yt,n,ki−xt,ni‖2+ε,δt,n,ki=(xt,ni−yt,n,ki)⊤ν~t,n,ki.\;u_t,n,k^i= y_t,n,k^i-x_t,n^i \|y_t,n,k^i-x_t,n^i \|_2+ ,\; _t,n,k^i= (x_t,n^i-y_t,n,k^i ) ν_t,n,k^i.Within this formulation, ut,n,kiu_t,n,k^i encodes the direction from the source point to the candidate, with ε>0 >0 ensuring numerical stability. The relative displacement Δyt,n,ki−Δxt,ni y_t,n,k^i- x_t,n^i captures approach or separation and tangential sliding, while ϕ~t,n,ki φ_t,n,k^i specifies the physical properties of the external object. We also introduce an estimated signed distance δt,n,ki _t,n,k^i, which measures the signed offset of the source point from the local tangent plane on the candidate surface. Unlike the unsigned distance, its sign explicitly distinguishes the two sides of the local surface, providing a simple geometric cue to identify the potential penetration. Point State Representation. To model how the source object responds to local interactions, we complement the contact descriptors with a point-wise state representation of its own geometry, motion, and physical properties. For point n on object i, we define this state as st,ni=[xt,ni−cti,Δxt,ni,νt,ni,ϕi,xt,ni−x0,ni]∈ℝ15.s_t,n^i= [x_t,n^i-c_t^i,\; x_t,n^i,\; _t,n^i,\;φ^i,\;x_t,n^i-x_0,n^i ] ^15. Within this representation, xt,ni−ctix_t,n^i-c_t^i locates the point relative to the object’s center of mass. The finite-difference displacement Δxt,ni=xt,ni−xt−1,ni x_t,n^i=x_t,n^i-x_t-1,n^i captures its motion over the preceding time step, while νt,ni _t,n^i describes the local surface normal. The object-level properties ϕi=[(mi)−1,μi,ei]φ^i=[(m^i)^-1,μ^i,e^i] specify the inverse mass, friction coefficient, and restitution coefficient that govern contact response. Finally, xt,ni−x0,nix_t,n^i-x_0,n^i records the point’s displacement from its initial position, encoding its accumulated motion relative to the reference configuration. Together, the point state st,nis_t,n^i and the local interaction descriptors ηt,n,kik=1K\ _t,n,k^i\_k=1^K capture the source point’s own geometry, motion, and physical properties alongside those of nearby external surfaces. 3.2 Intra-Object Contact Reasoning While contact neighborhoods identify potential interactions between nearby surfaces, predicting the resulting rigid-body motion requires reasoning jointly about contacts across the object’s surface. We therefore fuse local contact descriptors with each point’s state to form contact-conditioned tokens, then use a shared point Transformer to integrate these tokens within each object. Contact Feature Aggregation. We embed the point state st,nis_t,n^i and each local interaction descriptor ηt,n,ki _t,n,k^i using MLPs ϕs _s and ϕc _c, respectively. To prioritize nearby surfaces while retaining information from multiple contact candidates, we aggregate the interaction embeddings ϕc(ηt,n,ki) _c( _t,n,k^i) using distance-dependent weights and fuse them with the point state embedding ϕs(st,ni) _s(s_t,n^i). For the valid candidate set t,ni⊆1,…,KC_t,n^i \1,…,K\, we define ht,ni,0=RMSNorm(ϕs(st,ni)+∑k∈t,niαt,n,kiϕc(ηt,n,ki)), h_t,n^i,0=RMSNorm\! ( _s(s_t,n^i)+ _k _t,n^i _t,n,k^i _c( _t,n,k^i) ), (2) whereαt,n,ki=exp(−dt,n,ki/τ)∑l∈t,niexp(−dt,n,li/τ),anddt,n,ki=‖t,n,ki−t,ni‖2.\; _t,n,k^i= (-d_t,n,k^i/τ) _l _t,n^i (-d_t,n,l^i/τ),\;and\;d_t,n,k^i=\|y_t,n,k^i-x_t,n^i\|_2. Here, dt,n,kid_t,n,k^i is the Euclidean distance, and τ>0τ>0 is a learnable temperature controlling the concentration of the weights 22 2 The weights are normalized over valid candidates only, while points with no valid candidate receive a zero contact embedding. The fused representation is then normalized using RMSNorm, yielding ht,ni,0h_t,n^i,0 for subsequent intra-object reasoning.. Object-Wide Contact Integration. Predicting an object’s response to contact requires reasoning about its interactions with surrounding objects and static surfaces. Although these interactions are encoded locally, their combined effect on rigid-body motion depends on contact conditions across the object’s surface. As a result, for each object i, we collect the contact-conditioned point tokens as ti,0=[ht,1i,0,…,ht,Ni,0] h_t^i,0=[h_t,1^i,0,…,h_t,N^i,0] and process them independently using a shared B-block point Transformer θT_θ. Each block performs gated multi-head self-attention followed by a feed-forward network. For a single attention head in block b, we compute: Attnθb(ti,b) _ _b( h_t^i,b) =[ti,b⊙softmax((ti,bWQb)(ti,bWKb)⊤dattn)ti,bWVb]WOb, = [ g_t^i,b \! ( ( h_t^i,bW_Q^b)( h_t^i,bW_K^b) d_attn ) h_t^i,bW_V^b ]W_O^b, (3) where ti,b h_t^i,b denotes the point tokens of object i at block b, and ti,b=σ(ti,bWGb) g_t^i,b=σ\! ( h_t^i,bW_G^b ) is a learned sigmoid gate that modulates the attended features before output projection. WQbW_Q^b, WKbW_K^b, WVbW_V^b, WGbW_G^b, and WObW_O^b denote the query, key, value, gate, and output projections, respectively, and dattnd_attn is the query/key dimension of each head. After B blocks, the resulting point representations are denoted by ~ti=θ(ti,0) h_t^i=T_θ( h_t^i,0). The Transformer parameters are shared across objects, while all queries, keys, and values are derived exclusively from the point tokens of the same object. Importantly, since information from external surfaces is incorporated through the local contact fusion in Eq. 2, intra-object self-attention allows each point to access contact information from across its own object without dense cross-object attention. By restricting cross-object information exchange to sparse local contact neighborhoods, our method avoids dense point-level interactions across the entire scene. 3.3 Anchor-Based Rigid Motion Prediction and Training Anchor-based Motion Prediction. As a contact-aware neural simulator, RiCo predicts each rigid object’s future motion from its contact-conditioned tokens HtiH_t^i. Since all surface points move under the same rigid transformation, predicting their future positions separately is often unnecessary. To preserve rigidity, we follow prior anchor-based formulations (Dou et al., 2026) and select A anchor samples for each object using deterministic farthest point sampling (FPS) (Qi et al., 2017). The corresponding sample indices naia=1A\n_a^i\_a=1^A remain fixed throughout the rollout, such that the anchors track the same surface samples over time. We predict motion updates only at these anchors and use them to recover the rigid transformation of the entire object. Δ2^t,anci=ω(ti,~ti)wherezt,ai=h~t,naii+ϕq([xt,naii−cti,Δxt,naii]) ^2 x_t,anc^i=D_ω\! ( z_t^i, h_t^i )\;where\;z_t,a^i= h_t,n_a^i^i+ _q\! ( [x_t,n_a^i^i-c_t^i,\; x_t,n_a^i^i ] ) (4) Here, ti z_t^i collects the A anchor queries, and ωD_ω denotes the anchor-to-point attention decoder, including its output projection to second-order anchor displacements. By attending to all point features of its object, each anchor integrates contact information from across the surface to predict a motion update conditioned on its own position and recent displacement. To advance the anchors from their current motion, we use the predicted second-order displacements Δ2^t,anci ^2 x_t,anc^i as learned corrections to a constant-displacement extrapolation, yielding unconstrained next positions through a Verlet-style update: x~t+1,naii=2xt,naii−xt−1,naii+[Δ2^t,anci]a x_t+1,n_a^i^i=2x_t,n_a^i^i-x_t-1,n_a^i^i+ [ ^2 x_t,anc^i ]_a Here, [⋅]a[·]_a extracts the a-th row, which contains the predicted second-order displacement of the anchor indexed by nain_a^i. To recover a rigid transformation for the object, we align the current anchor positions with their predicted positions using the Kabsch algorithm (Kabsch, 1976) to obtain the recoverd rotation and translation (R^ti,b^ti)( R_t^i, b_t^i). We apply them to every surface point and the center of mass, while updating surface normals using only the rotation: x^t+1,ni=R^tixt,ni+b^ti x_t+1,n^i= R_t^ix_t,n^i+ b_t^i, ν^t+1,ni=R^tiνt,ni ν_t+1,n^i= R_t^i _t,n^i, c^t+1i=R^ticti+b^ti c_t+1^i= R_t^ic_t^i+ b_t^i. Anchor-Supervised Motion Training. Applying the recovered rigid transformation to the current anchors yields x^t+1,naii x_t+1,n_a^i^i. For the fixed anchor indices naia=1A\n_a^i\_a=1^A, we compute the ground-truth and rigid-projected second-order displacements, respectively, as: [Δ2t,anci,∗]a=xt+1,naii,∗−2xt,naii,∗+xt−1,naii,∗,[Δ2^t,anci,rigid]a=x^t+1,naii−2xt,naii,∗+xt−1,naii,∗. [ ^2 x_t,anc^i,* ]_a=x_t+1,n_a^i^i,*-2x_t,n_a^i^i,*+x_t-1,n_a^i^i,*, [ ^2 x_t,anc^i,rigid ]_a= x_t+1,n_a^i^i-2x_t,n_a^i^i,*+x_t-1,n_a^i^i,*. (5) Here, ∗* denotes ground-truth quantities and [⋅]a[·]_a extracts the a-th row. We supervise both the direct prediction and its rigid-projected counterpart in the following objective: ℒ=ℓSL1(Δ2^t,anci,Δ2t,anci,∗)+ℓSL1(Δ2^t,anci,rigid,Δ2t,anci,∗).L= _SL1\! ( ^2 x_t,anc^i, ^2 x_t,anc^i,* )+ _SL1\! ( ^2 x_t,anc^i,rigid, ^2 x_t,anc^i,* ). (6) where ℓSL1 _SL1 denotes the smooth-L1 loss. Both loss terms are computed in the normalized target space. Further implementation details are provided in Appendix B.1. 4 Experiment Datasets & Metrics. We evaluate on MOVi-Sphere, MOVi-A, and MOVi-B (Greff et al., 2022). The datasets span increasing geometric diversity, from spheres to primitive shapes and more complex object geometries. We report autoregressive position RMSE (m) and orientation RMSE (degrees) at 50, 75, and 100 frames (Allen et al., 2023; Rubanova et al., 2024; Wei and Fink, 2025). We additionally measure excess penetration frequency and mean penetration depth relative to the ground-truth trajectories in each dataset. More details of metrics can be found in appendix C.3. The best results are highlighted in bold, while the second-best results are indicated by underlined italics. Baselines. We compare RiCo with various baselines: mesh-based simulators (e.g. FIGNet, HOPNet), point-based methods (VPD (Whitney et al., 2024)), transformer-based approaches (HCMT (Yu et al., 2024)) and point-based simulator (RigidFormer (Dou et al., 2026)). For the main benchmark and cross-dataset generalization experiments, we adopt the results reported by RigidFormer, including its reimplementations of FIGNet, HCMT, and VPD. For the remaining experiments, we focus on HOPNet (Wei and Fink, 2025) and RigidFormer, two recent methods that are most relevant baselines of mesh-based and point-cloud-based methods, respectively. We use the officially released HOPNet checkpoint and our reimplementation of RigidFormer, as its official code is not published. 4.1 Motion Prediction Performance Across Rollout Horizons We evaluate the autoregressive prediction accuracy of RiCo against various baselines across different object geometries and rollout horizons. As shown in Table 1, RiCo consistently achieves lower position and orientation errors than all baselines on MOVi-A and MOVi-B across all evaluated horizons. On MOVi-A, compared with the strongest baseline, our model reduces position RMSE by approximately 35% and orientation RMSE by 38–43% over 50–100 prediction steps. Similarly, on MOVi-B, we obtain a 31–32% reduction in position error and a 35–38% reduction in orientation error. On the geometrically simpler MOVi-Sphere benchmark, our translational accuracy is comparable to RigidFormer, differing by at most 0.0010.001 m at each horizon, while consistently achieving lower orientation error. These results suggest that combining local contact representations with intra-object contact reasoning is particularly effective for predicting rigid-body interactions across diverse object shapes, while maintaining competitive performance in simpler spherical scenarios. Notably, the prediction improvements persist throughout the evaluated autoregressive rollout horizons. At 100 frames, RiCo achieves position and orientation RMSEs of 0.1150.115 m/11.40∘11.40 on MOVi-A and 0.1110.111 m/9.54∘9.54 on MOVi-B, respectively, substantially outperforming the corresponding baselines. These results highlight the effectiveness of our contact-conditioned point Transformer, which integrates sparse local contact information through intra-object contact reasoning, for long-horizon rigid-body motion prediction. Table 1: Performance on MOVi-A, MOVi-B, and MOVi-Sphere. Each cell reports position RMSE (m) / orientation RMSE (∘) at prediction horizons of 50, 75, and 100 frames. Model MOVi-A MOVi-B MOVi-Sphere 50 75 100 50 75 100 50 75 100 FIGNetreimpl_reimpl 0.132/7.10 0.285/14.62 0.492/23.30 0.141/7.39 0.300/15.16 0.516/24.96 N/A N/A N/A HCMTreimpl_reimpl 0.239/5.70 0.538/11.82 0.951/18.40 0.237/4.72 0.527/9.80 0.932/17.43 0.243/4.19 0.541/8.21 0.956/13.81 VPDreimpl_reimpl 0.235/5.10 0.489/11.66 0.827/20.37 0.275/4.65 0.581/9.70 0.987/16.99 0.244/4.47 0.510/9.50 0.855/17.65 HopNet 0.054/5.64 0.115/11.84 0.196/18.83 0.047/4.91 0.101/10.35 0.176/17.91 0.034/4.05 0.073/8.21 0.124/13.68 RigidFormer 0.049/5.06 0.103/10.90 0.177/18.32 0.050/3.97 0.095/8.51 0.161/15.33 0.026/3.00 0.057/6.48 0.099/11.19 RiCo 0.031/2.86 0.067/6.49 0.115/11.40 0.032/2.59 0.065/5.45 0.111/9.54 0.026/2.65 0.058/5.59 0.100/9.69 Note: The subscript reimpl denotes reimplementations conducted by Rigidformer authors. 4.2 Generalization Across Object Geometries A useful rigid-body dynamics model must predict interactions across diverse object geometries, including shapes that differ from those in its training data. We therefore evaluate how well dynamics learned from one geometry distribution transfer to another. As shown in Table 2, RiCo exhibits an asymmetric generalization pattern, with particularly strong performance when transferring from geometrically diverse training distributions to simpler ones. When trained on the geometrically diverse MOVi-B dataset, our model transfers particularly well to its simpler constituent distributions. From MOVi-B to MOVi-Sphere, our 100-step position RMSE is 0.0850.085 m, compared with 0.1600.160 m for HopNet and 0.2070.207 m for RigidFormer. Similarly, from MOVi-B to MOVi-A, our error is 0.1120.112 m/11.26∘11.26 , substantially lower than both baselines. Notably, the additional geometry diversity in MOVi-B does not lead to negative transfer on simpler shapes. We observe similar improvements when transferring from MOVi-A to MOVi-Sphere. In contrast, generalization from geometrically restricted training distributions to more diverse target distributions remains challenging. When transferring from MOVi-A to MOVi-B, RiCo achieves the lowest position error, although RigidFormer retains a slightly lower orientation error. When trained exclusively on MOVi-Sphere, our method does not consistently outperform RigidFormer on the more geometrically diverse MOVi-A and MOVi-B benchmarks. These results suggest that combining local contact representations with intra-object contact reasoning supports effective transfer across different geometry distributions, particularly when the training data provides sufficiently diverse geometric coverage. Nevertheless, generalization to more complex geometries remains dependent on the diversity of the training distribution. Table 2: Generalization performance across MOVi-Sphere (S), MOVi-A (A), and MOVi-B (B). Each reports position RMSE (m) / orientation RMSE (∘) at prediction horizons of 75 and 100 frames. Train Test HopNet RigidFormer RiCo 75 100 75 100 75 100 MOVi-S MOVi-A 0.112/11.29 0.197/17.74 0.107/11.27 0.183/17.92 0.110/10.15 0.190/16.79 MOVi-B 0.106/9.75 0.188/17.13 0.096/9.42 0.161/16.81 0.103/9.63 0.170/16.52 MOVi-A MOVi-S 0.100/9.16 0.172/15.17 0.087/7.94 0.153/14.27 0.055/5.63 0.094/9.83 MOVi-B 0.117/10.77 0.202/18.66 0.123/9.29 0.208/17.08 0.095/10.56 0.164/17.93 MOVi-B MOVi-S 0.095/9.09 0.160/15.04 0.123/10.52 0.207/17.88 0.049/5.31 0.085/9.29 MOVi-A 0.120/12.08 0.202/19.15 0.140/12.25 0.234/20.28 0.066/6.38 0.112/11.26 (a) Relative penetration statistics. (b) MOVi-B prediction error. Figure 3: Ablation of local contact reasoning on MOVi-B. (a) Ground-truth-relative penetration statistics. (b) Position and orientation RMSE at the 100-step prediction horizon. For each method, the left bar shows the time ratio (left y-axis), and the right bar shows penetration depth (right y-axis). (a) Relative penetration statistic. (b) MOVi-B prediction error. Figure 4: Resolution and efficiency analysis on MOVi-B. (a) Ground-truth-relative penetration statistics across different point-cloud resolutions. (b) Position and orientation prediction errors. For these curves, solid lines denote time ratios, measured on the left y-axis. Dashed lines denote penetration depths, measured on the right y-axis. 4.3 In-Depth Study on the Contact Fidelity While trajectory RMSE measures prediction accuracy, it does not directly measure the physical consistency of predicted interactions. To evaluate contact fidelity, we introduce ground-truth-relative penetration metrics. It accounts for the geometric overlaps that already present in ground-truth trajectories. Specifically, we report the differences in penetration-time ratio (Δ ) and mean penetration depth (Δ ) between predicted and ground-truth trajectories. Detailed implementation can be seen in appendix C.3. For these analyses, we evaluate RiCo against the released HOPNet checkpoint and our independent RigidFormer reimplementation using the same penetration evaluation procedure. We further investigate the effects of signed geometric information, contact neighborhood size, and point-cloud resolution on prediction accuracy and contact fidelity. We also conduct runtime analysis on point-cloud resolution in appendix C.4. Contact fidelity. As shown in Fig. 3(a), RiCo achieves substantially lower ground-truth-relative penetration statistics than both baselines. On MOVi-B, RiCo records a Δ of 11.0%11.0\% and a Δ of 2.222.22 m, compared with 42.7%42.7\% and 286.62286.62 m for HOPNet, and 44.8%44.8\% and 267.70267.70 m for our RigidFormer reimplementation. Together with the lower trajectory errors, these results indicate that RiCo improves prediction accuracy, while maintains high contact fidelity. Ablation of local contact representation. As detailed in Sec. 3.1, our local contact representation incorporates multiple geometric, motion, and physical features, most of which are either standard physical properties or have been explored in prior works, such as relative geometry and motion (Wei and Fink, 2025; Dou et al., 2026). Therefore, we focus on evaluating the contribution of our estimated signed distance and investigating the effect of contact neighborhood size. Estimated signed distance. To investigate the importance of signed geometric information, we replace the estimated signed distance with its absolute value while retaining all other input features. As shown in Fig. 3, this ablation only moderately affects trajectory RMSE but substantially degrades contact fidelity. Specifically, Δ increases from 11.0%11.0\% to 23.6%23.6\%, while Δ increases from 2.222.22 to 7.867.86 m. These results suggest that explicitly encoding the signed geometric relationship between nearby surfaces helps the model better capture contact configurations and reduce interpenetration, even when its overall trajectory prediction accuracy changes only marginally. Number of contact neighbors. We further vary the number of retained contact candidates K∈1,4,8K∈\1,4,8\. As shown in Fig. 3, the model maintains relatively stable prediction accuracy and penetration statistics across different neighborhood sizes. While K=8K=8 achieves slightly lower Δ , K=4K=4 achieves the lowest Δ and orientation RMSE. These results suggest that a compact contact neighborhood is sufficient to achieve strong prediction performance under our benchmark. Exploiting high-resolution contact geometry. We next investigate whether increasing point-cloud resolution provides additional benefits for contact modeling, as shown in Fig. 4(a) and 4(b). Increasing RiCo’s resolution from 512 to 1024 points yields only a marginal change in position RMSE, from 0.1120.112 to 0.1110.111 m, but substantially improves contact fidelity. Specifically, Δ decreases from 23.4%23.4\% to 11.0%11.0\%, while Δ decreases from 6.216.21 to 2.222.22 m. In comparison, our RigidFormer reimplementation exhibits nearly unchanged trajectory RMSE across different resolutions, while its penetration statistics remain considerably higher despite improvements from increased surface sampling. These suggest that RiCo’s local contact reasoning architecture is more effective in exploiting the richer contact information provided by high-resolution geometry. Compared with RigidFormer’s shape compression and global interaction reasoning, RiCo retains fine-grained surface information through sparse local contact representations, supporting more accurate contact modeling and lower trajectory errors. 4.4 Zero-shot Scalability to Large-scale Scenes We further investigate whether the learned dynamics can zero-shot generalize to large-scale scenes containing more interacting objects than those encountered during training. Inspired by the large-scale evaluation setting of SDF-Sim (Rubanova et al., 2024), we construct three large-scale scenarios, each containing 270 rigid objects. RiCo is trained exclusively on MOVi-B scenes containing 3–10 objects and directly evaluated on these large-scale scenarios. Table 3: Large-scale prediction. Position / Orientation RMSE Runtime Scene 120 240 360 FPS Spheres 0.029/1.73 0.243/43.07 0.713/89.90 5.02 Spots 0.019/1.21 0.211/22.90 0.785/86.74 6.50 Knots 0.017/0.81 0.139/13.43 0.843/78.00 6.54 As shown in Table 3, we evaluate the autoregressive prediction accuracy at 120, 240 and 360 steps. Since collision events occur mainly during the latter part of these trajectories, we extend the evaluation horizon to 360 steps to further examine long-term prediction accuracy after extensive object interactions. More results can be found from appendix C.5. Compared with RigidFormer, which trains on large-scale WreckingBall scenes containing up to 217 objects, our evaluation directly applies a model trained on small-scale scenes to substantially larger configurations. Although RigidFormer reports a throughput of approximately 20 FPS on its large-scale benchmark, its experimental setting differs in object geometry and training settings. 4.5 Transfer from Simulated to Real-World Dynamics To further investigate whether the learned local contact dynamics can generalize beyond simulated environments, we evaluate RiCo on real-world multi-ball collision experiments. In Figure 8, our real-world experiments are conducted on a billiard table with an effective playing area of 2.54×1.272.54× 1.27 m, using balls with a radius of 33.533.5 m. We construct a corresponding simulated environment with comparable geometric configurations and train RiCo exclusively on simulated trajectories. The trained model is evaluated zero-shot on five real-world experimental sessions. Further details and qualitative results of the experimental setup and simulation environment are provided in the appendix C.6. Quantitatively, RiCo achieves position RMSEs of 138.54138.54, 214.69214.69, and 272.42272.42 m at prediction horizons of 50, 75, and 100 frames. For reference, the physics engine achieves corresponding RMSEs of 65.3765.37, 147.84147.84, and 182.96182.96 m. We observe discrepancies when reconstructing real-world ball configurations in simulation using camera-estimated positions, suggesting that camera-based localization inaccuracy is a major source of error in our sim-to-real evaluation. These results provide preliminary evidence that the contact dynamics learned through sparse local geometric reasoning can transfer from simulation to real-world multi-object interactions. 5 Conclusion In this work, we introduce RiCo, a neural simulator that models rigid-body interactions through sparse local contact reasoning. By preserving fine-grained contact geometry while avoiding dense scene-level interactions, our approach achieves accurate and physically consistent predictions with high computational efficiency. Experiments demonstrate its ability to exploit high-resolution geometry, generalize across diverse object distributions, scale to scenes containing 270 objects and transfer to real world. Nevertheless, our current framework focuses on passive rigid-body dynamics and does not account for deformable objects or actuated systems. Extending local contact reasoning to these more complex physical interactions remains an important direction for future work. Acknowledgments We are especially grateful to Wanxin Jin and Zhixian Xie from Arizona State University for their insightful discussions and valuable advice. References Agarwal et al. (2025) N. Agarwal, A. Ali, M. Bala, Y. Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y. Chen, Y. Cui, Y. Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575. Cited by: §1. Allen et al. (2023) K. R. Allen, Y. Rubanova, T. Lopez-Guevara, W. Whitney, A. Sanchez-Gonzalez, P. W. Battaglia, and T. Pfaff Learning rigid dynamics with face interaction graph networks. In International Conference on Learning Representations, ICLR, Cited by: §1, §2, §2, §4. Assran et al. (2023) M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas Self-supervised learning from images with a joint-embedding predictive architecture. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 15619–15629. Cited by: §2. Battaglia et al. (2016) P. Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems 29. Cited by: §2. Brandstetter et al. (2022) J. Brandstetter, D. E. Worrall, and M. Welling Message passing neural PDE solvers. In The Tenth International Conference on Learning Representations, ICLR, Cited by: §1. Bruce et al. (2024) J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, Y. Aytar, S. Bechtle, F. M. P. Behbahani, S. C. Y. Chan, N. Heess, L. Gonzalez, S. Osindero, S. Ozair, S. E. Reed, J. Zhang, K. Zolna, J. Clune, N. de Freitas, S. Singh, and T. Rocktäschel Genie: generative interactive environments. In International Conference on Machine Learning, ICML, Vol. 235, p. 4603–4623. Cited by: §1. Chang et al. (2016) M. B. Chang, T. Ullman, A. Torralba, and J. B. Tenenbaum A compositional object-based approach to learning physical dynamics. arXiv preprint arXiv:1612.00341. Cited by: §1. Dou et al. (2026) Z. Dou, M. Guo, H. Wu, D. Roble, T. Stuyck, and W. Matusik RigidFormer: learning rigid dynamics using transformers. arXiv preprint arXiv:2605.09196. Cited by: §1, §2, §3.1, §3.3, §4.3, §4. Greff et al. (2022) K. Greff, F. Belletti, L. Beyer, C. Doersch, Y. Du, D. Duckworth, D. J. Fleet, D. Gnanapragasam, F. Golemo, C. Herrmann, et al. Kubric: a scalable dataset generator. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 3749–3761. Cited by: §1, §4. Ha and Schmidhuber (2018) D. Ha and J. Schmidhuber Recurrent world models facilitate policy evolution. Advances in neural information processing systems 31. Cited by: §2. Hafner et al. (2019) D. Hafner, T. P. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference on Machine Learning, ICML, Vol. 97, p. 2555–2565. Cited by: §1, §2. Hafner et al. (2023) D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104. Cited by: §1. Huang et al. (2026) W. Huang, Y. Chao, A. Mousavian, M. Liu, D. Fox, K. Mo, and L. Fei-Fei PointWorld: scaling 3d world models for in-the-wild robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 20765–20779. Cited by: §2. Kabsch (1976) W. Kabsch A solution for the best rotation to relate two sets of vectors. Foundations of Crystallography 32 (5), p. 922–923. Cited by: §3.3. Li et al. (2026) L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, F. Han, M. Yu, Z. Gao, N. Xue, X. Zhu, et al. Causal world modeling for robot control. arXiv preprint arXiv:2601.21998. Cited by: §1. Li et al. (2018) Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.01566. Cited by: §2. Lin et al. (2026) S. Lin, J. Chen, H. Xu, Z. Li, G. Wang, Y. Jing, S. Xu, R. Zhao, B. Sheil, L. Chau, et al. Roboflow4d: a lightweight flow world model toward real-time flow-guided robotic manipulation. arXiv preprint arXiv:2605.17522. Cited by: §2. Pfaff et al. (2020) T. Pfaff, M. Fortunato, A. Sanchez-Gonzalez, and P. W. Battaglia Learning mesh-based simulation with graph networks. arXiv preprint arXiv:2010.03409. Cited by: §1, §1, §2. Qi et al. (2017) C. R. Qi, L. Yi, H. Su, and L. J. Guibas Pointnet++: deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems. Cited by: §3.3. Rubanova et al. (2024) Y. Rubanova, T. Lopez-Guevara, K. R. Allen, W. F. Whitney, K. Stachenfeld, and T. Pfaff Learning rigid-body simulators over implicit shapes for large-scale scenes and vision. Advances in Neural Information Processing Systems. Cited by: §2, §4.4, §4. Sanchez-Gonzalez et al. (2020) A. Sanchez-Gonzalez, J. Godwin, T. Pfaff, R. Ying, J. Leskovec, and P. Battaglia Learning to simulate complex physics with graph networks. In International conference on machine learning (ICML), Cited by: Appendix A, §1, §2. Sanchez-Gonzalez et al. (2018) A. Sanchez-Gonzalez, N. Heess, J. T. Springenberg, J. Merel, M. Riedmiller, R. Hadsell, and P. Battaglia Graph networks as learnable physics engines for inference and control. In International conference on machine learning (ICML), Cited by: §2. Wei and Fink (2025) A. Wei and O. Fink Integrating physics and topology in neural networks for learning rigid body dynamics. Nature Communications 16 (1), p. 6867. External Links: ISSN 2041-1723, Document, Link Cited by: §C.1, §1, §1, §2, §3.1, §4.3, §4, §4. Whitney et al. (2024) W. Whitney, T. Lopez-Guevara, T. Pfaff, Y. Rubanova, T. Kipf, K. Stachenfeld, and K. Allen Learning 3d particle-based simulators from rgb-d videos. In International Conference on Learning Representations (ICLR), Cited by: §4. Wu et al. (2023) Z. Wu, N. Dvornik, K. Greff, T. Kipf, and A. Garg SlotFormer: unsupervised visual dynamics simulation with object-centric models. In The Eleventh International Conference on Learning Representations, ICLR, Cited by: §1. Ye et al. (2026) S. Ye, Y. Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y. L. Tan, C. Zhu, J. Xiang, et al. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922. Cited by: §1. Yu et al. (2024) Y. Yu, J. Choi, W. Cho, K. Lee, N. Kim, K. Chang, C. Woo, I. Kim, S. Lee, J. Y. Yang, et al. Learning flexible body collision dynamics with hierarchical contact mesh transformer. In International Conference on Learning Representations (ICLR), Cited by: §4. Zhou et al. (2024) G. Zhou, H. Pan, Y. LeCun, and L. Pinto Dino-wm: world models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983. Cited by: §2. Appendix A Problem Formulation Physical world modeling. A physical world model aims to capture how the physical environment evolves over time, enabling embodied agents to anticipate future states and reason about the consequences of their actions through predictive rollouts. Such predictive dynamics provide an important foundation for long-horizon reasoning, planning, and control, where an agent must evaluate possible future outcomes before acting. In physical environments, however, meaningful prediction requires not only modeling the motion of individual objects, but also reasoning about the interactions among them (Sanchez-Gonzalez et al., 2020). This is particularly important in multi-object rigid-body systems, where contact and collision govern the transfer of momentum and can induce abrupt changes in both translational and rotational motion. Accurately modeling collision dynamics is therefore a key capability for physical world models to move beyond simple motion extrapolation and faithfully predict the long-horizon evolution of interacting scenes. Rigid Body Contact Modeling. In this work, we study the dynamics of contact between rigid bodies. We consider a scene containing M rigid objects. At time t, object i is represented by N tracked surface samples, with positions ti∈ℝN×3 x_t^i ^N× 3 and surface normals ti∈ℝN×3 ν_t^i ^N× 3. We denote its center of mass by cti∈ℝ3c_t^i ^3 and its physical properties by ϕi=[(mi)−1,μi,ei]φ^i=[(m^i)^-1,μ^i,e^i], where mim^i, μiμ^i, and eie^i are the mass, friction coefficient, and coefficient of restitution, respectively. We collect the time-varying geometric quantities into the scene state ts_t and the time-invariant physical properties into ϕ φ. In this way, t=ti,ti,ctii=1Ms_t= \ x_t^i, ν_t^i,c_t^i \_i=1^M, and ϕ=ϕii=1M φ= \φ^i \_i=1^M. Surface-point identities are preserved over time, allowing us to compute their finite-difference displacements as Δti=ti−t−1i x_t^i= x_t^i- x_t-1^i. We additionally retain the reference geometry 0=0ii=1M x_0=\ x_0^i\_i=1^M. The dynamics model operates directly on these surface samples without requiring mesh connectivity. Let ξ denote the static environment together with its geometric and physical properties. Given two consecutive scene states, we learn a transition model ^t+1=fθ(t,t−1,ϕ,ξ), s_t+1=f_θ\! (s_t,s_t-1, φ,ξ ), (7) which predicts the next rigid-body configuration. At inference time, the same transition model is applied recursively to generate long-horizon rollouts. Appendix B Additional Method Details We provide implementation details of the surface-state representation, sparse contact construction, network architecture, rigid-motion projection, and training/inference procedures. Unless otherwise stated, the configurations below describe our standard fixed-resolution local-contact implementation, rather than separate resolution-ablation variants. B.1 Surface Representation and Feature Preprocessing Persistent surface representation. The offline dynamics model operates on object-aligned surface point sets stored in trajectory data. Its default configuration represents each active rigid body with N=1024N=1024 ordered surface points and corresponding surface normals. Point indices retain their physical correspondence across consecutive frames; therefore, a point’s finite-difference displacement is obtained directly by subtracting its position at the preceding frame, rather than by performing nearest-neighbor matching across frames. The standard MOVi training data uses Odata=10O_data=10 object slots per scene. An object-level activity mask identifies occupied slots, and inactive slots are excluded from network evaluation and losses. This fixed storage dimension is specific to the training data and does not constrain the number of objects supported by the model at inference time. The default configuration does not use a variable-length point mask within an active object: each occupied slot contains precisely 10241024 point entries. The neural runtime consumes these recorded point sets without resampling them at each prediction step. Object state and physical parameters. For a source point t,nix_t,n^i on object i, we use the physically defined center of mass tic_t^i from the trajectory state, not the arithmetic mean of the sampled surface points. The finite-difference displacement is Δt,ni=t,ni−t−1,ni _t,n^i=x_t,n^i-x_t-1,n^i; it is not divided by the simulation time step. Surface normals are normalized to unit length before constructing the descriptors. Object-level physical parameters ϕi=[(mi)−1,μi,ei] φ^i=[(m^i)^-1,μ^i,e^i] are broadcast to the points of that object. We further preserve the initial surface configuration 0iX_0^i as a reference throughout the trajectory, so that each point also encodes its displacement from its initial position. Table 4 specifies the exact order of the two feature vectors. The source-point descriptor has 1515 channels, while each external-contact descriptor has 1414 channels. The distance used for neighbor weighting is maintained separately and is not duplicated as an additional learned feature channel. Table 4: Input feature specification. All geometric positions and finite-difference displacements are expressed in the same scene coordinate system. Here y is an external candidate point, extn_ext its unit normal, and Δ its finite-difference displacement. Descriptor Channels (in implementation order) Dim. Source-point descriptor t,nis_t,n^i t,ni−tix_t,n^i-c_t^i 3 Δt,ni _t,n^i 3 t,ni ν_t,n^i 3 (mi)−1,μi,ei(m^i)^-1,\,μ^i,\,e^i 3 t,ni−0,nix_t,n^i-x_0,n^i 3 External-contact descriptor t,n,ki η_t,n,k^i (−)/(‖−‖2+ϵ)(y-x)/(\|y-x\|_2+ε) 3 (−)⊤ext(x-y) n_ext 1 extn_ext 3 Δ−Δ - 3 mext−1,μext,eextm_ext^-1,\, _ext,\,e_ext 3 σstatic∈0,1 _static∈\0,1\ 1 Feature and target normalization. The implementation uses three separate channel-wise normalizers: one for the 1515-dimensional source features, one for the 1414-dimensional contact features, and one for the 33-dimensional second-order anchor-displacement target. Each has the form ()=−,−1(^)=^⊙+,N(z)= z- μ σ, ^-1( z)= z σ+ μ, (8) where the division and multiplication act channel-wise. Normalization statistics are fixed during model training and saved with the checkpoint; they are not recomputed from validation or test trajectories. The provided statistics-generation script accumulates moments over active objects in the training split and computes contact statistics over valid contact entries only. It uses a shared scale for each selected three-dimensional vector group. In its physical-semantics policy, unit directions and normals, friction/restitution coefficients, and the static indicator remain in their original ranges, while the signed tangent-plane offset is divided by the contact radius. Other continuous channels and the anchor target use training-set statistics. An explicitly populated normalization configuration takes precedence over a generated dataset sidecar; consequently, the exact statistics used for a reported experiment are those saved in its checkpoint. B.2 Sparse Contact Neighborhood Construction Broad-phase object filtering. A contact neighborhood is recomputed from the current geometry at inference time. For each active object i, we construct an axis-aligned bounding box from its current surface points, with lower and upper corners il_i and iu_i. For every pair of distinct active objects (i,j)(i,j), their minimum box separation is dAABB(i,j)=‖max(i−j,j−i,)‖2,d_AABB(i,j)= \| \! (l_i-u_j,l_j-u_i,0 ) \|_2, (9) with the maximum taken element-wise. Only pairs satisfying dAABB(i,j)≤ρd_AABB(i,j)≤ρ are retained, where ρ is the contact radius. By default, we adopt ρ=0.1ρ=0.1 m. The pairwise geometric search is performed without gradient tracking. Point-level search and global candidate selection. For each retained unordered pair (i,j)(i,j), the implementation computes a pairwise Euclidean-distance matrix using the surface samples of the two objects. In both directions, it retains at most K nearest external samples for every source point. The candidates obtained from all eligible neighboring objects are then merged, and the closest K dynamic candidates are retained for each source point. Searches over eligible object pairs are processed in bounded batches (pair_batch_size=16=16 in the default configuration) The default MOVi setting also includes a static horizontal plane at height ztablez_table. For a source point =(x,y,z)x=(x,y,z), its static candidate is the analytic projection table=(x,y,ztable)y_table=(x,y,z_table), with normal (0,0,1)(0,0,1), zero displacement, zero inverse mass, and the plane’s assigned friction and restitution. This analytic candidate is combined with the K retained dynamic candidates, and a final nearest-K selection is performed across both types. A selected candidate is valid only if it comes from an eligible object or the plane and has a current Euclidean distance no greater than ρ. Contacts with the source object’s own points are excluded. The default number of retained contact candidates is K=4K=4. Estimated signed offset and contact masking. For a selected candidate y with unit outward normal extn_ext, we compute δ=(−)⊤ext.δ=(x-y) n_ext. (10) This quantity is the signed offset from the candidate’s local tangent plane; it is an estimated local contact descriptor, not an exact continuous-mesh signed-distance function. Its sign depends on the orientation of the supplied surface normal. Together with the unit direction, relative finite-difference displacement, external material parameters, and static flag, it forms the 1414-dimensional descriptor specified in Table 4. The feature builder uses ϵ=10−8ε=10^-8 in the direction normalization. Offline contact caches versus online contact queries. The training dataset stores a final Top-K target-index tensor at each recorded frame. During teacher-forced training, the implementation reads these target indices and reconstructs the relative contact features using the recorded current point positions, displacements, normals, and physical parameters; it does not redo nearest-neighbor search in every training iteration. Invalid cached candidates are encoded with a sentinel index and masked. By contrast, autoregressive inference invokes the live broad-phase and nearest-neighbor query on the model’s newly predicted geometry at every step, allowing neighborhoods to change as contacts appear and disappear. The contact-index selection itself is non-differentiable; the trainable contact encoder and subsequent network operate on the selected descriptors. B.3 Network Architecture Pointwise encoders and distance-weighted fusion. The source-point encoder is a two-layer MLP 15→128→38415→ 128→ 384 with a GELU activation. Each 1414-dimensional contact descriptor is independently embedded by another two-layer MLP 14→128→38414→ 128→ 384, using a Softplus activation. The contact embeddings are aggregated by a masked, distance-dependent softmax. In the implementation, the learnable temperature is parameterized as τ=softplus(τraw)+τmin,τmin=10−4,τ=softplus( _raw)+ _ , _ =10^-4, (11) and initialized so that τ=0.005τ=0.005. Softmax probabilities are computed over valid candidates only and renormalized after masking; a source point without any valid candidate receives a zero contact embedding. The sum of the source and aggregated contact embeddings is then normalized with RMSNorm to form the input point token. Shared intra-object Transformer. Each active object’s N tokens are processed independently by the same six-block point Transformer. Every block applies pre-normalized, 12-head self-attention followed by a separate pre-normalized feed-forward network, with residual connections around both components. At token width d=384d=384, each attention head has dimension 3232. For the gated-attention configuration, the implementation first computes standard scaled dot-product attention and then multiplies the attended features by a learned sigmoid gate computed from the normalized input tokens, before the attention output projection. The feed-forward network uses a 384→3072384→ 3072 projection followed by GLU, producing a hidden width of 15361536, and a final 1536→3841536→ 384 projection. The attention and feed-forward dropout settings are both zero in the default configuration. No attention operation in this encoder directly connects point tokens belonging to different objects. Anchor-query decoder. The decoder creates one query for each of A=8A=8 persistent anchor indices. A query combines (i) the encoded token at that anchor and (i) a geometry embedding of the anchor’s six-dimensional state, consisting of its position relative to the physical center of mass and its preceding finite-difference displacement. The geometry encoder is an MLP 6→128→3846→ 128→ 384 with GELU. The resulting anchor queries pass through six cross-attention blocks with 12 heads, attending to all N tokens of the same object. Like the point Transformer, each cross-attention block uses separate query/token RMSNorm, gated attention, residual connections, and a GLU feed-forward network of hidden width 15361536. An RMSNorm and an output MLP 384→128→3384→ 128→ 3 with GELU produce the predicted second-order displacement of every anchor. Decoder parameters are shared across all objects. Table 5: Default local-contact model architecture. The two Transformer stacks operate within each object; the indicated cross-attention is from anchor queries to the object’s own point tokens. Component Configuration Maximum object slots / points per object 1010 / 10241024 Anchors / contact candidates per point 88 / 44 Source feature MLP →38415\!→\!128\!→\!384, GELU Contact feature MLP →38414\!→\!128\!→\!384, Softplus Contact pooling Masked distance softmax Contact temperature (initial/minimum) 0.0050.005 / 10−410^-4 Feature fusion Sum + RMSNorm Point Transformer 66 blocks, 1212 heads Anchor cross-attention decoder 66 blocks, 1212 heads Token / per-head dimension 384384 / 3232 Transformer feed-forward hidden width 15361536, GLU Attention and FFN dropout 00 Anchor geometry MLP →3846\!→\!128\!→\!384, GELU Output head RMSNorm, →3384\!→\!128\!→\!3 Kabsch numerical threshold 10−610^-6 B.4 Anchor-Based Rigid Motion Prediction Persistent anchor indices and displacement decoding. The trajectory interface provides eight fixed, distinct anchor indices for each active object. These same indices are used when constructing the anchor queries, supervising predictions, and aligning the predicted configuration; the network does not dynamically reselect anchors in the middle of a rollout. Let t,aiq_t,a^i be the current position of anchor a on object i. The decoder outputs a three-dimensional second finite difference, denoted by Δ2^t,ai ^2q_t,a^i, without division by the physical time step squared. The unconstrained next anchor is ~t+1,ai=2t,ai−t−1,ai+Δ2^t,ai. q_t+1,a^i=2q_t,a^i-q_t-1,a^i+ ^2q_t,a^i. (12) The predicted normalized decoder output is converted back to physical units by the target normalizer before this update. Proper rigid alignment. We recover a single proper rigid transform independently for each active object through unweighted least-squares alignment between its current anchor positions and the unconstrained next positions. With column-vector convention, let ¯ q and ~¯ q be their respective centroids, and define =∑a=1A(t,a−¯)(~t+1,a−~¯)⊤,=⊤.H= _a=1^A(q_t,a- q)( q_t+1,a- q) , =U . (13) To exclude reflections, the implementation computes =diag(1,1,signdet(⊤))⊤,=~¯−¯.R=Vdiag\! (1,1,sign (VU ) )U , = q-R q. (14) The same SE(3)SE(3) transform advances all object surface points and its physical center of mass, while normals are updated by rotation alone and renormalized to unit length. For inactive object slots, the alignment implementation returns an identity rotation and zero translation; their output states are subsequently masked. The Kabsch implementation uses a numerical threshold of ϵK=10−6 _ K=10^-6 for masking and safe centroid computation. Autoregressive Inference. At inference time, we initialize RiCo with two consecutive observed states at times t−1t-1 and t to predict the state at t+1t+1. At each step t, the recovered rotation R^ti R_t^i and translation b^ti b_t^i update the surface positions x^t+1,ni x_t+1,n^i and center of mass c^t+1i c_t+1^i, while the rotation alone updates the surface normals ν^t+1,ni ν_t+1,n^i. We then recompute the sparse local contact representation from the predicted geometry at t+1t+1 for the next transition. Appendix C Experimental Setup and Implementation We describe the offline data interface, optimization setup, evaluation protocol, and metric implementations used in our local-contact experiments. The architectural details of the standard model are provided in Appendix B. C.1 Benchmarks and Data Processing Benchmark suites. We evaluate on MOVi-Sphere, MOVi-A, and MOVi-B using the publicly released datasets from HOPNet (Wei and Fink, 2025). These benchmarks span different geometric distributions, enabling evaluations of in-distribution rollout prediction and cross-dataset generalization. Our standard offline implementation consumes precomputed trajectories rather than rendering observations or resampling surface points during training. We follow the original train/test partitions provided by HOPNet, with training and evaluation trajectories stored separately. Trajectory representation. The standard MOVi training datasets are stored in HDF5 format, using Odata=10O_data=10 object slots per scene, N=1024N=1024 ordered surface points per active object, and A=8A=8 fixed anchor indices per object. This fixed object dimension is specific to the offline training data representation and does not impose a limit on the number of objects supported by RiCo. For a trajectory with T stored frames, the primary state tensors have conceptual shapes [T,Odata,N,3][T,O_data,N,3] for surface positions and normals, [T,Odata,3][T,O_data,3] for physical centers of mass, and [T,Odata][T,O_data] for inverse mass, friction, restitution, and object activity. Contact-target indices are stored as [T,Odata,N,K][T,O_data,N,K], where K=4K=4 in the standard setting; these indices refer to neighboring surface samples or designated static contacts. The dataset also stores trajectory lengths, terminal flags, and persistent anchor indices. The loader uses trajectory lengths to avoid padded trailing frames and excludes training windows that cross a terminal transition. Its two-frame input and one-frame target configuration produces three-frame training windows. Since RiCo shares network parameters across objects, its inference is not restricted to the object count used during training. This enables our zero-shot evaluation on scenes containing 270 objects, as described in Sec. 4.4. Cached contacts during training. For a ground-truth training window, the loader reads the cached contact target IDs corresponding to its current frame. Their positions, normals, finite-difference displacements, and physical parameters are then gathered from the current ground-truth state to reconstruct the contact descriptors. This avoids repeating the discrete nearest-neighbor search for every teacher-forced training instance. In contrast, autoregressive evaluation reconstructs contact neighborhoods from the model’s predicted current geometry at each new step. Consequently, the model does not use future ground-truth contact assignments during inference. Additional feature and target normalization details are given in Appendix B.1. C.2 Optimization and Training Configuration Offline supervised training. The standard training loop receives two consecutive ground-truth states and predicts the next state with teacher forcing. The optimizer is AdamW. For each valid training object, supervision is applied to the direct and rigid-projected second-order anchor displacements using coordinate-wise smooth-L1L1 losses in normalized target space; both losses have unit weight. Objects contribute only if their slots are active at both the current and target frames. Loss numerators are summed over valid anchor coordinates. Reference configuration. Table 6 summarizes the reference offline training configuration. Configurations associated with separate benchmark training runs and ablations are stored alongside their corresponding checkpoints. Table 6: Reference offline optimization settings for the local-contact model. Setting Value Optimizer AdamW Global batch size 128128 Per-device micro-batch size 88 Context / target frames 22 / 11 Precision bf16 Initial / final learning rate 10−410^-4 / 10−510^-5 Learning-rate schedule Cosine decay Weight decay 00 Gradient-norm clipping 1.01.0 Training epochs 6060 Autoregressive fine-tuning Disabled Direct / projected loss weights 1.01.0 / 1.01.0 Random seed 00 Implementation and model selection. The trainer supports gradient accumulation through micro-batches and multi-GPU training through distributed data parallelism. Its learning-rate schedule advances per optimizer update, and its bf16 setting applies automatic mixed precision to the neural forward computation. Normalization parameters are computed from training data or loaded from an explicitly specified configuration and are saved with the resulting checkpoint. The available validation routine evaluates autoregressive predictions at physical frame indices 5050, 7575, and 100100. These are frame indices in the stored trajectory, not a claim that exactly that many neural updates follow the two observation frames. The checkpoint selection criterion and computational resources are specified for each final experiment. C.3 Evaluation Metrics and Measurement Procedures Position and orientation errors. Let ^r,ti c_r,t^i and r,ti,∗c_r,t^i,* be the predicted and ground-truth physical centers of mass for object i in trajectory r at evaluation frame t. Let r,tV_r,t be the set of objects active in both states. We first compute a per-scene position error, Er,tpos=1|r,t|∑i∈r,t‖^r,ti−r,ti,∗‖22.E_r,t^pos= 1|V_r,t| _i _r,t \| c_r,t^i-c_r,t^i,* \|_2^2. (15) For orientation, we estimate the relative proper rotation Δr,ti∈SO(3) _r,t^i (3) between predicted and ground-truth surface-point configurations using rigid alignment. Its geodesic angle in degrees is θr,ti=180πarccos[clip(tr(Δr,ti)−12,−1,1)]. _r,t^i= 180π \! [clip\! ( tr( _r,t^i)-12,-1,1 ) ]. (16) The per-scene orientation error is then Er,trot=1|r,t|∑i∈r,t(θr,ti)2.E_r,t^rot= 1|V_r,t| _i _r,t( _r,t^i)^2. (17) The reported frame-specific values are the arithmetic means of the per-scene RMSEs across evaluated trajectories. Translation is expressed in meters and orientation in degrees. This distinction matters because averaging scene-level RMSEs is not generally equivalent to pooling all objects across the dataset before taking a square root. Collision geometry and penetration depth. Penetration is evaluated separately from the network’s input-point representation. The evaluation script loads the scaled collision meshes associated with each object’s shape, registers the mesh to the frame-zero ground-truth surface points, and uses rigidly estimated poses to place the mesh in every subsequent ground-truth or predicted frame. For an object pair, an AABB check first rejects pairs with disjoint boxes. For the remaining pairs, the script tests the vertices of each mesh for occupancy inside the other mesh and queries the distances of inside vertices to that mesh’s surface. It takes the maximum measured depth over the two query directions. Penetration of the horizontal ground plane is evaluated from the minimum height of the transformed mesh vertices. For each active object and frame, the recorded depth Dr,ti≥0D_r,t^i≥ 0 is the maximum across all detected object–object and object–ground violations; Pr,ti=[Dr,ti>0]P_r,t^i=1[D_r,t^i>0] indicates a detected penetration. These measurements use a geometric vertex-query approximation on the evaluation meshes, rather than an exact continuous-surface intersection-depth solver. No additional positive depth threshold is applied in the inspected script. Figure 5: Runtime analysis. Ground-truth-relative penetration statistics. Let S denote either the predicted or ground-truth trajectories, and let ar,ti∈0,1a_r,t^i∈\0,1\ indicate an active object. The evaluated frame set T excludes the two initial frames. Let Q contain all trajectory–object pairs active in at least one evaluated frame. The penetration-time ratio implemented by our evaluator is PTR(S)=1||∑(r,i)∈∑t∈ar,tiPr,ti,S∑t∈ar,ti.PTR(S)= 1|Q| _(r,i) _t a_r,t^iP_r,t^i,S _t a_r,t^i. (18) Thus, the evaluator first computes the fraction of penetrating active frames for each trajectory–object pair, then averages these fractions. The mean penetration depth is computed over all detected penetrating active object–frame observations, MPD(S)=∑r,i∑t∈ar,tiPr,ti,SDr,ti,S∑r,i∑t∈ar,tiPr,ti,S,MPD(S)= _r,i _t a_r,t^iP_r,t^i,SD_r,t^i,S _r,i _t a_r,t^iP_r,t^i,S, (19) with MPD set to zero if the denominator vanishes. Depths are computed in meters and converted to millimeters for display. Finally, the ground-truth-relative values printed by the current evaluation script are the differences of these aggregated statistics, ΔPTR=PTR(pred)−PTR(GT),ΔMPD=MPD(pred)−MPD(GT). =PTR(pred)-PTR(GT), =MPD(pred)-MPD(GT). (20) The first quantity is reported in percentage points and the second in millimeters. These are differences of dataset-level summaries; they are not the frequency or conditional depth of a separately defined per-frame positive excess max(0,Dr,ti,pred−Dr,ti,GT) (0,D_r,t^i,pred-D_r,t^i,GT). This distinction is relevant when interpreting the physical-consistency results in the main paper. C.4 Runtime analysis Experimental setup. We evaluate computational efficiency on a single NVIDIA GeForce RTX 5090 GPU using the same 120 test trajectories for all methods. Runtime is measured with a batch size of one over 100-step autoregressive rollouts. Each rollout is initialized with two ground-truth frames, after which subsequent states are predicted autoregressively without accessing future ground-truth states. We perform five warm-up iterations before timing and synchronize CUDA execution immediately before and after each measured rollout. Dataset loading, disk I/O, visualization, and rendering are excluded from the runtime measurements. Computational efficiency. We examine whether retaining high-resolution geometry introduces substantial computational overhead. As shown in Fig. 5, we evaluate baselines and RiCo on the same 120 test trajectories using a single NVIDIA RTX 5090 GPU. Despite operating on 1024 surface points per object, RiCo achieves 47.8947.89 FPS, slightly faster than our RigidFormer reimplementation (41.7341.73 FPS) and over an order of magnitude faster than HOPNet (2.612.61 FPS). Reducing RiCo’s resolution to 512 and 256 points further increases throughput to 87.6987.69 and 130.87130.87 FPS, respectively. These results demonstrate that sparse local contact reasoning enables high-resolution geometric modeling without sacrificing computational efficiency. Figure 6: Qualitative results of three large-scale scenarios. Baseline implementation details. We reproduce RigidFormer for our experiments rather than using runtime measurements reported in the original paper. Although we follow its proposed architecture and inference pipeline, implementation-level differences may affect the measured throughput. Therefore, the reported RigidFormer runtime should be interpreted as the performance of our reimplementation rather than a definitive measurement of the original implementation. For HOPNet, we introduce additional implementation optimizations to improve inference efficiency while preserving its underlying modeling approach. Consequently, the reported HOPNet throughput reflects our optimized implementation rather than its unmodified implementation. All three methods are benchmarked on the same GPU and test trajectories. Nevertheless, the reported runtime comparison should be interpreted in light of these implementation differences. (a) Position offset in the centroid of active objects. (b) Orientation Offset in the centroid of active objects. Figure 7: Temporal error accumulation in large-scale scenes. Position and orientation errors throughout autoregressive rollouts of three 270-object scenarios. The dashed line indicates an approximate transition point after which object collisions become more frequent. C.5 Temporal Error Accumulation in Large-Scale Scenes To further analyze long-horizon behavior in the 270-object scenarios (Fig. 6), we record the temporal evolution of prediction errors throughout autoregressive rollouts. Figure 7 shows the position and orientation errors as functions of rollout step. In each plot, the dashed vertical line marks an approximate transition point after which collisions become more frequent, based on qualitative observation of the scene dynamics. As shown in the figure, prediction errors remain relatively small during the earlier stage of the rollout, and then increase more rapidly after the onset of more frequent multi-object interactions. This trend is consistent across the large-scale scenarios and helps explain why we report errors not only at 120 frames, but also at longer horizons of 240 and 360 frames in Table 3. These results complement the endpoint evaluations in the main paper by illustrating how prediction errors accumulate over time as long-horizon autoregressive simulation becomes increasingly challenging. Figure 8: Illustrating 10 pairs qualitative comparison of real-world observations (upper, labeled by Real Obs.) and autoregressive predictions by RiCo (lower, labeled by RiCo Pred.) in three- and four-ball collision scenarios. RiCo’s predictions are initialized by the balls’ observed positions and estimated velocities in the real-world experiments, then replayed in the simulated environment. Under this setting, smaller distances between corresponding predicted and observed positions indicate greater prediction accuracy. C.6 Real-world Experimental Setup and Simulated Training Real-world experimental setup. We conduct real-world experiments on a standard billiard table with an effective playing area of 2.54×1.272.54× 1.27 m. Each billiard ball has a radius of 33.533.5 m. We consider multi-ball collision scenarios involving three and four balls, capturing their motion and interactions through camera-based observations. Simulated training environment. To investigate zero-shot sim-to-real transfer, we construct a corresponding simulated billiard environment with geometric configurations comparable to those of the real-world setup. The simulation reproduces the table dimensions and ball geometry used in the experiments. To generate diverse collision dynamics, we randomize the initial conditions of the simulated training scenarios. Specifically, the initial position of each ball is randomly perturbed within ±0.25± 0.25 m, while the initial speed of the moving ball is sampled from 1.51.5–2.52.5 m/s. Its initial moving direction is additionally perturbed within ±5∘± 5 relative to the nominal direction. These randomized configurations expose the model to different contact locations, collision angles, and subsequent multi-object interactions. RiCo is trained exclusively on 960 simulated trajectories with a duration 2s, without incorporating any real-world observations during training or performing real-world fine-tuning. Zero-shot evaluation protocol. We evaluate the trained model on five independent real-world experimental sessions, comprising 23 valid ball samples in total. Real-world ball configurations are reconstructed using camera-estimated positions. Starting from the observed initial states, the model predicts subsequent object motion through autoregressive rollout. We compare the predicted ball trajectories against the corresponding real-world observations and report position RMSE at evaluation frames 50, 75, and 100. For reference, we also evaluate trajectories generated by the physics engine. Both sets of predictions are evaluated against the real-world observations.