Paper deep dive
Multi-Person Human Motion Forecasting in Complex Scenes
Serdar Ozsoy, Lars Doorenbos, Juergen Gall
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/28/2026, 4:34:29 AM
Summary
The paper introduces Object-Conditioned Social Diffusion (OCSD), a conditional diffusion model for multi-person human motion forecasting in complex scenes. OCSD integrates motion history, social interactions via a social encoder, and object cues via an object-conditioning mechanism using FiLM modulation. It achieves state-of-the-art results on the HiK and HOI-M3 benchmarks, significantly reducing path error and improving forecast realism and diversity.
Entities (10)
Relation Signals (10)
OCSD → evaluatedon → HiK
confidence 95% · Extensive experiments show that OCSD achieves state-of-the-art results on the Humans in Kitchens (HiK) and HOI-M3 benchmarks.
OCSD → evaluatedon → HOI-M3
confidence 95% · Extensive experiments show that OCSD achieves state-of-the-art results on the Humans in Kitchens (HiK) and HOI-M3 benchmarks.
Juergen Gall → affiliatedwith → Lamarr Institute for Machine Learning and Artificial Intelligence
confidence 90% · Affiliation: Lamarr Institute for Machine Learning and Artificial Intelligence, Germany
Serdar Ozsoy → affiliatedwith → Lamarr Institute for Machine Learning and Artificial Intelligence
confidence 90% · Affiliation: Lamarr Institute for Machine Learning and Artificial Intelligence, Germany
Lars Doorenbos → affiliatedwith → Lamarr Institute for Machine Learning and Artificial Intelligence
confidence 90% · Affiliation: Lamarr Institute for Machine Learning and Artificial Intelligence, Germany
Serdar Ozsoy → affiliatedwith → University of Bonn
confidence 90% · Affiliation: University of Bonn, Germany
Lars Doorenbos → affiliatedwith → University of Bonn
confidence 90% · Affiliation: University of Bonn, Germany
Juergen Gall → affiliatedwith →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Accurately forecasting the movement of people in complex scenes requires reasoning over the past and present state of the entire environment. In this context, effectively incorporating object information and social interactions into a unified framework remains particularly challenging. To address this, we propose Object-Conditioned Social Diffusion (OCSD), a conditional diffusion model that integrates motion history, multi-person interactions, and object cues into a single framework. OCSD uses an object-conditioning mechanism that modulates denoising at every timestep, enabling fine-grained human-object reasoning, and a social encoder that models the interactions between all humans in the scene. As a result, our model naturally handles varying group sizes, complex social interactions, and supports sampling multiple plausible futures. Extensive experiments show that OCSD achieves state-of-the-art results on the Humans in Kitchens (HiK) and HOI-M3 benchmarks. It reduces the two-second path error by 121.5 mm (31.3%) on HiK and 130.5 mm (33.2%) on HOI-M3 compared to prior work, and produces more realistic long-term forecasts.
Tags
Links
- Source: https://arxiv.org/abs/2608.27039v1
- Canonical: https://arxiv.org/abs/2608.27039v1
Trouble viewing inline? Open PDF directly →
Full Text
67,797 characters extracted from source content.
Expand or collapse full text
Multi-Person Human Motion Forecasting in Complex Scenes Serdar Ozsoy †thanks: Corresponding Author Affiliation: University of Bonn, Germany Affiliation: Lamarr Institute for Machine Learning and Artificial Intelligence, Germany https://serdarozsoy.github.io/OCSD-project/ E-mail soezsoy,doorenbos,gall@iai.uni-bonn.de Lars Doorenbos Affiliation: University of Bonn, Germany Affiliation: Lamarr Institute for Machine Learning and Artificial Intelligence, Germany https://serdarozsoy.github.io/OCSD-project/ E-mail soezsoy,doorenbos,gall@iai.uni-bonn.de Juergen Gall Affiliation: University of Bonn, Germany Affiliation: Lamarr Institute for Machine Learning and Artificial Intelligence, Germany https://serdarozsoy.github.io/OCSD-project/ E-mail soezsoy,doorenbos,gall@iai.uni-bonn.de Abstract Accurately forecasting the movement of people in complex scenes requires reasoning over the past and present state of the entire environment. In this context, effectively incorporating object information and social interactions into a unified framework remains particularly challenging. To address this, we propose Object-Conditioned Social Diffusion (OCSD), a conditional diffusion model that integrates motion history, multi-person interactions, and object cues into a single framework. OCSD uses an object-conditioning mechanism that modulates denoising at every timestep, enabling fine-grained human-object reasoning, and a social encoder that models the interactions between all humans in the scene. As a result, our model naturally handles varying group sizes, complex social interactions, and supports sampling multiple plausible futures. Extensive experiments show that OCSD achieves state-of-the-art results on the Humans in Kitchens (HiK) and HOI-M3 benchmarks. It reduces the two-second path error by 121.5 m (31.3%) on HiK and 130.5 m (33.2%) on HOI-M3 compared to prior work, and produces more realistic long-term forecasts. Keywords: Human motion forecasting Diffusion models 1 Introduction Anticipating how people will move in complex scenes is a crucial capability for autonomous systems to operate safely in various domains, including robotics [13, 15, 45] and autonomous driving [7, 23, 24, 37]. However, achieving the accuracy desired to do so is a challenging task. A key part of this challenge lies in understanding the factors that influence human motion at different timescales. Even for short time horizons, accurate human motion forecasting requires more than extrapolating past poses: models must reason jointly about their own history, interactions with other people in the scene, as well as the objects. Longer time horizons introduce even more complexity for forecasting. Specifically, the mapping from observed motion to future behavior of people in complex scenes becomes an inherently ambiguous problem, where multiple motions can be correct given the same context. Short- and long-horizon forecasting are typically considered separately. For instance, short-term works tend to disregard the probabilistic characteristic of human motion and output only the most likely prediction (e.g., [39]). In contrast, works focusing on longer horizons produce stochastic predictions with generative models, but, as a result, tend to underperform deterministic models in terms of local realism [31]. Capturing these different factors within a unified framework remains an open problem. Figure 1: Multi-person human motion forecasts in complex scenes with OCSD. We show the last observed frame on the left. The next two panels display forecast poses after 1 second (middle) and 2 seconds (right), with grey skeletons denoting the ground truth and colored skeletons representing the predictions. The gray skeletons are well aligned with the forecast poses, which shows that our model can generate plausible poses in complex scenes by integrating context. To address this challenge, we propose a conditional denoising diffusion model that holistically integrates the entire scene context. Our model uses an object-conditioning mechanism that modulates the denoising at each timestep, through which objects can impact the predicted future motion. Additionally, we use a social encoder that extracts the social context and integrates it into our model. As a result, our model can capture fine-grained human-object dependencies while also regarding social context, as shown in Fig. 1. We show through extensive experiments that these modeling capabilities translate into strong empirical performance. We evaluate our method on two benchmarks, Humans in Kitchens (HiK) [42] and HOI-M3 [58], on both short- and long-term human motion forecasting. For short horizons, the sequences our model generates outperform those from eight baselines in accuracy of predicted paths and poses, while for longer horizons our method reaches high realism and diversity. For example, on HiK, which has the most people in the scene, we reduce the state-of-the-art two-second path error by 121.5121.5m, representing an improvement of 31.331.3%. For longer forecasts up to ten seconds, our model achieves 25.925.9% higher realism than the previous state-of-the-art approaches. Similarly, our model reduces the two-second path error by 130.5 m (33.2%) on HOI-M3 compared to prior work. In short, our main contributions are as follows: • We introduce OCSD, a conditional diffusion framework that unifies past conditioning, social context, and object signals for multi-person motion forecasting in complex scenes. • We set a new state-of-the-art across various metrics on the HiK and HOI-M3 datasets. 2 Related Works Single-Person Human Motion Prediction. Early works in human motion prediction operate in single-person settings, with both deterministic models that predict a single future sequence and probabilistic models that predict multiple diverse future motions. Deterministic methods typically use Recurrent Neural Networks [14, 19, 30], Graph Convolutional Networks [25, 28, 27], Transformers [2, 9], or Multilayer Perceptron-based [8, 17, 49] approaches, while Generative Adversarial Networks [6, 22] and Variational Autoencoders [56] form the foundations for probabilistic human motion prediction. Subsequent work in probabilistic prediction mostly focuses on increasing the diversity of the predictions [3, 4, 12]. In addition, [29, 52, 54, 57] not only diversify predictions but also offer a way to control the diversity of predicted motions. Recently, diffusion-based models [5, 10, 11, 36, 40] achieve high-fidelity forecasts while providing diversity, for instance, by re-framing the prediction as a masked motion completion task [10], or introducing latent diffusion [5]. Multi-Person Human Motion Prediction. The setting of multi-person human motion prediction introduces an extra challenge of modeling interdependent social dynamics. Transformer-based architectures [20, 32, 46, 47, 50, 53] became a common approach in this domain. For example, [47] uses a local-range encoder for individual motion and a global-range encoder for social interactions. Subsequent work refined this by modeling interactions at a finer granularity. For instance, [32] proposes modeling interactions at the body-part level, while [53] introduces an auxiliary loss to predict the future distance between joints. Besides transformer-based models, [60] proposes computationally efficient modeling of social interaction via an MLP-based framework. Generative models are also used in this context to capture social stochasticity: [55] introduces a dual-level generative modeling framework to separately model independent individual motion and global social interactions, and [44] leverages a diffusion model with an order-invariant aggregation function to flexibly manage a varying number of people. Human Motion Prediction with Object Context. Integrating object information is typically considered separate from social context and has its own set of approaches. As an early method, [26] proposes a two-stage pipeline to first predict future contact maps from a scene point cloud, and then forecasts the pose conditioned on these maps. [38] builds on this with a three-stage pipeline by predicting contact points, root trajectory, and full pose, respectively. While [51] proposes to constrain the whole-body motion by introducing the Signed Distance Function (SDF) volume as a dense global scene representation, [48] uses segmented object instances from 3D point clouds to predict diverse human motion. Only recently, unified models attempt to merge both social and scene context. [1] is an early attempt using GNNs with a pretrained object detector. [31] introduces a diffusion-based model for multi-person settings that uses scene object information while targeting long-term (up to 10s) generation with an emphasis on diversity and plausibility. In contrast, [39] also offered a unified hierarchical framework that operates on scene objects while focusing on more short-term (0.5 - 2s) prediction accuracy. Our model bridges these two directions: we achieve state-of-the-art short-term (0.5-2s) path and pose accuracy while also ensuring long-term (10s) plausibility and diversity with both object and social context. 3 Method Forecasting the motion of multiple people in complex scenes requires modeling several factors: the past motion, the interactions between people in the scene, and the type and location of various objects. We denote the observed motion sequence of a group of P people for a duration of TinT_in frames by a tensor of coordinates Tin∈ℝTin×P×J×3X_T_in ^T_in× P× J× 3, where J represents the number of joints, and we refer to the pose of a single person p at frame t as t(p)x_t^(p). If there are fewer than P people in the scene, the superfluous dimensions are filled with zeros. The object information is encoded as a set of M objects =o1,⋯,oMO=\o_1,·s,o_M\, with each object o represented by N vertices o∈ℝN×3V_o ^N× 3, fixed at the last frame of the observations, and a semantic type y. Given this context, the goal is to forecast the poses of all people in the future ToutT_out frames, Tout∈ℝTout×P×J×3X_T_out ^T_out× P× J× 3, conditioned on the observed sequences TinX_T_in and the object information O. The full sequence thus lasts T=Tin+ToutT=T_in+T_out frames. Learning this mapping from observed motion to future behavior of people in complex scenes is an inherently ambiguous problem: for the same input, multiple plausible motions can be correct, especially for longer-term predictions. Therefore, we frame the motion forecasting task as learning a conditional distribution of future motion based on the observed motion and scene context, for which we rely on conditional diffusion models [18]. The core of our model, OCSD, lies in how it represents and fuses information, which we describe in the sections below. We provide an overview of our method in Fig. 2. Figure 2: Overview of OCSD. Top: Our method takes a sequence of poses for a dynamic number of people in the scene, along with object information. These inputs are split into the self-stream and two conditioning streams for the objects and social context. From this, OCSD predicts the poses of all people for the next ToutT_out frames. Bottom: Details of our main model, following a U-Net architecture. We integrate the social context for each reference person into the bottleneck, while object information is added throughout the network via FiLM modules. 3.1 Input Representations Pose representation. The observed poses TinX_T_in contain the joint coordinates in the global coordinate system. However, directly operating in the global coordinates is suboptimal for pose forecasting [16]. Therefore, we use an alternative representation as in [44], and transform each per-person pose vector by t(p)=[μt(p)∈ℝ3,rt(p)∈ℝ3,qt(p)∈ℝ3J]x_t^(p)= [ _t^(p) ^3,r_t^(p) ^3,q_t^(p) ^3J ]. Here, μt(p) _t^(p) is the global translation given by the 3D world-coordinate position of the hip center, and rt(p)r_t^(p) is the global rotation of the body’s 2D-plane facing direction represented as a 3D rotation. qt(p)q_t^(p) are the local joint positions, representing the 3D positions of the J joints where the person is translated to the origin and rotated such that the hips are parallel to the x-axis. The total dimensionality of t(p)x_t^(p), therefore, is D=6+3JD=6+3J. We further standardize the local joints by the mean and standard deviation of each joint in the training set. Social context representation. After this processing, all pose vectors are represented in their own local coordinate system. While this is important for stable learning [41], it destroys all information about the position of a person relative to the other people in the scene. To use this information in the model, we design a social representation that considers the relative position of other people in the scene from the perspective of each person. In this context, we refer to the person whose perspective is considered as the primary person p⋆p . To obtain the social information for p⋆p , we establish a translation (p⋆)T^(p ) and rotation (p⋆)R^(p ) with regard to their pose in the last observed frame treft_ref. Then, for every person p, we transform their entire pose vector t(p)x_t^(p) into this coordinate system. Using these pose vectors, we construct a pairwise tensor ∈ℝT×P×P×D X ^T× P× P× D where each slice t(p⋆,p) X_t^(p ,p) contains t(p)x_t^(p) from the perspective of p⋆p . From X, we extract two separate representations. These are the self-stream, representing the pose vector of all individual people in the scene (i.e., the diagonal of X), and the social-stream, which contains pose vectors of all other people in the scene relative to the reference person p⋆p . The self-stream is directly fed into the U-Net [35] of fθf_θ. In contrast, the social-stream will be used to control the generation process, as described in Sec. 3.2. Object context representation. To represent the objects present in the scene, we create personalized object information representations in a similar way to the social representation, encoding the relationship between a reference person and the objects in the scene. We canonicalize the vertices of an object oV_o into each person’s p⋆p reference frame using the same translation (p⋆)T^(p ) and rotation (p⋆)R^(p ) as in social context. Besides flattening N vertices to 3N points, we encode the semantics of the object into a one-hot vector. These are then combined into the object representations ~ V by concatenation, leading to a final representation of Do=3N+CD_o=3N+C, with C the number of object types present. 3.2 Integrating Social and Object Context We propose to use two mechanisms to integrate the social and object context representations into the pose forecasting model fθf_θ to control the generation process, which we show schematically in Fig. 2 and describe in turn below. Social integration. The social context is represented in the social-stream obtained from the pairwise relational tensor X. Similar to [44], we pass this information through a learned encoder and integrate the resulting embedding in the bottleneck of the U-Net. We utilize this Social Encoder to temporally downsample the social-stream through three successive blocks, each consisting of three strided 1D convolutions with a SiLU activation function and layer normalization. This encoder outputs a set of features ∈ℝT/8×P×P×d′E ^T/8× P× P× d , and (p⋆,p)e^(p ,p) denotes the encoded representation for person p relative to primary person p⋆p . One challenge to consider is that there can be a dynamic number of people present in the scene. Therefore, to obtain a single and fixed-dimensional social representation for a primary person, (p⋆)e^(p ), we mask out any zero-padded entries of E that are not related to any person. Then, we perform mean pooling across the second P dimension. The final vector (p⋆)e^(p ) summarizes the entire social context from the perspective of p⋆p and is concatenated to the p⋆p ’s self-stream features at the U-Net bottleneck, followed by a 1D Convolution, denoted as “Merge” in Fig. 2. Thereby, the model jointly processes the primary person’s own motion while integrating the surrounding social dynamics. Object context integration. We design our object conditioning as a multi-scale modulation mechanism applied at multiple stages s, which correspond to the features H ∈ [d1d_1, d2d_2, d3d_3, h2h_2, u3u_3, u2u_2] of the U-Net. This process is done for each person p and consists of three parts: First, the canonicalized object vertices are projected via an MLP to match the feature dimension of that stage. The resulting features form a set of M object tokens for each person p, which encode local scene information and can be integrated into the model. Second, the motion features Ht(p,s)H^(p,s)_t for person p at layer s for each timeframe t∈Tt∈ T are processed by cross-attention modules. In this setting, Ht(p,s)H^(p,s)_t acts as the query, and the M projected object tokens serve as keys and values. This way, the attention mechanism produces an embedding that summarizes the objects most relevant to the motion of the person p in frame t at stage s. Finally, we integrate these features via Feature-wise Linear Modulation (FiLM) [33]. The object embedding is passed through a linear projection to predict a per-timeframe scale vector γt(p,s)γ^(p,s)_t and a bias vector βt(p,s)β^(p,s)_t to modulate the motion features: H^t(p,s)=Ht(p,s)⊙(1+γt(p,s))+βt(p,s). H^(p,s)_t\;=\;H^(p,s)_t (1+γ^(p,s)_t )+β^(p,s)_t. (1) In the first half of the U-Net, FiLM modulation is applied through the skip connections. In contrast, we apply it directly to the features in the second half. This process allows object context to precisely influence the motion generation. 3.3 Conditional diffusion model Training. Our diffusion model fθf_θ is trained to predict the clean motion 0X_0 from a noised sample τX_τ based on the past motion and scene context. Let 0X_0 be the full clean motion sequence, which includes the past TinX_T_in and future ToutX_T_out. Given a diffusion noise schedule ατ=1K\ _τ\_τ=1^K with K denoising steps and its cumulative product α¯τ=∏s=1ταs α_τ= _s=1^τ _s, we form the input tensor τX_τ by combining the clean past and the noised future: τ=[Tin;α¯τTout+1−α¯τε],X_τ= [\,X_T_in\;;\; α_τ\,X_T_out+ 1- α_τ\, \, ], (2) where ε∼(0,I) (0,I). Our network fθf_θ is trained to predict the full clean motion 0X_0 from τX_τ, conditioned on the denoising step τ and object context O with a ℒ1L_1 reconstruction objective: ℒx0=τ[‖0−fθ(τ,τ,)‖1].L_x_0=E_τ\! [ \|\,X_0-f_θ\! (X_τ,\,τ,\,O ) \|_1 ]. (3) We modify this base loss by applying a frame mask fraM_fra to compute the loss only for frames t>Tint>T_in , and a presence mask presM_pres to ignore absent individuals. Moreover, we decompose the masked loss into three components in our hybrid pose representation to ensure stable training: root translation μ, root orientation r, and local joint pose q. Each is normalized by its number of valid elements: ℒ=‖Δμ‖1Nμ+‖Δr‖1Nr+‖Δq‖1Nq.L= \| _μ\|_1N_μ+ \| _r\|_1N_r+ \| _q\|_1N_q. (4) Here, Δμ _μ, Δr _r, and Δq _q denote the masked per-element errors for root translation, root orientation, and local joint pose, respectively, and NμN_μ, NrN_r, and NqN_q are the valid element count for each component. Inference. We sample using the standard DDPM reverse process. To ensure the generated future is coherent with the given past TinX_T_in, we apply an inpainting strategy. A single noise vector pastz_past is drawn for the past frames, and at each denoising step τ, a noised version of the past is concatenated with the predicted τ−1X_τ-1 before passing it to the next step: τ−1=[α¯τTin+1−α¯τpast;τ−1predicted]X_τ-1\;=\;[ α_τ\,X_T_in+ 1- α_τ\,z_past\,;\,X_τ-1^predicted] (5) which prevents the observed past from drifting during the sampling process and avoids discontinuities between observed and predicted poses. 4 Experiments We describe datasets, metrics and implementation details briefly below and provide more details in the supplementary material. Datasets. We evaluate our method on two recent multi-person human motion forecasting datasets. Humans in Kitchens [42] (HiK) is a motion-capture dataset of people interacting in four kitchen environments, containing up to 16 people and 50 annotated objects in a scene. Following the standard protocol [44, 31, 39], we train on three kitchens (A, B, and C) and evaluate on curated sequences from kitchen D, focusing on transitional moments. HOI-M3 [58] is a large-scale multi-human multi-object dataset captured in realistic indoor rooms. It contains 199 long sequences with up to 5 people and at least 5 objects per scene. We use the 49 living room sequences publicly available (1-51, except non-released 43 and 45), and set aside 20%20\% as the test set following [39]. Metrics. We assess our model’s performance using four metrics. We evaluate short-term forecasts as in [39] with Path error, i.e., the error of root joint trajectory in millimeters (m), and Pose error, which is the Mean Per Joint Position Error. Long-term predictions are scored with Normalized Directional Motion Similarity [43] (NDMS), which measures the realism of generated motion, and Unique Motion Word Ratio [31] (UMWR), which tests diversity. Implementation Details. We train OCSD for 9090 epochs using the Adam [21] optimizer with a learning rate of 3×10−43× 10^-4. We define one epoch as 20,000 training samples. We follow the protocol from [31, 39] and use the HiK dataset at its original 25 FPS. The HOI-M3 dataset is downsampled from 60 FPS to 30 FPS. For short-term experiments, we provide 1 second of motion as input to predict the next 2 seconds. For long-term experiments, we use 2 seconds of input to predict 10 seconds. For evaluation, as our method is stochastic, we forecast four pose sequences per input motion, compute the corresponding metrics, and report the average performance. Table 1: Short-term pose forecasting results. Baseline results are taken from [39]. Our approach consistently reaches the best results. Dataset Method Path Error (m) Pose Error (m) 0.5s 1.0s 1.5s 2.0s mean 0.5s 1.0s 1.5s 2.0s mean HIK [42] ContAware [26] 138.3 251.3 352.4 430.8 239.3 87.8 117.1 136.1 147.8 106.8 GIMO [59] 143.0 259.7 384.2 487.3 258.6 85.5 121.6 142.0 153.0 109.3 STAG [25] 124.7 245.4 352.4 479.2 239.7 81.7 110.9 132.5 140.9 100.6 MutualDistance [51] 128.7 253.2 372.5 479.2 246.0 82.9 117.2 138.5 148.2 105.9 T2P [20] 88.6 199.6 318.8 447.1 208.7 74.2 108.6 127.5 142.6 96.9 IAFormer [50] 83.9 195.0 311.1 434.9 200.1 71.5 106.5 125.9 137.7 95.0 SAST [31] 86.7 187.4 284.9 398.1 189.0 72.3 101.4 118.0 128.6 93.2 HUMOF [39] 78.8 177.4 278.8 388.4 180.7 71.2 100.6 116.9 127.1 90.2 OCSD 78.9 132.7 197.8 266.9 137.6 69.8 99.4 112.3 121.9 88.4 HOI-M3 [58] ContAware [26] 125.6 239.9 285.4 432.9 236.9 106.2 152.8 174.3 197.1 137.5 GIMO [59] 131.4 247.7 300.9 454.4 255.2 107.9 155.9 182.6 207.1 141.0 STAG [25] 128.1 234.4 289.5 438.1 239.7 102.5 145.0 167.1 185.6 131.2 MutualDistance [51] 83.6 169.7 278.8 402.8 189.9 94.4 137.1 158.2 181.3 125.3 T2P [20] 74.2 168.8 296.9 429.2 194.1 88.0 135.8 160.9 183.2 124.6 IAFormer [50] 69.0 166.6 290.1 423.5 186.3 86.1 135.0 165.9 180.7 121.6 SAST [31] 75.0 166.2 280.4 403.9 184.8 89.2 133.8 167.0 182.9 122.3 HUMOF [39] 67.1 156.6 268.4 393.1 174.6 86.3 129.6 155.0 172.1 117.9 OCSD 52.4 115.5 185.9 262.6 123.5 63.8 93.1 112.5 127.9 87.3 Table 2: Long-term pose forecasting results on HiK. Baseline results are taken from [31]. Unlike previous approaches, our method performs well even for a 10-second horizon. NDMS ↑ UMWR ↑ 2s 4s 6s 8s 10s MRT [47] 0.16 0.09 0.08 0.07 0.07 - HisRep [28] 0.23 0.12 0.07 0.06 0.06 0.06 SiMLPe [17] 0.27 0.15 0.07 0.07 0.06 0.06 TriPod [1] 0.13 0.18 0.14 0.12 0.11 0.11 SAST [31] 0.17 0.41 0.21 0.15 0.14 0.15 OCSD 0.34 0.47 0.45 0.45 0.44 0.44 4.1 Main Results We conduct a comprehensive quantitative evaluation against recent state-of-the-art methods on the two datasets. We show our main results on HiK and HOI-M3 for short-term forecasting in Tab. 1 and long-term forecasting in Tab. 2. Our model demonstrates superior performance in both short-term path and pose accuracy and long-term motion realism and diversity, outperforming specialized methods on their respective benchmarks. Specifically, our method achieves the best performance in all metrics for HiK in the short-term setting, except for the 0.50.5-second path error, where we trail HUMOF by 0.10.1 m. Notably, the longer the time horizon considered, the larger the relative benefit of our method: after one second, we improve over HUMOF by 44.744.7 m, while after two seconds this increases to 121.5121.5 m (31.3%). The pose error results are better for all four time-points considered; however, the margins are narrower. On HOI-M3, a similar trend is visible where OCSD outperforms the baselines in all aspects. The comparison to long-term pose forecasting models in both realism and diversity further validates the benefits of our approach. We keep a high diversity (UMWR) across the entire ten seconds. This demonstrates that our model captures a much wider distribution of plausible human motion. In contrast, predictions by methods such as SAST tend to “freeze” after a certain amount of time, where the people stop moving entirely. We provide the related examples in the supplementary material. Furthermore, we achieve state-of-the-art results for the realism metric NDMS, improving over SiMLPe by 25.9%25.9\%, from 0.270.27 to 0.340.34. This shows that our predicted motion segments are more similar to the actual motion. 4.2 Ablation studies While we discuss the three main ablations regarding object conditioning, social and object context, and loss term below, we provide additional experiments in the supplementary material. Object Conditioning. We validate the design of our object conditioning mechanism by comparing it with two other variants. First, we remove FiLM and use the underlying cross-attention directly to inject the object condition, denoted as “only Cross-Att”. Second, we remove the cross-attention and instead compute a masked mean-pooling over valid object tokens to obtain only a global object context per person and timeframe, denoted as “only FiLM”. From the results in Tab. 3, we find that FiLM itself provides the weakest form of conditioning. Vanilla cross-attention improves upon FiLM by providing more detailed context, as is visible in the 7.87.8 m improvement in the mean path error. Nonetheless, our proposed object context integration module yields further improvements in the results. Specifically, we increase the mean path error by another 4.24.2 m compared to cross-attention and 12.012.0 m with respect to FiLM. Social and Object Context. Next, we validate the benefits of incorporating social interaction and object information against three ablated versions: “self + social”, which removes the entire object-conditioning pathway, “self + object”, which removes the social encoder and merging layers, and “only self”, which removes both social and object context. Table 3: Ablating object integration via cross-attention and FiLM. We report the path and pose error on HiK. Our combined mechanism is important to get the best results. Method Path Error (m) Pose Error (m) 0.5s 1.0s 1.5s 2.0s mean 0.5s 1.0s 1.5s 2.0s mean OCSD 78.9 132.7 197.8 266.9 137.6 69.8 99.4 112.3 121.9 88.4 only Cross-Att. 83.0 138.5 204.5 268.8 141.8 70.3 100.2 113.3 121.6 88.9 only FiLM 87.5 146.3 213.9 284.7 149.6 74.5 102.3 114.3 123.9 91.6 Table 4: The importance of context for short-term forecasting. We show the path and pose error on HiK. Integrating all forms of context leads to the best performance. Method Path Error (m) Pose Error (m) 0.5s 1.0s 1.5s 2.0s mean 0.5s 1.0s 1.5s 2.0s mean OCSD 78.9 132.7 197.8 266.9 137.6 69.8 99.4 112.3 121.9 88.4 self + social 81.6 141.9 213.8 281.5 146.5 69.7 100.3 112.9 124.3 89.1 self + object 83.1 140.1 208.1 277.4 144.7 72.5 102.7 114.9 123.5 90.4 only self 84.5 146.2 217.7 289.8 149.9 70.8 99.2 112.2 122.9 88.5 Table 5: The impact of context for long-term forecasting. On the HiK dataset, realism score (NDMS) increases with the context while diversity score (UMWR) is lower because of the introduced constraints from environment and people. Method NDMS ↑ UMWR ↑ 2s 4s 6s 8s 10s OCSD 0.34 0.47 0.45 0.45 0.44 0.44 self + social 0.33 0.47 0.45 0.44 0.44 0.44 self + object 0.33 0.47 0.45 0.45 0.44 0.44 only self 0.32 0.48 0.46 0.45 0.45 0.45 For short-term forecasting, Tab. 4 shows the effect of removing object and social information. For path error, both context are clearly beneficial. Incorporating social context reduces mean path error by 3.43.4 m, while object information reduces it by 5.25.2 m. By combining both forms, our full model achieves the best result overall, lowering the mean path error by 12.312.3 m. This result demonstrates that the two sources of context are complementary rather than redundant, which is intuitive since the global trajectory is affected by where other people and objects are. In contrast, local pose in short horizons is primarily determined by the person’s own dynamics, so the “only self” model already achieves a low pose error. Therefore, the impact of context is not obvious in the mean over different types of activities. We provide more details on the context contribution in the supplementary material by reporting the metrics for representative activities. For long-term forecasting, Tab. 5 shows that the NDMS realism metric also improves with more context. Interestingly, the diversity of samples as measured by the UMWR metric is slightly higher without any context, as more movements are plausible without any constraints from the environment. Nonetheless, the diversity of the different variants is similar. Table 6: Ablating loss weighting. We report the average path and pose error over four samples on HiK. Using the loss weighting improves the results. Method Path Error (m) Pose Error (m) 0.5s 1.0s 1.5s 2.0s mean 0.5s 1.0s 1.5s 2.0s mean OCSD 78.9 132.7 197.8 266.9 137.6 69.8 99.4 112.3 121.9 88.4 w/o weighting 82.1 139.8 208.2 277.4 144.8 69.6 101.6 117.3 124.8 90.0 Figure 3: Qualitative examples of 10-second predictions for the HiK dataset. We visualize two generated samples for the same input. We show the last observed frame on the left, followed by the final predicted frame from two distinct generated samples. These examples demonstrate the model’s ability to generate diverse predictions, as shown by the purple/yellow person walking to different locations. Figure 4: 2-second prediction for the HOI-M3 dataset. We show the last observed frame on the left. The next two panels display forecast poses after 1 second (middle) and 2 seconds (right). Purple skeletons represent our predictions, while gray skeletons represent ground-truth poses. The model correctly continues the stand up motion from the sofa and the walk motion of the other person. Loss Aggregation. Finally, we ablate the way the diffusion loss is aggregated over channels. We compare to a baseline where we remove the component-wise decomposition of Eq. (4). Instead, we compute a single scalar loss as the mean of the prediction error over all elements. The results in Tab. 6 show that our component-wise decomposition, which assigns more weight to the global positioning of the sequences, is important to reach a good performance. 4.3 Qualitative Results Beyond quantitative metrics, we provide qualitative visualizations to demonstrate the fidelity, realism, and diversity of our generated motions. Specifically, a key advantage of our diffusion framework is the ability to model a distribution of plausible futures. Fig. 3 illustrates this by showing two distinct 10-second predictions sampled from the same input. This capability to generate a wide range of realistic behaviors directly explains our model’s significantly higher UMWR (diversity) score compared to all baselines, which tend to produce only a single, deterministic-looking outcome. In Fig. 4 from HOI-M3, the person sitting on the sofa stands up correctly and continues moving, the walking person continues walking smoothly, and the standing person achieves the same pose as the ground truth at a slightly different but reasonable location. We provide additional qualitative results and discuss limitations in the supplementary material. 5 Conclusion We tackled the challenging problem of forecasting multi-person human motion in complex scenes. We presented OCSD, a model that integrates individual dynamics, social interactions, and object influence into a unified generative framework. By modulating the denoising process with object cues at every timestep and integrating per-person social context, OCSD captures fine-grained relations while maintaining flexibility in group size, interaction complexity, and the diversity of possible futures. Our experiments on the HiK and HOI-M3 benchmarks demonstrate substantial improvements over existing methods. Overall, our results show that holistic approaches like OCSD, which integrate both social and object information, are necessary for accurate motion forecasting in complex scenes. Acknowledgements The work has been supported by the ERC Consolidator Grant FORHUE (101044724). References [1] V. Adeli, M. Ehsanpour, I. Reid, J. C. Niebles, S. Savarese, E. Adeli, and H. Rezatofighi (2021) Tripod: human trajectory and pose dynamics forecasting in the wild. In IEEE/CVF International Conference on Computer Vision, p. 13390–13400. Cited by: §2, Table 2. [2] E. Aksan, M. Kaufmann, P. Cao, and O. Hilliges (2021) A spatio-temporal transformer for 3d human motion prediction. In 2021 International Conference on 3D Vision (3DV), p. 565–574. Cited by: §2. [3] S. Aliakbarian, F. Saleh, L. Petersson, S. Gould, and M. Salzmann (2021) Contextually plausible and diverse 3d human motion prediction. In IEEE/CVF International Conference on Computer Vision, p. 11333–11342. Cited by: §2. [4] S. Aliakbarian, F. S. Saleh, M. Salzmann, L. Petersson, and S. Gould (2020) A stochastic conditioning scheme for diverse human motion prediction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 5223–5232. Cited by: §2. [5] G. Barquero, S. Escalera, and C. Palmero (2023) Belfusion: latent diffusion for behavior-driven human motion prediction. In IEEE/CVF International Conference on Computer Vision, p. 2317–2327. Cited by: §2. [6] E. Barsoum, J. Kender, and Z. Liu (2018) Hp-gan: probabilistic 3d human motion prediction via gan. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, p. 1418–1427. Cited by: §2. [7] I. Batkovic, M. Zanon, N. Lubbe, and P. Falcone (2018) A computationally efficient model for pedestrian motion prediction. In 2018 European Control Conference (ECC), Vol. , p. 374–379. Cited by: §1. [8] A. Bouazizi, A. Holzbock, U. Kressel, K. Dietmayer, and V. Belagiannis (2022) Motionmixer: mlp-based 3d human body pose forecasting. In Thirty-First International Joint Conference on Artificial Intelligence, p. 791–798. Cited by: §2. [9] Y. Cai, L. Huang, Y. Wang, T. Cham, J. Cai, J. Yuan, J. Liu, X. Yang, Y. Zhu, X. Shen, et al. (2020) Learning progressive joint propagation for human motion prediction. In European Conference on Computer Vision, p. 226–242. Cited by: §2. [10] L. Chen, J. Zhang, Y. Li, Y. Pang, X. Xia, and T. Liu (2023) Humanmac: masked motion completion for human motion prediction. In IEEE/CVF International Conference on Computer Vision, p. 9544–9555. Cited by: §2. [11] C. Curreli, D. Muhle, A. Saroha, Z. Ye, R. Marin, and D. Cremers (2025) Nonisotropic gaussian diffusion for realistic 3d human motion prediction. In Computer Vision and Pattern Recognition Conference, p. 1871–1882. Cited by: §2. [12] L. Dang, Y. Nie, C. Long, Q. Zhang, and G. Li (2022) Diverse human motion prediction via gumbel-softmax sampling from an auxiliary space. In 30th ACM International Conference on Multimedia, p. 5162–5171. Cited by: §2. [13] A. F. Foka and P. E. Trahanias (2010) Probabilistic autonomous robot navigation in dynamic environments with human motion prediction. International Journal of Social Robotics 2 (1), p. 79–94. Cited by: §1. [14] K. Fragkiadaki, S. Levine, P. Felsen, and J. Malik (2015) Recurrent network models for human dynamics. In IEEE International Conference on Computer Vision, p. 4346–4354. Cited by: §2. [15] D. Fridovich-Keil, A. Bajcsy, J. F. Fisac, S. L. Herbert, S. Wang, A. D. Dragan, and C. J. Tomlin (2020) Confidence-aware motion prediction for real-time collision avoidance1. The International Journal of Robotics Research 39 (2-3), p. 250–265. Cited by: §1. [16] C. Guo, S. Zou, X. Zuo, S. Wang, W. Ji, X. Li, and L. Cheng (2022) Generating diverse and natural 3d human motions from text. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 5152–5161. Cited by: §3.1. [17] W. Guo, Y. Du, X. Shen, V. Lepetit, X. Alameda-Pineda, and F. Moreno-Noguer (2023) Back to mlp: a simple baseline for human motion prediction. In IEEE/CVF Winter Conference on Applications of Computer Vision, p. 4809–4819. Cited by: §2, Table 2. [18] J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, Vol. 33, p. 6840–6851. Cited by: §3. [19] A. Jain, A. R. Zamir, S. Savarese, and A. Saxena (2016) Structural-rnn: deep learning on spatio-temporal graphs. In IEEE Conference on Computer Vision and Pattern Recognition, p. 5308–5317. Cited by: §2. [20] J. Jeong, D. Park, and K. Yoon (2024) Multi-agent long-term 3d human pose forecasting via interaction-aware trajectory conditioning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 1617–1628. Cited by: §2, Table 1, Table 1. [21] D. P. Kingma and J. Ba (2015) Adam: a method for stochastic optimization. In 3rd International Conference on Learning Representations, Cited by: §4. [22] J. N. Kundu, M. Gor, and R. V. Babu (2019) Bihmp-gan: bidirectional 3d human motion prediction gan. In AAAI Conference on Artificial Intelligence, Vol. 33, p. 8553–8560. Cited by: §2. [23] K. Li, S. Eiffert, M. Shan, F. Gomez-Donoso, S. Worrall, and E. Nebot (2021) Attentional-gcnn: adaptive pedestrian trajectory prediction towards generic autonomous vehicle use cases. In 2021 IEEE International Conference on Robotics and Automation (ICRA), p. 14241–14247. Cited by: §1. [24] K. Li, M. Shan, K. Narula, S. Worrall, and E. Nebot (2020) Socially aware crowd navigation with multimodal pedestrian trajectory prediction for autonomous vehicles. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), p. 1–8. Cited by: §1. [25] T. Ma, Y. Nie, C. Long, Q. Zhang, and G. Li (2022) Progressively generating better initial guesses towards next stages for high-quality human motion prediction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6437–6446. Cited by: §2, Table 1, Table 1. [26] W. Mao, R. I. Hartley, M. Salzmann, et al. (2022) Contact-aware human motion forecasting. In Advances in Neural Information Processing Systems, Vol. 35, p. 7356–7367. Cited by: §2, Table 1, Table 1. [27] W. Mao, M. Liu, M. Salzmann, and H. Li (2019) Learning trajectory dependencies for human motion prediction. In IEEE/CVF International Conference on Computer Vision, p. 9489–9497. Cited by: §2. [28] W. Mao, M. Liu, and M. Salzmann (2020) History repeats itself: human motion prediction via motion attention. In European Conference on Computer Vision, p. 474–489. Cited by: §2, Table 2. [29] W. Mao, M. Liu, and M. Salzmann (2021) Generating smooth pose sequences for diverse human motion prediction. In IEEE/CVF International Conference on Computer Vision, p. 13309–13318. Cited by: §2. [30] J. Martinez, M. J. Black, and J. Romero (2017) On human motion prediction using recurrent neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, p. 2891–2900. Cited by: §2. [31] F. B. Mueller, J. Tanke, and J. Gall (2024) Massively multi-person 3d human motion forecasting with scene context. In European Conference on Computer Vision Workshops, p. 130–147. Cited by: Appendix 0.A, 4th item, Appendix 0.D, §1, §2, Table 1, Table 1, Table 2, Table 2, Table 2, §4, §4, §4. [32] X. Peng, S. Mao, and Z. Wu (2023) Trajectory-aware body interaction transformer for multi-person pose forecasting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 17121–17130. Cited by: §2. [33] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville (2018) Film: visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence, Vol. 32, p. 3942–3951. Cited by: §3.2. [34] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, p. 8748–8763. Cited by: Appendix 0.E. [35] O. Ronneberger, P. Fischer, and T. Brox (2015) U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 234–241. Cited by: §3.1. [36] S. Saadatnejad, A. Rasekh, M. Mofayezi, Y. Medghalchi, S. Rajabzadeh, T. Mordan, and A. Alahi (2023) A generic diffusion-based approach for 3d human pose prediction in the wild. In International Conference on Robotics and Automation (ICRA), Cited by: §2. [37] C. Schöller, V. Aravantinos, F. Lay, and A. Knoll (2020) What the constant velocity model can teach us about pedestrian motion prediction. IEEE Robotics and Automation Letters 5 (2), p. 1696–1703. Cited by: §1. [38] L. Scofano, A. Sampieri, E. Schiele, E. De Matteis, L. Leal-Taixé, and F. Galasso (2023) Staged contact-aware global human motion forecasting. In British Machine Vision Conference, p. 589–594. Cited by: §2. [39] C. Sun, Y. Sun, X. Han, Z. Yang, J. Liu, X. Zhu, S. M. Yiu, and Y. Ma (2025) HUMOF: human motion forecasting in interactive social scenes. arXiv preprint arXiv:2506.03753. Cited by: 2nd item, Appendix 0.A, Appendix 0.D, §1, §2, Table 1, Table 1, Table 1, Table 1, §4, §4, §4. [40] J. Sun and G. Chowdhary (2024) Comusion: towards consistent stochastic human motion prediction via motion diffusion. In European Conference on Computer Vision, p. 18–36. Cited by: §2. [41] K. Sun, C. Lan, J. Xing, W. Zeng, D. Liu, and J. Wang (2017) Human pose estimation using global and local normalization. In IEEE International Conference on Computer Vision, p. 5599–5607. Cited by: §3.1. [42] J. Tanke, O. Kwon, F. B. Mueller, A. Doering, and J. Gall (2023) Humans in kitchens: a dataset for multi-person human motion forecasting with scene context. In Advances in Neural Information Processing Systems, Vol. 36, p. 10184–10196. Cited by: 1st item, Appendix 0.A, §1, Table 1, §4. [43] J. Tanke, C. Zaveri, and J. Gall (2021) Intention-based long-term human motion anticipation. In 2021 International Conference on 3D Vision (3DV), p. 596–605. Cited by: 3rd item, §4. [44] J. Tanke, L. Zhang, A. Zhao, C. Tang, Y. Cai, L. Wang, P. Wu, J. Gall, and C. Keskin (2023) Social diffusion: long-term multiple human motion anticipation. In IEEE/CVF International Conference on Computer Vision, p. 9601–9611. Cited by: §2, §3.1, §3.2, §4. [45] V. V. Unhelkar, P. A. Lasota, Q. Tyroller, R. Buhai, L. Marceau, B. Deml, and J. A. Shah (2018) Human-aware robotic assistant for collaborative assembly: integrating human motion prediction with planning in time. IEEE Robotics and Automation Letters 3 (3), p. 2394–2401. Cited by: §1. [46] E. Vendrow, S. Kumar, E. Adeli, and H. Rezatofighi (2022) Somoformer: multi-person pose forecasting with transformers. arXiv preprint arXiv:2208.14023. Cited by: §2. [47] J. Wang, H. Xu, M. Narasimhan, and X. Wang (2021) Multi-person 3d motion prediction with multi-range transformers. In Advances in Neural Information Processing Systems, Vol. 34, p. 6036–6049. Cited by: §2, Table 2. [48] T. Wang, Z. Song, Z. Lou, Q. Cui, L. Zhang, C. Cheng, H. Wang, X. Tang, H. Li, and H. Zhou (2024) Harmonizing stochasticity and determinism: scene-responsive diverse human motion prediction. In Advances in Neural Information Processing Systems, Vol. 37, p. 39784–39811. Cited by: §2. [49] M. Wei, L. Miaomiao, S. Mathieu, and L. Hongdong (2019) Learning trajectory dependencies for human motion prediction. In IEEE/CVF International Conference on Computer Vision, p. 9488–9496. Cited by: §2. [50] P. Xiao, Y. Xie, X. Xu, W. Chen, and H. Zhang (2024) Multi-person pose forecasting with individual interaction perceptron and prior learning. In European Conference on Computer Vision, p. 402–419. Cited by: §2, Table 1, Table 1. [51] C. Xing, W. Mao, and M. Liu (2024) Scene-aware human motion forecasting via mutual distance prediction. In European Conference on Computer Vision, p. 128–144. Cited by: §2, Table 1, Table 1. [52] G. Xu, J. Tao, W. Li, and L. Duan (2024) Learning semantic latent directions for accurate and controllable human motion prediction. In European Conference on Computer Vision, p. 56–73. Cited by: §2. [53] Q. Xu, W. Mao, J. Gong, C. Xu, S. Chen, W. Xie, Y. Zhang, and Y. Wang (2023) Joint-relation transformer for multi-person motion prediction. In IEEE/CVF International Conference on Computer Vision, p. 9816–9826. Cited by: §2. [54] S. Xu, Y. Wang, and L. Gui (2022) Diverse human motion prediction guided by multi-level spatial-temporal anchors. In European Conference on Computer Vision, p. 251–269. Cited by: §2. [55] S. Xu, Y. Wang, and L. Gui (2023) Stochastic multi-person 3d motion forecasting. In The Eleventh International Conference on Learning Representations, Cited by: §2. [56] X. Yan, A. Rastogi, R. Villegas, K. Sunkavalli, E. Shechtman, S. Hadap, E. Yumer, and H. Lee (2018) Mt-vae: learning motion transformations to generate multimodal human dynamics. In European Conference on Computer Vision, p. 265–281. Cited by: §2. [57] Y. Yuan and K. Kitani (2020) Dlow: diversifying latent flows for diverse human motion prediction. In European Conference on Computer Vision, p. 346–364. Cited by: §2. [58] J. Zhang, J. Zhang, Z. Song, Z. Shi, C. Zhao, Y. Shi, J. Yu, L. Xu, and J. Wang (2024) HOI-m3 3: capture multiple humans and objects interaction within contextual environment. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 516–526. Cited by: 2nd item, §1, Table 1, §4. [59] Y. Zheng, Y. Yang, K. Mo, J. Li, T. Yu, Y. Liu, C. K. Liu, and L. J. Guibas (2022) Gimo: gaze-informed human motion prediction in context. In European Conference on Computer Vision, p. 676–694. Cited by: Table 1, Table 1. [60] Y. Zheng, R. Yu, and J. Sun (2025) Efficient multi-person motion prediction by lightweight spatial and temporal interactions. In IEEE/CVF International Conference on Computer Vision, p. 10844–10853. Cited by: §2. Multi-Person Human Motion Forecasting in Complex Scenes Supplementary Material Appendix 0.A Datasets Here, we provide the full details on the datasets used in our paper: • Humans in Kitchens [42]. HIK is a motion-capture dataset of people interacting in four kitchen environments with minimal scripting and rich everyday activities. Each recording contains a dynamic number of people with a maximum of 16, and 50 annotated objects visible at the same time, providing complex social interactions and dense scene context. Following the standard protocol, we train on three kitchens (A, B, and C) and evaluate on curated sequences from kitchen D, with a focus on transitional moments. • HOI-M3 [58]. HOI-M3 is a large-scale multi-human multi-object dataset captured in realistic indoor rooms. It contains 199 long sequences with up to 5 people and at least 5 objects per scene, and all humans and objects are tracked in 3D. We use object vertices as object representation and 49 livingroom sequences publicly available (1-51, except 43 and 45). We set aside 20%20\% as the test set following [39]. For the HiK dataset, we adhere to the standard evaluation protocol, utilizing the official test split and evaluation script released by [42, 31]. For the HOI-M3 dataset, we evaluated performance on a test set split by using the data processing pipeline of [39]. During inference, we predicted future motions for all frames in these sequences using a sliding window approach with a stride of 11 second. Appendix 0.B Metrics The metrics used in our work are as follows: • Path Error. This metric evaluates the accuracy of the predicted global trajectory. We define it as the Euclidean distance (L2L_2 norm) between the predicted 3D position of the root joint (p^root p_root) and its corresponding ground-truth position (prootp_root). For a future time step t, the path error is: ℰpath(t)=‖p^root−proot‖2E_path(t)=|| p_root^t-p_root^t||_2 (S1) • Pose Error. This metric evaluates the accuracy of the predicted local body configuration, independent of global translation. We use the standard Mean Per Joint Position Error (MPJPE), which is the mean Euclidean distance across all J−1J-1 non-root joints after aligning the predicted and ground-truth poses at the root. For a future time step t, the pose error is: ℰpose(t)=1J−1∑j=1J−1‖(p^jt−p^root)−(pjt−proot)‖2E_pose(t)= 1J-1 _j=1^J-1||( p_j^t- p_root^t)-(p_j^t-p_root^t)||_2 (S2) where p^jt p_j^t and pjtp_j^t are the predicted and ground-truth 3D positions of the j-th joint. • Normalized Directional Motion Similarity (NDMS). For long-term and stochastic forecasting, per-frame accuracy metrics are insufficient due to the fact that they penalize plausible predictions that deviate from the single ground truth sequence. Therefore, we evaluate the realism of generated motion using the Normalized Directional Motion Similarity (NDMS) score [43], which compares generated motion against a distribution of real-world examples. To evaluate a predicted motion sequence χ, NDMS divides it into overlapping short segments, referred to as motion words. For each motion word, which is 88 frames long, the metric calculates a similarity score by finding its Nearest Neighbor (N) in a reference database (D) composed of real motion snippets. This use of a reference database and N search allows the measure to account for the multi-modal nature of human movement. The core similarity function Ψ compares two motion words (x and y) by evaluating two key factors for each joint j at time t: 1. Directional Similarity: The alignment of the joint velocity vectors (˙t,j x_t,j and ˙t,j y_t,j). 2. Magnitude Ratio: The ratio of the velocity magnitudes, ensuring similar speeds. The resulting NDMS score provides a plausibility measure between 00 and 11, where higher values indicate more realistic motion. NDMS penalizes discontinuities which are highly visible to human observers, such as abrupt transitions between the observed and forecast motion. • Unique Motion Word Ratio (UMWR). A good stochastic model should not only produce realistic motions but also avoid generating the same plausible motion repeatedly. For this, we use the Unique Motion Word Ratio (UMWR) introduced in [31]. The UMWR calculation leverages the same underlying mechanism as NDMS. It uses the same reference database D, composed of motion words of 88 frames. For a single-person predicted sequence χ, the UMWR is calculated as the ratio of unique Nearest Neighbors (N) found in the reference set D to the total number of motion words in the sequence. Formally, the metric is defined in [31] as UMWR(χ)=|N(χ1:κ),N(χ2:κ+1),…N(χOPEN|χ|+1−κ:|χ|)||χ|+1−κUMWR(χ)= |\N( _1:κ),N( _2:κ+1),…N( _|χ|+1-κ:|χ|)\||χ|+1-κ (S3) where κ=8κ=8 is the length of the motion words and χ predicted motion of the selected person with last observed frame. A score closer to 1.01.0 signifies high diversity, as the model generates a wide variety of movements that map to many different real motion words. A low score indicates repetitive or frozen motion, where the entire sequence maps to only a few unique motion words from D. Appendix 0.C Additional Implementation Details We trained all models on a single RTX-40904090 GPU. To accommodate memory constraints during training, we adjusted the batch size according to the prediction horizon: we used a batch size of 6464 for short-term (22-second) prediction models and a batch size of 1616 for long-term (1010-second) prediction models, where we also adjust the learning rate to 1×10−41× 10^-4 to accommodate the reduced batch size. Moreover, in the long-term setting, each temporal feature sequence at the bottleneck (h1h_1) is passed through multi-head self-attention with pre-norm and a residual connection to capture long-range temporal dependencies within each local context. Each residual block uses two successive dilated temporal convolutions (d=2 and d=4) instead of unit dilation to extend the temporal receptive field without increasing parameter count. We employ a dynamic sampling strategy. Rather than using fixed, strided subsequences, each training sample is a clip randomly selected from a valid start position within any sequence. We use 1000 denoising steps. For inference on the HOI-M3 dataset, our approach requires as SAST 2s. To put this into context, HUMOF reports 43ms. Appendix 0.D Additional Qualitative Results In this section, we show further qualitative results of our model for both datasets. We use SAST [31] for comparison, as the code for HUMOF [39] was not publicly available at the time of preparing the manuscript. 0.D.1 HiK dataset: 2-second prediction In Fig. S1, we visualize the predicted motions against ground truth (GT) for a 22-second prediction window. Our model simultaneously predicts the future motions of all individuals in the scene. As demonstrated in the figure, our method generates highly accurate future poses. Even for the purple/yellow skeleton, which moves most in the scene, the predicted motion aligns closely with the ground truth (gray skeleton), making them difficult to distinguish even at frame 74. This validates the model’s ability to predict future motion precisely. In addition, examples for the whiteboard and armchair activities are provided in Fig. S2. (a) Frame 24 (t=1st=1s) (b) Frame 74 (t=3st=3s) (c) Bird’s-eye Figure S1: Qualitative comparison on the HiK dataset (2-second prediction). We compare our predictions to the ground truth (GT). Red/Blue: Past GT poses. Green/Yellow: Predicted future poses. Gray: Future GT for comparison. Purple/Yellow: Predicted future and past GT poses of a person with large motion. The long gray and purple lines show the GT and predicted trajectory, respectively. OCSD’s predictions overlap significantly with the GT (Gray), indicating high accuracy. (a) whiteboard (b) sitting down Figure S2: Object interactions on the HiK dataset (2-second prediction). Each row shows the last observed frame (Frame 24), 1-second prediction (Frame 49), and 2-second prediction (Frame 74). Visualization scheme follows Fig. S1. Row (a) shows the writing activity with interaction with the whiteboard, while row (b) shows the sitting down activity with interaction with the armchair. Comparison with SAST (2-second). In Fig. S3, we compare our method (OCSD) against the SAST baseline. Both models are trained to predict a 22-second horizon given a 11-second input. While our model correctly infers the walking trajectory of the purple/yellow person, SAST predicts that the person will stay close to the last observed position. (a) OCSD (ours) (b) SAST Figure S3: Qualitative comparison with SAST (HiK dataset, 2-second prediction). Each row shows the last observed frame (Frame 24), 11-second prediction (Frame 49), and 22-second prediction (Frame 74). Visualization scheme follows Fig. S1. SAST (row b) does not forecast that the purple person walks from the left group to the right group, while OCSD (row a) successfully predicts the walking motion. 0.D.2 HOI-M3 dataset: 2-second prediction In column (c) of Fig. S4, the central person continues to walk. The predicted motion follows with the correct poses but a little bit slower. In the mean time, the interaction of the left person with the chair continues closely to the real motion. The model correctly identifies the affordance for interaction and also generates a valid walking path. (a) Frame 29 (t=1t=1) (b) Frame 59 (t=2st=2s) (c) Frame 89 (t=3st=3s) Figure S4: 2-second prediction on the HOI-M3 dataset. Purple skeletons represent our predictions, while gray skeletons represent GT. The model accurately captures the pose dynamics over the 2-second horizon. 0.D.3 HiK dataset: 10-second prediction In Fig. S5, we examine the stochastic nature of our model by generating multiple predictions for a single 10-second scenario. While column (a) shows the initial state, columns (b) and (c) display two different samples generated by our model for frame 299. As observed, the model generates diverse but plausible futures. The purple/yellow person walks to different spatial locations in sample 1 versus sample 2. This confirms that OCSD is not collapsing to a single deterministic mean, but is capable of modeling the natural variance in human motion over long horizons. (a) Last Ground Truth (b) Prediction Sample 1 (c) Prediction Sample 2 Figure S5: Qualitative comparison on the HiK dataset (10-second prediction). We visualize two stochastic samples for the same scene. (a) Last observed GT at t=2t=2 s (frame 49). (b - c) Final predicted frame at t=12t=12 s (frame 299) from two distinct generated samples. Visualization scheme follows Fig. S1. The purple/yellow person walks to different locations in (b) and (c). This demonstrates the model’s ability to generate diverse predictions. Comparison with SAST (10-second). We further compare long-term prediction capabilities in Fig. S6. The SAST model (row b) forecasts that the purple/yellow person sits down at a location where there is no chair. In contrast, OCSD (row a) maintains motion plausibility throughout the 1010-second prediction window. (a) OCSD (ours) (b) SAST Figure S6: Qualitative comparison with SAST (HiK dataset, 10-second prediction). Each row shows the last observed GT at Frame 49 (t=2t=2 sec), prediction at Frame 174 (t=7t=7 sec), and prediction at Frame 299 (t=12t=12 sec). Visualization scheme follows Fig. S1. For SAST, the purple/yellow person hallucinates a chair and sits down at a location where there is no chair (row b). Our model produces plausible motion for the persons (row a) over the full 1010-second prediction horizon. Appendix 0.E Limitations (a) Frame 29 (t=1t=1) (b) Frame 59 (t=2st=2s) (c) Frame 89 (t=3st=3s) Figure S7: Limitation regarding moving objects. Purple skeletons represent our predictions, while gray skeletons represent GT. Since the future position of the objects are unknown for the model during the prediction interval, predictions deviate from the actual human-object interaction. This is a 2-second prediction on the HOI-M3 dataset. The first limitation regards moving objects in the prediction interval. Our model uses the last observed positions of the objects for that interval. This avoids providing the model with oracle knowledge of future object positions, but it means that predictions cannot adapt to objects that move in the future. For example in Fig. S7, the positions of the bag, dumbbell and book change during the prediction interval, and the predicted motions remain anchored to their last observed positions. This causes predictions to deviate from the actual human-object interaction, and is observed in a subset of HOI-M3. HiK is largely unaffected since their objects are mostly static. Addressing this would require jointly predicting the future positions of the objects alongside human motion, a promising direction for future work. A second limitation concerns the semantic object representation. Our one-hot encoding for object category information is effective for the closed object vocabularies of HiK and HOI-M3, but it is not suitable for open-vocabulary generalization. Preliminary experiments with CLIP [34] text embeddings of object names did not improve our approach. As future work, vision-language embeddings could be integrated into the model for more generalizable object representations. Appendix 0.F Additional Experiment Results We perform a range of additional ablation studies to further justify the components of our method, report additional metrics focused on motion quality, evaluate the effect of the number of samples on the diversity, and report best-of-K results instead of the average score. 0.F.1 Social encoder design We validate our social encoder design by showing that our mean pooling outperforms cross-attention for encoding social information in Tab. 1(a), and that injecting this into the bottleneck gives better results than doing so throughout the network in Tab. 1(b). Table S1: Social Encoder ablation on HiK. Replacing mean pooling with cross-attention or integrating social context at multiple U-Net layers instead of the bottleneck increases mean path error (m) and pose error (m). (a) Mean pooling vs. Cross-attention Method Path Error↓ Pose Error↓ Mean pooling 137.6 88.4 Cross-attn 142.3 91.0 (b) Bottleneck vs. Multi-layer Social context Path Error↓ Pose Error↓ Bottleneck 137.6 88.4 Multiple layers 140.2 89.9 0.F.2 Context contribution While both social and object context have a clear positive effect on the path error, the averaged results in the main paper do not show the full picture for the pose error. To demonstrate this, we report the results of Tab. 4 for three representative activities in Tab. S2. We find that for the walking activity, where context is important for the path but not necessarily for the pose, including the context slightly increases the pose error. In contrast, for the more static, object-centered social activities, such as interacting with a coffee machine and the surrounding people, the inclusion of this context leads to better predictions. For the path error, context is always helpful. Table S2: Context effect per activity on HiK. Context improves mean path error for all three representative activities. For pose error, benefits provided by context are only evident in the object-centered social activities sink and coffee. Method Path Error (m) ↓ Pose Error (m) ↓ walking sink coffee walking sink coffee OCSD 306.0 57.3 73.6 100.5 57.3 58.3 self + social 328.3 78.3 94.0 98.6 65.0 62.2 self + object 319.8 68.4 84.9 99.5 65.8 62.2 only self 334.1 81.4 96.0 98.3 63.8 62.4 0.F.3 U-Net vs. DiT We exchange our U-Net backbone for a DiT, keeping the conditioning and training protocol identical. Tab. S3 shows that DiTs overfit on the relatively small multi-person motion forecasting datasets with object information and U-Nets remain the better choice. Table S3: U-Net vs. DiT backbone on HiK. Diffusion transformer performs worse than U-Net. Backbone Path Error (m) ↓ Pose Error (m) ↓ 0.5s 1.0s 1.5s 2.0s mean 0.5s 1.0s 1.5s 2.0s mean U-Net 78.9 132.7 197.8 266.9 137.6 69.8 99.4 112.3 121.9 88.4 DiT 88.5 148.6 223.6 294.3 154.0 74.7 103.2 121.3 133.7 94.5 0.F.4 Additional motion quality metrics We report four additional metrics on HOI-M3 in Table S4: Foot sliding (m/s), Interpenetration depth (m), Interpenetration rate, and the Mean Per Joint Velocity Error (MPJVE) (m/s). We compare our results against ground truth (GT) motion. Foot sliding of predictions is within 5% of the GT. Interpenetration statistics are likewise close to GT. Note that ground-truth motion has penetrations as well due to approximations of the object surfaces. The results show that OCSD does not suffer from strong motion quality artifacts. This is also evident from the NDMS scores, which explicitly compares the velocity-magnitude ratios against real motion, reported in the paper. Table S4: Additional motion quality metrics on HOI-M3. OCSD closely matches ground truth foot sliding and penetration statistics. Method Foot sliding (m/s) Interp.depth (m) Interp.rate MPJVE (m/s) GT 219.3 38.5 0.0338 N/A OCSD 208.2 42.3 0.0343 330.2 0.F.5 Number of evaluation samples Tab. S5 shows that increasing the number of samples used for evaluation from 4 to 16 increases the UMWR, especially for 2-second predictions. This confirms that diversity is not saturated at four samples. Table S5: Effect of sample count on diversity (UMWR) on HiK. Diversity scores increase with more samples. Samples UMWR ↑ 2s 4s 6s 8s 10s 4 0.47 0.45 0.45 0.44 0.44 16 0.56 0.48 0.46 0.46 0.45 0.F.6 Best-of-K analysis Tab. S6 reports best-of-K path and pose errors on HiK for K ∈ 1,5,10,20\1,5,10,20\. Results are already good for K=1K=1 compared to the baselines, but they get even better for higher values. This supports the previous results, showing that OCSD produces diverse, yet consistently high-quality predictions. Table S6: Best-of-K analysis on HiK. Accuracy improves steadily with K. OCSD produces diverse, yet consistently high-quality predictions. K Path Error (m) ↓ Pose Error (m) ↓ 0.5s 1.0s 1.5s 2.0s mean 0.5s 1.0s 1.5s 2.0s mean 1 79.0 131.0 200.8 270.9 139.0 69.5 101.8 115.0 121.8 89.0 5 57.5 90.7 134.8 181.4 95.5 54.2 75.0 87.5 93.1 68.3 10 51.8 79.2 116.7 156.1 83.5 50.1 68.3 79.8 85.1 62.5 20 46.5 69.3 103.0 137.4 74.0 47.1 63.3 74.0 79.4 58.3