Paper deep dive
Unified Pedestrian Path Prediction Using Inverse Reinforcement Learning
Šimon Sukup, Ariyan Bighashdel, Pavol Jancura
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/22/2026, 3:19:05 AM
Summary
This paper adapts the Spatial-Temporal Graph Attention Network (STGAT) to a unified pedestrian path prediction (PPP) framework based on Markov Decision Processes (MDP). It introduces new state and action definitions to support deterministic and stochastic policies, one-time (ODM) and sequential decision-making (SDM), and reinforcement learning algorithms like REINFORCE and Proximal Policy Optimization (PPO). The study demonstrates that reformulating the learning task from standard supervised learning to inverse reinforcement learning (IRL) improves prediction performance on benchmark datasets (ETH, UCY) compared to previous shallow network studies.
Entities (12)
Relation Signals (7)
Lemmens et al. → introduced → Unified PPP Framework
confidence 95% · Lemmens et al. introduced unified framework for PPP [4].
STGAT → isadaptedto → Unified PPP Framework
confidence 95% · This paper adapts the Spatial-Temporal Graph Attention Network (STGAT) to a unified pedestrian path prediction framework
Unified PPP Framework → isformulatedas → Markov Decision Process
confidence 95% · The framework defines PPP as a decision-making problem in the form of a Markov decision process (MDP).
Inverse Reinforcement Learning → improves → Prediction Performance
confidence 92% · The newly designed learning task formulations achieve performance improvement on all chosen benchmark datasets compared to the standard supervised learning task.
STGAT → isusedon → ETH Dataset
confidence 90% · The proposed learning-task formulations improve prediction performance across the selected benchmark datasets... ETH dataset
STGAT → isusedon → UCY Dataset
confidence 90% · The proposed learning-task formulations improve prediction performance across the selected benchmark datasets... UCY dataset
Unified PPP Framework → supports → Inverse Reinforcement Learning
confidence 90% · The resulting formulations support... reinforcement-learning algorithms including REINFORCE and proximal policy optimization.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Pedestrian path prediction is crucial for enhancing the safety of autonomous vehicles and advanced driver-assistance systems. Previous studies explored different learning-task formulations for pedestrian path prediction and compared these formulations using shallow neural networks, but did not extend this analysis to more complex deep-learning models. This paper adapts the Spatial-Temporal Graph Attention Network (STGAT) to a unified pedestrian path prediction framework and introduces state and action definitions specific to STGAT. The resulting formulations support deterministic and stochastic policies, one-time and sequential decision-making, and reinforcement-learning algorithms including REINFORCE and proximal policy optimization. The proposed learning-task formulations improve prediction performance across the selected benchmark datasets compared with the standard supervised-learning formulation. These results demonstrate that reformulating the decision process and training objective can improve an advanced pedestrian trajectory prediction architecture and may provide a path toward improving other graph-based prediction models.
Tags
Links
- Source: https://arxiv.org/abs/2608.15929v1
- Canonical: https://arxiv.org/abs/2608.15929v1
Trouble viewing inline? Open PDF directly →
Full Text
46,856 characters extracted from source content.
Expand or collapse full text
1Eindhoven University of Technology, Eindhoven, The Netherlands 2Delft University of Technology, Delft, The Netherlands Corresponding author: s.sukup@student.tue.nl Unified Pedestrian Path Prediction Using Inverse Reinforcement Learning Šimon Sukup1, Ariyan Bighashdel1,2, and Pavol Jancura1 Abstract Pedestrian path prediction (P) is crucial for enhancing the safety of autonomous vehicles (AVs) and advanced driving assistance systems (ADAS). Previous studies explored various learning task formulations for P and performed comparative analyses of the formulations on shallow neural networks. However, such studies did not extend to the more complex deep learning models. This paper utilizes a unified framework for P and adapts a widely used baseline model, the Spatial-Temporal Graph Attention Network (STGAT), to different learning task formulations. New state and action definitions specific to STGAT are introduced. The newly designed learning task formulations achieve performance improvement on all chosen benchmark datasets compared to the standard supervised learning task. This paper demonstrates, for the first time, that using alternative formulations, as opposed to the widely used supervised learning, can improve an advanced P model under a unified P framework. The designed formulations have a significant potential for improving a wide range of state-of-the-art (SOTA) architectures. †aftertitle: Fig. 1. Proposed RL formulation. STGAT [1] jointly encodes and predicts all pedestrians in a scene. For optimization, the batched output is decomposed into parallel per-pedestrian episodes εii=1N\ _i\_i=1^N governed by one shared MDP template. Each state combines the encoded scene context with previously predicted positions for pedestrian i. The D-LSTM emits either one trajectory-level action (ODM) or 12 next-position actions (SDM); expert demonstrations define the distance-based return used by REINFORCE or PPO. Code: https://github.com/Simsuk/IRL_STGAT I Introduction Human errors contribute significantly to road traffic accidents [2]. Advancements in AVs and ADAS have increased the potential to reduce such human-caused incidents and enhance road safety. A critical component of realizing this potential is the ability to accurately predict pedestrian behaviour, which allows AVs to comprehend their environment more effectively. The P challenge lies in the crowd interactions and the underlying decision-making process of each pedestrian. These aspects necessitate various learning task formulations introduced throughout the literature [3]. To address the problem of objective comparison of different learning task formulations, Lemmens et al. introduced unified framework for P [4]. The framework defines P as a decision-making problem in the form of a Markov decision process (MDP). Lemmens et al. demonstrated improvement in the performance of a shallow neural network policy when transferring from standard supervised learning settings to inverse reinforcement learning (IRL) settings or more advanced reinforcement learning (RL) algorithms [4]. However, the work has not extended the framework to more advanced deep learning architectures that are predominant in the field of P. This paper addresses an unresolved research question of whether the framework can enhance more advanced models. It contributes by extensive modification of the framework to adopt the STGAT [1], a widely used advanced architecture for P. STGAT was utilized as a stochastic and deterministic policy through the unified P framework while using advanced RL algorithms. The framework was extended with novel state and action definitions to design new learning task formulations specific to STGAT, including supervised learning for sequential decision-making and modified stochastic policy gradient one-time decision-making. Furthermore, this paper introduced two value network architectures specific to STGAT based on the designed state-actions pairs. The last contribution includes an ablation study on a newly defined sequential decision-making process. This paper demonstrates that transitioning from supervised learning tasks to inverse reinforcement learning (IRL) with advanced algorithms can improve the performance of more complex P models. The current results and further extension of the framework can potentially lead to the development of better P models based on the graph attention network (GAT) architecture and improve a wide range of current SOTA architectures for P. I Related Work The STGAT leverages deep learning techniques for time series forecasting and interaction modelling [1]. This sections, therefore, covers the preceding works in the field that led to the development of STGAT and compares it to other literature. Furthermore, the motivation for the use of the unified pedestrian path prediction framework is explained with respect to previous works in the field. I-A Advances in Pedestrian Path Prediction and Interaction Modelling Early P models lacked in generalization ability and scalability[5], [6] leading to the rise of deep learning methodologies. Significant attention in P has been given to the use of recurrent neural networks (RNNs), long-short-term-memory networks (LSTMs) and more advanced architectures based on gated recurrent units (GRUs) or transformers [7],[8],[9],[10], [11]. Key studies, such as those by Alahi et al., have utilized LSTMs to address temporal dependencies within the data [12]. Aside from temporal dependencies, a series of studies have shown that understanding social interactions play a crucial role in P. The development of methods for modelling intentions and social behaviour drove research into techniques such as spatio-temporal graphs which integrate both spatial and temporal data. Key contributions to this branch of the field include the Social Attention and Trajectron models which use LSTMs to manage structured sequence data effectively [13],[14]. Recent publications have shown the importance of graph-structured networks in modelling interactions, like Graph Convolutional Networks (GCNs) [15] and GATs [16]. STGAT, or Spatio-Temporal Graph Attention Network, represents a significant improvement while building on the GAT methodology for modelling of dynamic interactions among pedestrians [1]. This model is distinct in its ability to not only account for spatial interactions among individuals but also to capture the temporal evolution of these interactions over time. At the core of STGAT is the integration of GAT with LSTM networks. The GAT component effectively aggregates the hidden states from multiple LSTMs, allowing the model to assign variable importance to different pedestrians based on their proximity and relative significance in the scene. STGAT is utilized in this paper due to its wide use in the field and high performance, as well as being the foundation for current SOTA architectures. I-B Unified Pedestrian Path Prediction Framework While the majority of state-of-the-art methods for P featured learning directly from ground truth data through supervised learning, there have been instances of research that utilized alternative formulations. Such a significant line of research in this domain involves generative models. Gupta et al. introduced a generative adversarial network (GAN) formulation which is based on a min-max game between the trajectory generator and a discriminator where the generator distinguishes between real and generated trajectories [17]. Extensions of this include the use of IRL while utilizing algorithms based on an algorithm based on generative adversarial imitation learning algorithm (GAIL) [18], such as SA-GAIL [19]. Other works used generative adversarial imitation learning that involves learning from expert demonstrations through rewards received from interactions with the environment. While these distinct models with different learning formulations are often presented with improvement, it is often not clear if the improvement is made by the model architecture and modules or the new learning task formulation. To address this challenge, Lemmens et al. introduced a unified P framework [4]. The original paper [4] utilized a simple shallow neural network which did not guarantee that the found performance improvement with advanced RL algorithms would be present while using more advanced models. Our work therefore extends the framework by designing new state-action formulations and adopting widely used advanced deep learning architecture STGAT [1]. I Methods The extension of the unified framework required extensive modification of the codebase while introducing novel state-action definitions, value network architecture and loss definitions in order to utilize the architecture of the STGAT. While the STGAT paper [1] utilized simple supervised learning, this work treats path prediction as a reinforcement learning task where different framework settings lead to one of the newly designed formulations. I-A Reinforcement Learning Problem Setup RL is a machine learning paradigm where an agent learns to make decisions by interacting with an environment, modelled as a Markov decision process (MDP). In RL, the agent’s goal is to discover a policy, a strategy for choosing actions based on states, that maximizes the cumulative reward over time. MDP is defined by the following components. S represents the state space, A is the action space, the reward function ℛ:×→ℝR:S×A assigns a reward value ℛ(s,a)R(s,a) for taking action a in state s, the state transition function :×→T:S×A describes how the state changes, after taking an action a in the state s. The discount factor γ is a value between 0 and 1 that determines the weight of future rewards compared to immediate ones. Finally, the initial state distribution b0:→[0,1]b_0:S→[0,1] gives the probability b0(s)b_0(s) of starting in state s. The decision of an agent can either be based on stochastic policy πθ(a∣s):S×A→[0,1] _θ (a s ):S× A→[0,1], which assigns probability to a state-action pair, or deterministic policy μθ(a∣s):S→A _θ (a s ):S→ A, which assigns single action to every state. In this work, the policy is a mapping parameterized by the chosen learnable parameters of STGAT, denoted θ. In this paper, the goal is to observe the trajectory at the initial time steps 11 to Toτ=8T_o^τ=8 of all pedestrians in the observed scene and forecast the future trajectory steps steps at times tτt^τ for Toτ+1≤tτ≤Toτ+TfτT_o^τ+1≤ t^τ≤ T_o^τ+T_f^τ, where Tfτ=12T_f^τ=12. For each tτt^τ the position of the pedestrian is represented by a vector of coordinates in the scene ptτp_t^τ =(xtτ,ytτ)(x_t^τ,y_t^τ). The observed trajectory is then denoted as τf=p1,…,pToτ _f=\p_1,…,p_T_o^τ\ while the forecasted trajectory τ^f=p^Toτ+1,…,p^Toτ+Tfτ τ_f=\ p_T_o^τ+1,…, p_T_o^τ+T_f^τ\, where p^tτ p_t^τ denotes a predicted position at time step T0τ+1≤tτ≤TfτT_0^τ+1≤ t^τ≤ T_f^τ. Let one-time decision-making (ODM) be a process where the initial state of observed steps is followed by a single action that represents all 1212 predicted steps for the pedestrian. The agent receives a single reward from the environment. Let ϵε denote the set of states, actions and rewards for a trajectory, also called an episode. For ODM, there is single step in episode Tε=1T =1 and ε=(s0,a0,r0),(s1) = \ (s_0,a_0,r_0 ), (s_1 ) \. Using ODM, the single action incorporates all pedestrian steps from Toτ+1T_o^τ+1 to TfτT_f^τ and action becomes a0=(x^Toτ+1,y^Toτ+1,x^Toτ+2,y^Toτ+2,…,x^Toτ+Tfτ,y^Toτ+Tfτ)a_0= ( x_T_o^τ+1, y_T_o^τ+1, x_T_o^τ+2, y_T_o^τ+2,…, x_T_o^τ+T_f^τ, y_T_o^τ+T_f^τ ) where x^tτ,y^tτ x_t^τ, y_t^τ are the predicted coordinates by the policy. While ODM is standardly used in the literature, it does not appropriately reflect the decision-making process of a pedestrian. The SDM is therefore employed, which considers the pedestrian steps as separate actions and each episode then becomes ε=(s0,a0),…,(sTε−1,aTε−1),(sTε) = \ (s_0,a_0 ),…, (s_T -1,a_T -1 ), (s_T ) \ with Tε=12T =12 steps in the episode. The action becomes atϵ=(x^Toτ+1+tϵ,y^Toτ+1+tϵ)a_t^ε= ( x_T_o^τ+1+t^ε, y_T_o^τ+1+t^ε ) for any step in the episode 0≤tϵ<Tϵ0≤ t^ε<T^ε. Compared to RL, in IRL the real rewards are unknown and are estimated from the expert behaviour. In P, the expert behaviour is observed in the ground truth trajectories. Taking the IRL approach, let the estimated reward function be defined by a distance function, squared L2 norm, ℛ(stϵ,atϵ)=rtϵ=∥((s^tϵ,a^tϵ),(stϵ,atϵ)∥22,R(s_t^ε,a_t^ε)=r_t^ε=\|( ( s_t^ε, a_t^ε ), (s_t^ε,a_t^ε )\|_2^2, (1) where index tϵt^ε denotes a time step in the episode, s^tϵ,a^tϵ s_t^ε, a_t^ε are the current state and action coming from the predicted trajectory according to policy. The stϵ,atϵs_t^ε,a_t^ε denote ground truth states and actions. Commonly, the rewards received when starting from a specific state are discounted in order to allow a choice of focus on future states. This results in the definition of discounted return G(ε∣stϵ)=∑l=tϵTϵ−1γl−tϵrl.G ( s_t^ε )= _l=t^ε^T^ε-1γ^l-t^εr_l. (2) The state value function is defined as, V(s,θ)=(ε∣stϵ=s)∼pθ[G(ε∣stϵ=s)]V(s;θ)=E_ ( s_t^ε=s ) p_θ [G ( s_t^ε=s ) ] (3) The pθp_θ denotes either stochastic policy πθ _θ or deterministic policy μθ _θ depending on the experiment. (ε∣stϵ=s)∼pθE_ ( s_t^ε=s ) p_θ denotes expectation over episodes starting from state s, when following the policy pθp_θ. According to the policy gradient theorem[20], the parameters are updated according to the objective function, which is the expected value of initial states J(θ)=s0∼b0(s)V(s0,θ).J(θ)=E_s_0 b_0(s)V(s_0;θ). (4) where s∼b0(s)E_s b_0(s) denotes expectation over initial states sampled from the initial state distribution. The parameters of the policy are then updated using a standard stochastic gradient descent algorithm with learning rate α, θk+1=θk+α∇θJ(πθk). _k+1= _k+α _θJ ( _ _k ). (5) I-B STGAT Architecture and Implementation STGAT comprises several modules: the encoder, intermediate state vectors, and the decoder. The encoder uses two separate LSTM modules, one for modelling temporal correlations between interactions in time, denoted as G-LSTM, and the second M-LSTM to model the spatial interactions. The STGAT models the interactions in the encoder using only one scene of pedestrians at a time. Noise is added to the outputs of the encoder for multimodality. The decoder comprises of a single LSTM (D-LSTM) that takes the intermediate state vectors as input sequences and outputs the next steps of each pedestrian separately. In the first two phases of training, taking 250 epochs, the STGAT learns to reconstruct the observed trajectory through the encoder. In the third phase, the model trains the whole pipeline, including the D-LSTM in 150 epochs. After pretraining on the first 2 phases, the model checkpoint was reused for the third phase, where the formulations described in Section I-C and Section I-D were used. I-C ODM Formulations I-C1 SL-MSE The supervised learning mean-square-error (SL-MSE) formulation corresponds to the ODM process. Because STGAT considers the whole scenes of pedestrians as single input in order to model the interactions, the states are defined as the set of 8 observed steps of all pedestrians in the scene. The action is defined as the next 12 steps of a single pedestrian. After the first action, the final state becomes the whole trajectory of 20 steps. For SL-MSE the discount factor is λ=0λ=0. To enable multimodality, Gaussian noise was incorporated into the input states for training and evaluation. Using equations (1), (2) and (3) the objective function gradient simplifies to ∇θJ(θ) _θJ(θ) =s0∼b0(s0),(ε∣s0)∼πθ[∑tτ=Toτ+1Toτ+Tfτ∇θ((x^tτ−xtτ)2 =E_s_0 b_0(s_0),\,( s_0) _θ [ _t^τ=T_o^τ+1^T_o^τ+T_f^τ _θ (( x_t^τ-x_t^τ)^2 +(y^tτ−ytτ)2)], +( y_t^τ-y_t^τ)^2 ) ], (6) where tτt^τ denotes the corresponding timestamp in the trajectory, xtτ^,ytτ x_t^τ, y_t^τ are predicted coordinates, and xtτ,ytτx_t^τ,y_t^τ. I-C2 SL-MSE-SPG The stochastic gradient policy variant of the supervised learning formulation required modification of the model. Because stochastic policies in RL commonly use Gaussian distributions to sample every next step of each pedestrian sequentially, the noise added to the outputs of the encoder of STGAT was fixed for each initialization of the model. This allowed finding the true probabilities by sampling from output Gaussian distributions with a fixed standard deviation of 0.05. This was necessary in order to preserve the model capacity, which is dependent on the encoded state’s dimensions. The states and actions were defined in the same way as in the case of SL-MSE. Using the stochastic policy gradient theorem[20], (1), (2) and (3) results in an expression ∇θJ(θ)=s0∼b0(s0),(ε∣s0)∼πθ[∑t=Toτ+1Toτ+Tfτ((x^tτ−xtτ)2 _θJ(θ)=E_s_0 b_0(s_0),\,( s_0) _θ -1.0mu [ _t=T_o^τ+1^T_o^τ+T_f^τ -11.0mu (( x_t^τ-x_t^τ)^2 +(y^tτ−ytτ)2)×∇θ(∑tτ=Toτ+1Toτ+Tfτlogπθ(atτ^∣stτ^))], +( y_t^τ-y_t^τ)^2 )× _θ ( _t^τ=T_o^τ+1^T_o^τ+T_f^τ _θ ( a_t^τ s_t^τ ) ) ], (7) where logπθ(atτ^∣stτ^) _θ ( a_t^τ s_t^τ ) denotes a logarithm of conditional probability in trajectory time step tτt^τ when choosing an action given the current state. I-D SDM Formulations (BEP Extension) Scenes involving pedestrians inherently represent multi-agent scenarios where each pedestrian interacts with surrounding neighbours. The SDM (Single Decision Maker) approach simplifies the problem by focusing on individual pedestrian decision-making, treating each trajectory as an independent episode and keeping the interactions in the observed state. In STGAT, the initial encoded state of the whole scene is used, but subsequent predictions rely only on the hidden state of the D-LSTM, ignoring interactions with other agents. Therefore, the SDM formulation reduces to a single-agent problem, where pedestrian trajectories are treated independently, and all observed steps in the scene serve as an initial state for each pedestrian. I-D1 SL-MSE-SDM In order to utilize a step-wise decision-making process of the pedestrian, the action was defined as the single step of a pedestrian in the trajectory. The next state was defined to be a vector consisting of the previous state and current action. The number of steps in an episode for SDM is Tϵ=12T^ε=12 and the episode time step is 0≤tϵ<Tϵ0≤ t^ε<T^ε. To be mathematically sound, let action atϵ∈ℝ2a_t^ε ^2 such that atϵ=(x^Toτ+1+tϵ,y^Toτ+1+tϵ)a_t^ε= ( x_T_o^τ+1+t^ε, y_T_o^τ+1+t^ε ). Let the initial state s0∈ℝ2N0s_0 ^2N_0, where N0N_0 is a number of x-y pairs of observed trajectories of all pedestrians in the scene, each having 8 steps. Let stϵ∈ℝ2Ntϵs_t^ε ^2N_t^ε be state vector stϵ=(xs,0,ys,0,…,xs,Ntϵ,ys,Ntϵ)s_t^ε=(x_s,0,y_s,0,...,x_s,N_t^ε,y_s,N_t^ε) comprising of 2Ntϵ2N_t^ε components. The number of x-y pairs NtϵN_t^ε of state stϵs_t^ε at the time step of the episode tϵt^ε is therefore dependent on the number of pedestrians in the observed scene. The transition function can then be defined as mapping :ℝN×ℝ2→ℝN+2T:R^N×R^2 ^N+2 such that stϵ+1 s_t^ε+1 =(stϵ,atϵ)=(xs,0,ys,0,…CLOSE, =T(s_t^ε,a_t^ε)=(x_s,0,y_s,0,..., OPENxs,Ntϵ,ys,Ntϵ,x^Toτ+1+tϵ,y^Toτ+1+tϵ). x_s,N_t^ε,y_s,N_t^ε, x_T_o^τ+1+t^ε, y_T_o^τ+1+t^ε). (8) The termination state is reached when tϵ=Tϵ−1t^ε=T^ε-1. Taking into account the definition of the reward function, the resulting policy gradient becomes ∇θJ(θ)=s0∼b0(s0),(ε∣s0)∼πθ _θJ(θ)=E_s_0 b_0 (s_0 ), ( s_0 ) _θ [∇θ∑tτ=Toτ+1Toτ+Tfτγtτ−Toτ−1∑l′=Toτ+1tτ((xl′^−xl′)2+(yl′^−yl′)2)]. [ _θ _t^τ=T_o^τ+1^T_o^τ+T_f^τγ^t^τ-T_o^τ-1 -14.0mu _l =T_o^τ+1^t^τ -14.0mu (( x_l -x_l )^2+( y_l -y_l )^2 ) ]. (9) IV Experimental Settings IV-A Datasets ETH [6] and UCY [21] are standard benchmarking datasets for P also used in this work. The datasets include 2D coordinates and time steps of each step. Pedestrians are divided into scenes, which are sets of observed and next-step trajectories in a single location. Such scenes therefore contain only pedestrians who were present at the location at the same time and possibly interacted. The ETH dataset consists of two subsets, ETH and Hotel. Similarly, the UCY dataset consists of three subsets, Zara1, Zara2 and Univ. The results are reported as per individual subsets of the two datasets. The subsets are further divided into training, validation and testing sets. The validation set was generated using a leave-one-out strategy where four-fifths of the original training set of trajectories were employed as the train set while the remaining fifth of trajectories was used for evaluation. IV-B Evaluation and Metrics The Average Displacement Error (ADE) and Final Displacement Error (FDE) were used as the metrics most widely used in the field of P. The mean of output distribution STGAT was used as an action for ADE and FDE calculation to make SL-MSE and other formulations comparable. Additionally, the prediction was made 20 times, and the ADE and FDE were calculated for each pedestrian. The minimum out of 20 runs per pedestrian averaged over all pedestrians in the set were reported as mean minFDE and mean minADE. The results are reported as averages over 5 runs rounded to 2 decimals. Lower values indicate better prediction performance for all metrics in this paper. IV-C Training and Optimization In the first two pretraining phases, the STGAT model was trained five times with different initializations on each of the subsets of ETH and UCY. Similarly, every formulation and algorithm used for SL-MSE-SPG has been trained 5 times in the third learning phase. For each run of the training, a pre-trained model with unique initialization has been used. The models have been fine-tuned for each algorithm and formulation on the ETH training and validation set. In this paper, the optimal learning rate was kept identical over all modules of the STGAT in the third phase of learning. RL algorithms used included REINFORCE [20] and proximal policy optimization (PPO) [22]. IV-D Value function REINFORCE with baseline and PPO utilized a value function approximation, which relied on a network predicting expected returns based on state inputs. To handle the large input space of full pedestrian scenes, this paper introduces a network comprising of a shallow 3-layer dense neural network paired with the STGAT’s shared encoder, referred to as the full state baseline in the tables. This structure avoided extensive hyperparameter tuning and sped up processing by only updating the shallow layers and reusing the STGAT encoder. Additionally, a value network that takes simplified subsets of the states was tested. This value network used a simplified input state of only a single pedestrian observed steps but maintained the same neural architecture as the full state network. This network is referred to as a simplified baseline. V Results and Discussion TABLE I: Comparison of stochastic policy and deterministic policy results. SL-MSE SL-MSE-SPG Formulation ADE minADE FDE minFDE ADE minADE FDE minFDE ETH 0.95 0.89 1.84 1.69 0.95 0.87 1.85 1.69 Hotel 0.61 0.55 1.19 1.07 0.60 0.52 1.19 1.06 Zara1 0.43 0.39 0.95 0.85 0.45 0.40 0.97 0.85 Zara2 0.37 0.32 0.79 0.70 0.37 0.35 0.77 0.70 Univ 0.53 0.51 1.15 1.11 0.63 0.63 1.30 1.27 Average 0.5801 0.5334 1.1824 1.0843 0.5987 0.5598 1.2197 1.1144 Initially, this section examines both stochastic and deterministic policies in the context of ODM. The stochastic policy trained through REINFORCE, REINFORCE with baseline, PPO and PPO with and without baseline is evaluated. Subsequently, REINFORCE with baseline and PPO with baseline are compared while using different architectures of value function. Lastly, the SL-MSE-SDM is analyzed to assess SDM benefits on agent decision-making with STGAT. V-A Comparing Deterministic and Stochastic ODM Policy with REINFORCE As can be seen in the Table I, SL-MSE-SPG formulation has better or similar performance for the ETH, Hotel and Zara2. On the other hand, SL-MSE-SPG underperformed in the case of Zara1 and Univ subsets. This difference in performance can be attributed to the simplicity of the REINFORCE algorithm which was used in the case of SL-MSE-SPG. As the simplest Monte Carlo stochastic policy gradient method, REINFORCE conveys high variance in gradient steps that lead to slower convergence and fluctuations in performance on the validation set, which was encountered during training. During the training, the best-performing iteration on the validation set often did not correspond to the best-performing iteration on the test set which was caused by the high variance in the loss. This major drawback prevented the stochastic policy from reaching the same results as the deterministic policy on the fixed 150 epochs of the third phase of training resulting in all overall averages of SL-MSE-SPG higher. To assess the performance difference, further tuning of both algorithms on parameters such as batch size would be necessary. TABLE I: Comparison of different Algorithm results with stochastic and deterministic policies. SL-MSE SL-MSE-SPG REINFORCE with Full State Baseline PPO with Full State Baseline PPO without Baseline Dataset ADE minADE FDE minFDE ADE minADE FDE minFDE ADE minADE FDE minFDE ADE minADE FDE minFDE ADE minADE FDE minFDE ETH 0.95 0.89 1.84 1.69 0.95 0.87 1.85 1.69 0.93 0.86 1.82 1.67 0.91 0.84 1.79 1.64 0.94 0.86 1.85 1.69 Hotel 0.61 0.55 1.19 1.07 0.60 0.52 1.19 1.06 0.61 0.57 1.21 1.11 0.54 0.51 1.11 1.03 0.56 0.52 1.10 1.03 Zara1 0.43 0.39 0.95 0.85 0.45 0.40 0.97 0.85 0.42 0.38 0.93 0.81 0.42 0.38 0.92 0.80 0.43 0.39 0.94 0.82 Zara2 0.37 0.32 0.79 0.70 0.37 0.35 0.77 0.70 0.34 0.34 0.71 0.68 0.33 0.34 0.71 0.69 0.34 0.35 0.75 0.71 Univ 0.53 0.51 1.15 1.11 0.63 0.63 1.30 1.27 0.53 0.54 1.15 1.13 0.52 0.53 1.13 1.11 0.56 0.56 1.19 1.17 Average 0.5801 0.5334 1.1824 1.0843 0.5987 0.5598 1.2197 1.1144 0.563 0.535 1.168 1.084 0.5460 0.5227 1.1335 1.0551 0.5644 0.5370 1.1675 1.0844 TABLE I: Comparison of different value network implementations. REINFORCE with Simplified Baseline REINFORCE with Full State Baseline PPO with Simplified Baseline PPO with Full State Baseline Dataset ADE minADE FDE minFDE ADE minADE FDE minFDE ADE minADE FDE minFDE ADE minADE FDE minFDE ETH 0.92 0.85 1.81 1.66 0.93 0.86 1.82 1.67 0.92 0.84 1.79 1.63 0.91 0.84 1.79 1.64 Hotel 0.67 0.63 1.35 1.26 0.61 0.57 1.21 1.11 0.53 0.50 1.10 1.03 0.54 0.51 1.11 1.03 Zara1 0.43 0.38 0.92 0.80 0.42 0.38 0.93 0.81 0.42 0.38 0.92 0.80 0.42 0.38 0.92 0.80 Zara2 0.39 0.39 0.74 0.69 0.34 0.34 0.71 0.68 0.33 0.34 0.72 0.69 0.33 0.34 0.71 0.69 Univ 0.73 0.72 1.47 1.44 0.53 0.54 1.15 1.13 0.52 0.53 1.12 1.10 0.52 0.53 1.13 1.11 Average 0.6282 0.5965 1.2603 1.1718 0.563 0.535 1.168 1.084 0.5434 0.5208 1.1304 1.0534 0.5460 0.5227 1.1335 1.0551 V-B Comparing REINFORCE and PPO variations The hyperparameter tuning of parameters specific to each algorithm resulted in improvement on all datasets for PPO with full state baseline. This can be seen in Table I. PPO with a full state baseline was the most advanced algorithm used in this paper. The highest performance gain for ADE was seen for ETH with 4% decrease, and for Hotel with over 11% decrease. A similar decrease was shown for FDE which ranged between 2% and 10% for all datasets. The second algorithm that resulted in the most prominent improvement in the metrics was the REINFORCE with a full state baseline. This training algorithm underperformed on several metrics compared to SL-MSE for Hotel and Univ while reaching the same or better results as PPO with a shared encoder baseline on 4 metrics for Zara1 and Zara2. This can be seen in Table I. The reason why this algorithm underperformed on Univ and Hotel while improving performance on ETH compared to SL-MSE can be attributed to the fact that the model was fine-tuned on ETH. Further improvement on the Univ and Hotel could be achieved by fine-tuning on these data subsets separately. Considering that mere use of the stochastic policy did not yield any overall improvement in Table I, while PPO and REINFORCE with baseline did, this indicates the need for advanced RL algorithm to achieve improvement with regards to SL-MSE. V-C Value Function Variations In parallel with the simulations with a value network with a shared STGAT encoder, a network taking simplified state input has been tested. The ablation study on the choice of the value network can be seen in Table I. Comparing the two REINFORCE implementations, the use of the full state through the value network with shared STGAT encoder resulted in an improvement of up to 27% ADE of Univ and between 8% to 13% in the case of metrics of Zara2 and Hotel. There has been a 1% decline in the performance of metrics on ETH when using REINFORCE with full state baseline. The full state baseline therefore improved results compared to the simplified baseline for REINFORCE on average. A higher number of runs of the model training would be needed to justify the better performance of REINFORCE with a simplified baseline for FDE and minFDE metrics of Zara1. In contrast, PPO with value network with the shared encoder did not result in major improvement or loss of performance for any of the metrics. The two value network approaches can therefore be considered equivalently well suited for the purpose of improving the model performance through the PPO in the unified framework. It needs to be noted that PPO with a simplified baseline takes subsets of states as input as has been explained in the Section IV-D. Furthermore, when comparing Table I and Table I it becomes evident that PPO without baseline had worse performance compared to the shared encoder baseline and PPO with full state baseline. This indicates that using a separate network for modelling value function is a key towards improved performance. V-D SDM Ablation Study (BEP extension) The results of SDM for deterministic policy in Table IV showcase a decrease in metrics for all datasets compared to SL-MSE. Particularly for the λ=0.4λ=0.4, which the model was fine-tuned on, the best results were reached with decreasing between 2% and 13% and improvement on the majority of the rest of the metrics. When comparing the Table I, the results for λ=0.4λ=0.4 outperform the PPO with a baseline on ETH while PPO performed better on the part of metrics of Zara2 and Univ in the range of 1%-2%. Such improvement can be attributed to the choice of the SDM state-action formulation introduced in this work. Because of the cumulative nature of the state transitions, the loss function in equation (I-D1) results in gradients that put more focus on learning the initial steps of the predicted trajectory. This indicates that the initial steps were decisive factors for good performance and focusing on the initial decision process of the pedestrian immediately after the observed 8 steps is key for model performance. This is further supported by the improvement of FDE considering that the final pedestrian location is highly affected by the initial decisions of the agent inference. Overall, by choosing an appropriate combination of the discount factor and the state-action definitions, an improvement can be achieved compared to the simple supervised method. TABLE IV: Comprehensive comparison of different algorithmic results with stochastic and deterministic policies. SL-MSE-SDM SL-MSE-ODM λ=0.2λ=0.2 λ=0.4λ=0.4 λ=0.6λ=0.6 λ=0.8λ=0.8 Dataset ADE minADE FDE minFDE ADE minADE FDE minFDE ADE minADE FDE minFDE ADE minADE FDE minFDE ADE minADE FDE minFDE ETH 0.95 0.89 1.84 1.69 0.91 0.84 1.9 1.75 0.90 0.85 1.88 1.74 0.91 0.86 1.87 1.73 0.93 0.87 1.87 1.73 Hotel 0.61 0.55 1.19 1.07 0.52 0.48 1.12 1.02 0.53 0.49 1.09 1.01 0.53 0.49 1.1 1.03 0.55 0.51 1.12 1.03 Zara1 0.43 0.39 0.95 0.85 0.45 0.41 1.05 0.93 0.42 0.39 0.95 0.87 0.43 0.39 0.95 87 0.43 0.39 0.96 0.88 Zara2 0.37 0.32 0.79 0.70 0.35 0.32 0.81 0.71 0.34 0.31 0.76 0.69 0.34 0.31 0.76 0.70 0.35 0.32 0.77 0.70 Univ 0.53 0.51 1.15 1.11 0.52 0.50 1.14 1.09 0.52 0.50 1.13 1.09 0.52 0.51 1.13 1.10 0.52 0.51 1.14 1.10 Average 0.5801 0.5334 1.1824 1.0843 0.5511 0.5099 1.2030 1.1010 0.5439 0.5079 1.1646 1.0822 0.5476 0.5126 1.1650 1.0875 0.5545 0.5178 1.1704 1.0921 VI Conclusion In this work, STGAT was implemented in a unified P framework by designing new state action definitions, novel supervised SDM formulation and architectures for the value network. Algorithms including PPO, PPO without a baseline, REINFORCE and REINFORCE with a baseline were studied. The use of ODM stochastic policy formulation with PPO with baseline and the designed supervised SDM formulation offer a substantial improvement in the performance of STGAT on all datasets. This result indicates that incorporating more sophisticated RL algorithms or learning task formulations that account for the stochastic nature of human movement offer an advantage in the learning process. This study therefore demonstrates that the unified framework can effectively enhance the performance of complex deep learning P architecture. Further research on STGAT could investigate if additional improvement can be achieved through stochastic STGAT policy combined with SDM or by utilizing IRL and GAN formulations with a parameterized discriminator network that were not studied in this work. The key area of future work would be to apply the designed formulations and unified framework on a current SOTA architecture for P. Overall, the study shows a strong potential to improve a majority of the SOTA models in the P field through the unified P framework and advanced RL algorithms. References [1] Y. Huang, H. Bi, Z. Li, T. Mao, and Z. Wang, “STGAT: Modeling spatial-temporal interactions for human trajectory prediction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019, p. 6272–6281. [2] D. J. Fagnant and K. Kockelman, “Preparing a nation for autonomous vehicles: Opportunities, barriers and policy recommendations,” Transportation Research Part A: Policy and Practice, vol. 77, p. 167–181, 2015. [3] N. Sharma, C. Dhiman, and S. Indu, “Pedestrian intention prediction for autonomous vehicles: A comprehensive survey,” Neurocomputing, vol. 508, p. 120–152, 2022. [4] J. L. A. Lemmens, A. Bighashdel, P. Jancura, and G. Dubbelman, “Unified pedestrian path prediction framework: A comparison study,” in 2023 IEEE Intelligent Vehicles Symposium (IV), 2023, p. 1–8. [5] D. Helbing and P. Molnár, “Social force model for pedestrian dynamics,” Physical Review E, vol. 51, p. 4282–4286, 1995. [6] S. Pellegrini, A. Ess, K. Schindler, and L. Van Gool, “You’l never walk alone: Modeling social behavior for multi-target tracking,” in 2009 IEEE 12th International Conference on Computer Vision, 2009, p. 261–268. [7] Y. Yao, M. Xu, C. Choi, D. J. Crandall, E. M. Atkins, and B. Dariush, “Egocentric vision-based future vehicle localization for intelligent driving assistance systems,” in 2019 International Conference on Robotics and Automation (ICRA), 2019, p. 9711–9717. [8] A. Rasouli, I. Kotseruba, T. Kunic, and J. K. Tsotsos, “PIE: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, p. 6261–6270. [9] X. Du, R. Vasudevan, and M. Johnson-Roberson, “Bio-LSTM: A biomechanically inspired recurrent neural network for 3-d pedestrian pose and gait prediction,” IEEE Robotics and Automation Letters, vol. 4, no. 2, p. 1501–1508, 2019. [10] J. Liang, L. Jiang, J. C. Niebles, A. G. Hauptmann, and L. Fei-Fei, “Peeking into the future: Predicting future person activities and locations in videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. [11] Y. Yuan, X. Weng, Y. Ou, and K. M. Kitani, “AgentFormer: Agent-aware transformers for socio-temporal multi-agent forecasting,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021, p. 9793–9803. [12] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese, “Social LSTM: Human trajectory prediction in crowded spaces,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, p. 961–971. [13] A. Vemula, K. Muelling, and J. Oh, “Social attention: Modeling attention in human crowds,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, p. 4601–4607. [14] B. Ivanovic and M. Pavone, “The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs,” CoRR, vol. abs/1810.05993, 2018. [Online]. Available: https://arxiv.org/abs/1810.05993 [15] P. R. G. Cadena, Y. Qian, C. Wang, and M. Yang, “Pedestrian graph+: A fast pedestrian crossing prediction model based on graph convolutional networks,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 11, p. 21 050–21 061, 2022. [16] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in International Conference on Learning Representations (ICLR), 2018. [17] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi, “Social GAN: Socially acceptable trajectories with generative adversarial networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, p. 2255–2264. [18] J. Ho and S. Ermon, “Generative adversarial imitation learning,” in Advances in Neural Information Processing Systems, vol. 29, 2016. [19] H. Zou, H. Su, S. Song, and J. Zhu, “Understanding human behaviors in crowds by imitating the decision-making process,” CoRR, vol. abs/1801.08391, 2018. [Online]. Available: https://arxiv.org/abs/1801.08391 [20] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in Advances in Neural Information Processing Systems, vol. 12, 1999, p. 1057–1063. [21] A. Lerner, Y. Chrysanthou, and D. Lischinski, “Crowds by example,” Computer Graphics Forum, vol. 26, no. 3, p. 655–664, 2007. [22] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” CoRR, vol. abs/1707.06347, 2017. [Online]. Available: https://arxiv.org/abs/1707.06347 VII Appendices VII-A Evaluation Metrics In the context of the formulations of this paper, the ADE is an average squared geometric distance between the forecasted and the ground truth trajectories, which can be written as ADE=s0∼b0(s)[1Tfτ∑tτ=Toτ+1Toτ+Tfτ((x^tτ−xtτ)2+(y^tτ−ytτ)2)]. =E_s_0 b_0 (s ) [ 1T_f^τ -4.0mu _t^τ=T_o^τ+1^T_o^τ+T_f^τ -13.0mu (( x_t^τ-x_t^τ)^2+( y_t^τ-y_t^τ)^2 ) ]. (10) The FDE considers only the final step of each pedestrian and it simplifies to FDE=s0∼b0(s)[(x^Toτ+Tfτ−xToτ+Tfτ)2 =E_s_0 b_0 (s ) [( x_T_o^τ+T_f^τ-x_T_o^τ+T_f^τ)^2 +(y^Toτ+Tfτ−yToτ+Tfτ)2]. +( y_T_o^τ+T_f^τ-y_T_o^τ+T_f^τ)^2 ]. (11) The coordinates are indexed by the trajectory time tτt^τ. VII-B Hyperparameters Choice for Each Formulation The following table shows the chosen parameters, where the policy learning rate and the value function learning rate were the only parameters that had a noticeable impact on the performance and were used in the hyperparameter space searching. The PPO clipping factor was fixed at 0.20.2. The value function iterations for PPO were kept at 11. The learning rate is denoted as LR. TABLE V: Standard Learning Formulations Formulation Policy Learning Rate SL-MSE 7.58×10−57.58× 10^-5 SL-MSE-SPG 8.62×10−58.62× 10^-5 SL-MSE-SDM 1.04×10−41.04× 10^-4 TABLE VI: Algorithms with value function Formulation Policy LR Value Function LR REINFORCE with Simplified Baseline 9.5×10−59.5× 10^-5 3.0×10−43.0× 10^-4 REINFORCE with Full State Baseline 2.32×10−42.32× 10^-4 3.0×10−43.0× 10^-4 TABLE VII: PPO-based algorithms Formulation Policy LR Value Function LR PPO Iterations PPO - Simplified Baseline 2.9×10−52.9× 10^-5 1.0×10−31.0× 10^-3 7 PPO - Full State Baseline 2.8×10−52.8× 10^-5 1.0×10−31.0× 10^-3 7 PPO without Baseline 7.8×10−57.8× 10^-5 - 6 VII-C SL-MSE-SDM Objective Function Derivation Following the definition of the objective function, for SL-MSE-SDM, the time horizon Tϵ=12T^ε=12 and the discount factor is now nonzero. The objective function can be calculated as ∇θJ(θ) _θJ(θ) =s0∼b0(s)∇θV(s0,θ)= =E_s_0 b_0 (s ) _θV (s_0;θ )= =s0∼b0(s)(ε∣st=s0)∼πθ[∇θG(ε∣stϵ=s0)] =E_s_0 b_0 (s )E_ ( s_t=s_0 ) _θ [ _θG ( s_t^ε=s_0 ) ] =s0∼b0(s),(ε∣s0)∼πθ[∇θ∑tϵ=0Tε−1γtϵrtϵ] =E_s_0 b_0 (s ), ( s_0 ) _θ [ _θ _t^ε=0^T -1γ^t^εr_t^ε ] =s0∼b0(s),(ε∣s0)∼πθ[∇θ∑tϵ=0Tε−1γtϵ‖(s^tϵ,a^tϵ)−(stϵ,atϵ)‖22] =E_s_0 b_0 (s ), ( s_0 ) _θ -6.0mu [ _θ -6.0mu _t^ε=0^T -1 -10.0muγ^t^ε -8.0mu \| ( s_t^ε, a_t^ε ) -4.0mu- -4.0mu (s_t^ε,a_t^ε ) \|_2^2 ] =s0∼b0(s),(ε∣s0)∼πθ[∇θ∑tτ=Toτ+1Toτ+Tfτγk−Toτ−1∑l′=Toτ+1tτ =E_s_0 b_0 (s ), ( s_0 ) _θ [ _θ _t^τ=T_o^τ+1^T_o^τ+T_f^τγ^k-T_o^τ-1 _l =T_o^τ+1^t^τ ((xl′^−xl′)2+(yl′^−yl′)2)]. (( x_l -x_l )^2+( y_l -y_l )^2 ) ]. (12) VII-D SL-MSE-SPG Loss Function Derivation The policy gradient theorem [20] uses so called parametrization and score function tricks to achieve the expression that uses logarithmic probabilities. Following the policy gradient theorem, Tϵ=1T^ε=1 and λ=0λ=0 the derivation follows as ∇θJ(θ) _θJ(θ) =s0∼b0(s)∇θV(s0,θ) =E_s_0 b_0 (s ) _θV (s_0;θ ) =s0∼b0(s),(ε∣s0)∼πθ[∇θ∑tϵ=0Tε−1γtϵrt] =E_s_0 b_0 (s ), ( s_0 ) _θ [ _θ _t^ε=0^T -1γ^t^εr_t ] =s0∼b0(s),(ε∣s0)∼πθ[∇θ∑tϵ=0Tε−1γtϵ‖(s^tϵ,a^tϵ)−(stϵ,atϵ)‖22] =E_s_0 b_0 (s ), ( s_0 ) _θ -6.0mu [ _θ -6.0mu _t^ε=0^T -1 -10.0muγ^t^ε -8.0mu \| ( s_t^ε, a_t^ε ) -4.0mu- -4.0mu (s_t^ε,a_t^ε ) \|_2^2 ] =s0∼b0(s),(ε∣s0)∼πθ[∑tτ=Toτ+1Toτ+Tfτ((x^tτ−xtτ)2+ =E_s_0 b_0(s),\,( s_0) _θ [ _t^τ=T_o^τ+1^T_o^τ+T_f^τ -13.0mu (( x_t^τ-x_t^τ)^2+ (y^tτ−ytτ)2)×∇θ(∑tτ=Toτ+1Toτ+Tfτlogπθ(atτ^∣stτ^))]. ( y_t^τ-y_t^τ)^2 )× _θ ( _t^τ=T_o^τ+1^T_o^τ+T_f^τ _θ ( a_t^τ s_t^τ ) ) ]. (13)