Paper deep dive
Comfort by Construction: Adaptive, Comfort-Bounded Action Spaces for Learned Driving Policies
Anna Rothenhäusler, Daniel Jost, Raghu Rajan, Faris Janjos, Oliver Scheel, Andreas Look, Joschka Boedecker
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/20/2026, 2:43:32 AM
Summary
This paper addresses the issue of uncomfortable driving behaviors in reinforcement learning policies within data-driven simulators like PufferDrive. Standard simulators use fixed action grids that allow abrupt maneuvers violating human comfort bounds (acceleration and jerk). The authors propose an adaptive action parameterization that re-discretizes the action grid at every step to span exactly the per-step feasible control set, derived from closed-form inversion of lateral-jerk constraints. This prevents 'grid collapse' where fine-grained control is lost at high speeds. They also introduce PufferDrive-Editor, a tool for auditing kinematics. Experiments on the Waymo Open Motion Dataset and a slalom course show the adaptive model maintains comfort violations below 1% while outperforming baselines in navigability.
Entities (10)
Relation Signals (8)
PufferDrive-Editor → isusedfor → Auditing Kinematics
confidence 96% · PufferDrive-Editor, a browser-based tool to audit realized kinematics and author kinematically challenging scenes.
Adaptive Action Parameterization → solves → Grid Collapse
confidence 95% · We propose an adaptive action parameterization that re-discretizes the grid at every step... to span exactly the per-step feasible control set... preventing grid collapse.
Adaptive Action Parameterization → enforces → OPM (Occupant's Preference Metric)
confidence 94% · We adopt the Occupant’s Preference Metric (OPM)... we use the comfort envelope to constrain the policy’s action space directly
Adaptive Action Parameterization → uses → Kinematic Bicycle Model
confidence 93% · The vehicle model that turns actions into motion... kinematic bicycle model... We parametrize the model at the rear axle.
Classic → suffersfrom → Grid Collapse
confidence 92% · clamping a static grid saturates it and destroys fine-grained control ('grid collapse')... Classic commands an acceleration and a steering command directly from a fixed 7x13 grid
Adaptive Action Parameterization → outperforms → Jerk
confidence 91% · our adaptive model holds comfort violations below 1% while outperforming clipped-grid and direct-jerk baselines in navigability.
Adaptive Action Parameterization → outperforms → Clipped
confidence 91% · our adaptive model holds comfort violations below 1% while outperforming clipped-grid and direct-jerk baselines in navigability.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Data-driven driving simulators command accelerations and steering rates from a fixed grid without constraining the realized accelerations and jerks. As a result, reinforcement-learning policies inflate safety metrics through abrupt, last-second maneuvers that lie far outside the range of human driving and would be unacceptable to occupants of a real vehicle, so the metrics measure simulator permissiveness rather than policy quality. Enforcing comfort bounds naively is not enough: lateral limits shrink quadratically with speed, so clamping a static grid saturates it and destroys fine-grained control ("grid collapse"). We propose an adaptive action parameterization that rediscretizes the grid at every step to span exactly the per-step feasible control set, via closed-form inversion of the lateral-jerk constraint. We further present PufferDrive-Editor, a browser-based tool to audit realized kinematics and author kinematically challenging scenes. On the Waymo Open Motion Dataset and a hand-authored slalom, our adaptive model holds comfort violations below 1% while outperforming clipped-grid and direct-jerk baselines in navigability.
Tags
Links
- Source: https://arxiv.org/abs/2609.13011v1
- Canonical: https://arxiv.org/abs/2609.13011v1
Trouble viewing inline? Open PDF directly →
Full Text
48,931 characters extracted from source content.
Expand or collapse full text
Comfort by Construction: Adaptive, Comfort-Bounded Action Spaces for Learned Driving Policies Anna Rothenhäusler †thanks: These authors contributed equally to this work. Affiliation: University of Freiburg, Freiburg, Germany E-mail rothenha, jostd, rajanr, jboedeck@cs.uni-freiburg.de Daniel Jost⋆ Affiliation: University of Freiburg, Freiburg, Germany E-mail rothenha, jostd, rajanr, jboedeck@cs.uni-freiburg.de Raghu Rajan Affiliation: University of Freiburg, Freiburg, Germany E-mail rothenha, jostd, rajanr, jboedeck@cs.uni-freiburg.de Faris Janjos Affiliation: Bosch Center for Artificial Intelligence, Stuttgart, Germany E-mail faris.janjos, oliver.scheel@de.bosch.com Oliver Scheel Affiliation: Bosch Center for Artificial Intelligence, Stuttgart, Germany E-mail faris.janjos, oliver.scheel@de.bosch.com Andreas Look Affiliation: Coburg University of Applied Sciences, Coburg, Germany E-mail andreas.look@hs-coburg.de Joschka Boedecker Affiliation: University of Freiburg, Freiburg, Germany E-mail rothenha, jostd, rajanr, jboedeck@cs.uni-freiburg.de Abstract Data-driven driving simulators command accelerations and steering rates from a fixed grid without constraining the realized accelerations and jerks. As a result, reinforcement-learning policies inflate safety metrics through abrupt, last-second maneuvers that lie far outside the range of human driving and would be unacceptable to occupants of a real vehicle, so the metrics measure simulator permissiveness rather than policy quality. Enforcing comfort bounds naively is not enough: lateral limits shrink quadratically with speed, so clamping a static grid saturates it and destroys fine-grained control (“grid collapse”). We propose an adaptive action parameterization that re-discretizes the grid at every step to span exactly the per-step feasible control set, via closed-form inversion of the lateral-jerk constraint. We further present PufferDrive-Editor, a browser-based tool to audit realized kinematics and author kinematically challenging scenes. On the Waymo Open Motion Dataset and a hand-authored slalom, our adaptive model holds comfort violations below 1%1\% while outperforming clipped-grid and direct-jerk baselines in navigability. 1 Introduction Learned driving policies are increasingly trained in data-driven simulators like Nocturne [24], Waymax [12], or GPUDrive [13]. These simulators are built for closed-loop reinforcement learning (RL): RL trains on policy-induced states, enabling learning from rare safety-critical situations. While achieving high goal completion, their behavior is shaped by the kinematic bicycle model [14, 18] translating actions into motion. Standard interfaces use a fixed grid of accelerations and steering rates that does not constrain the realized accelerations and jerks. Consequently, RL policies learn to avoid collisions through abrupt, last-second maneuvers (sharp braking combined with hard steering) whose realized accelerations and, especially, jerks fall well outside the comfort envelope of even aggressive human driving [3, 11]. Safety metrics thus measure the permissiveness of the simulator’s kinematics rather than policy quality, crediting evasions that no comfort-respecting vehicle would reproduce. This work studies the problem in PufferDrive, whose actuation interface is representative of the simulators above; the analysis and the proposed parameterization carry over to any simulator built on the same kinematic bicycle model. Naively enforcing comfort bounds is difficult because the feasible action set depends nonlinearly on the vehicle state. In particular, the admissible steering range shrinks rapidly with speed, alongside additional nonlinearities from the vehicle kinematics [14, 3]. A static grid tuned for low speed then falls largely outside this range at higher speeds, and clamping collapses many grid points onto the constraint boundary, degrading fine-grained control (“grid collapse”). Because widely used data-driven simulators expose discrete action grids [12, 13], we focus on maintaining a feasible discrete grid without collapse. To inspect these failure modes, we present PufferDrive-Editor, a browser-based tool for visualizing kinematic violations and authoring challenging scenarios. It lets us trace where policies breach the comfort envelope: qualitatively, most often just before sharp turns and narrow gaps. To keep control feasible in exactly these situations without collapsing the grid, we propose an adaptive action parameterization that computes the exact feasible control set at each step via closed-form inversion of the lateral-jerk constraint, re-discretizing the grid over it. Our reference points are the established kinematics models used in PufferDrive: Classic, which commands acceleration and steering from a fixed grid and enforces no comfort limits at all, and Jerk, which commands jerk and integrates it into bounded accelerations. We take the steering rate as the default steering command, as every constrained model below uses it, and report the original steering-angle parameterization as Classic (steer angle) alongside it. Against them we put three improved variants: Jerk equipped with realistic comfort bounds, and two versions of Classic that resolve their commands against the per-step feasible set: one by clipping the fixed grid into it (Clipped), one by re-discretizing the grid over it (Adaptive). Our contributions are as follows: 1. We show that unconstrained action spaces let learned policies inflate safety metrics through maneuvers that leave the comfort envelope of even aggressive human driving, and that naive clamping leads to grid collapse. 2. We propose a constraint-aware action parameterization that adaptively re-discretizes the per-step feasible control set. 3. We present PufferDrive-Editor, a browser-based tool for auditing realized kinematics and authoring targeted scenarios, providing a general debugging tool for exposing and analyzing failure cases in PufferDrive. 2 Problem Description We first describe the vehicle model that turns actions into motion (Sec. 2.1), then the behavioral comfort envelope that a realized trajectory should respect (Sec. 2.2), before reviewing how prior work enforces such constraints (Sec. 2.3) and showing why the two action spaces PufferDrive contains fail to do so (Sec. 2.4). 2.1 Kinematic bicycle model Vehicle motion is commonly abstracted by the kinematic bicycle model [19, 14, 18]: the two wheels of each axle are collapsed into a single wheel on the longitudinal axis, tire slip is neglected, and the vehicle is described by its pose, speed, and front-wheel steering angle (Fig. 1). Despite its simplicity it tracks real vehicle trajectories closely in the moderate lateral-acceleration regime of everyday driving [18], which makes it the standard choice for planning and for driving simulators. We parametrize the model at the rear axle. This is the reference point at which the tire-slip-free kinematic bicycle is exact: the rear wheel cannot slip sideways, so the rear axle’s velocity always points along the vehicle heading and its speed is simply the state speed v — no slip angle appears. Choosing it as the reference point makes the comfort quantities of Sec. 2.2 identical across every dynamics model we compare, including those that never model slip at all. xxyyψt _tδt _tvtv_t(xt,yt)(x_t,y_t)L Figure 1: Rear-axle kinematic bicycle model. Each axle is collapsed into a single wheel, separated by the wheelbase L. The state is the rear-axle position (xt,yt)(x_t,y_t) (orange), heading ψt _t, signed speed vtv_t, and steering angle δt _t. The rear wheel does not slip, so the velocity (orange) points along the heading ψt _t and vtv_t is the rear axle’s own speed. Comfort is judged at this point. All controlled agents follow this model with state (xt,yt,ψt,vt,δt)(x_t,y_t, _t,v_t, _t) as visualized in Fig. 1. Given a commanded acceleration ata_t and a steering rate δ˙t δ_t, one step of length Δt t reads vt+1 v_t+1 =clip(vt+atΔt,±vmax), =clip\! (v_t+a_t t,\ ± v_ ), δt+1 _t+1 =clip(δt+δ˙tΔt,±δmax), =clip\! ( _t+ δ_t t,\ ± _ ), ωt+1 _t+1 =vttanδt+1L, = v_t _t+1L, ψt+1 _t+1 =ψt+ωt+1Δt, = _t+ _t+1 t, (1) with yaw rate ω and wheelbase L. Comfort is judged on the realized, not the commanded, longitudinal and lateral accelerations, and both are read at the rear axle. The longitudinal acceleration is the change in rear-axle speed, and the lateral (centripetal) acceleration is that speed times the rear-axle yaw rate: at+1lon=vt+1−vtΔt,at+1lat=vt+1ωt+1,ωt+1=vttanδt+1L,a^lon_t+1= v_t+1-v_t t, a^lat_t+1=v_t+1\, _t+1, _t+1= v_t _t+1L, (2) with the realized jerks the backward differences jt+1lon=(at+1lon−atlon)/Δtj^lon_t+1=(a^lon_t+1-a^lon_t)/ t and jt+1lat=(at+1lat−atlat)/Δtj^lat_t+1=(a^lat_t+1-a^lat_t)/ t. 2.2 Comfort bounds on vehicle kinematics alata^latalona^lon0.930.93−0.93-0.93−4.0-4.04.04.0−5.6-5.65.65.60.930.93−0.93-0.931.471.47−2.0-2.03.073.07−5.08-5.08Acceleration (m/s2m/s^2)jlatj^latjlonj^lon0.60.6−0.6-0.60.90.9−0.9-0.92.02.0−2.0-2.00.60.6−0.6-0.60.90.9−0.9-0.92.02.0−2.0-2.0Jerk (m/s3m/s^3)public transportnormalaggressive Figure 2: Comfort limits of the OPM driver profiles of [3], marked on each axis; each profile lies strictly inside the next. We use normal and aggressive in our experiments and show public transport for reference. (Left) Acceleration plane (alat,alon)(a^lat,a^lon) in m/s2m/s^2; the longitudinal limits of the driving profiles are asymmetric, since braking tolerance exceeds throttle tolerance. (Right) Jerk plane (jlat,jlon)(j^lat,j^lon) in m/s3m/s^3, with symmetric limits. Occupants perceive vehicle motion through acceleration and its time derivative, jerk; abrupt or sustained accelerations can cause discomfort and motion sickness [8, 9]. We adopt the Occupant’s Preference Metric (OPM) of Bae et al. [3], which defines a behavioral comfort envelope via five parameters: OPM=amaxlon,aminlon,amaxlat,jmaxlon,jmaxlat,OPM\;=\; \\,a^lon_ ,\;a^lon_ ,\;a^lat_ ,\;j^lon_ ,\;j^lat_ \, \, (3) where amaxlona^lon_ and aminlona^lon_ denote the throttle and braking thresholds, respectively, while the lateral acceleration and jerk limits are symmetric about zero. Accordingly, the five OPM parameters define lower and upper bounds on the four realized quantities of Sec. 2.1: a∙∈[a¯∙,a¯∙],j∙∈[j¯∙,j¯∙],∙∈lon,lat,a ∈[ a , a ], j ∈[ j , j ], ∈\lon,lat\, with a¯lon=aminlon a^lon=a^lon_ , a¯lon=amaxlon a^lon=a^lon_ , a¯lat=−amaxlat a^lat=-a^lat_ , a¯lat=amaxlat a^lat=a^lat_ , and j¯∙=−jmax∙ j =-j _ , j¯∙=jmax∙ j =j _ . Figure 2 illustrates these limits. Bae et al. anchor the tightest profile in public-transport data, where freestanding passengers lose posture [16]; we show it for reference but drive with the two automotive profiles. The normal profile uses seated-occupant thresholds, and the aggressive profile uses observed human driving extremes; naturalistic driving studies likewise treat sustained high jerk not as a physical impossibility but as the signature of an aggressive driver [11]. These are behavioral envelopes, sitting well inside the physical handling limits: leaving the aggressive band is not impossible, only something essentially no human driver would choose and no occupant would accept. We therefore call a control feasible relative to the enforced profile rather than in an absolute physical sense. In the original OPM system, bounds are soft constraints enforced at the path-planning level [3]. Data-driven traffic simulators also evaluate these comfort quantities post hoc using metrics or Wasserstein distance penalties [26, 17]. In contrast, we use the comfort envelope to constrain the policy’s action space directly: at each step, we analytically compute the feasible set of controls whose realized body-frame quantities stay inside the envelope (Figure 2) and re-discretize the action grid over this set (Section 3.1), guaranteeing comfort by construction. 2.3 Kinematic constraints in simulation and reinforcement learning Kinematic models in driving simulators. Data-driven simulators such as Nocturne [24], Waymax [12], and GPUDrive [13] share a common actuation interface: a kinematic bicycle model [14, 18] driven from a fixed grid of controls that is clamped only to static input ranges. Nothing bounds the realized accelerations or jerks, the gap our work targets in Section 3. Feasibility in learned models. Prior works build kinematic constraints into learned motion models, such as routing trajectories through a bicycle-model layer (e.g., Deep Kinematic Models [5], Trajectron++ [20]) or propagating uncertainty through stochastic kinematic layers [27]. While these methods make predicted trajectories dynamically consistent by construction, they act downstream of action selection: a policy remains free to command an out-of-envelope maneuver that the layer then realizes rather than forbids. We instead constrain selection itself, bounding the feasible action set of a closed-loop policy at every simulation step under a behavioral comfort envelope. We expect this to matter most for the adaptive variant: by re-discretizing a full-resolution grid over the feasible set, it lets the policy optimize entirely within the envelope, so the constraint is reflected in what the policy learns rather than corrected after the fact. Constrained reinforcement learning. The simplest approach leaves the action space untouched and penalizes violations in the reward, as Gigaflow [6] does for harsh acceleration and jerk. Comfort then competes with progress through a tunable weight: violations get rarer but are never ruled out. Hard enforcement is typically done via external layers that mask or project actions [7, 4, 15, 25]. However, when the feasible set shrinks (e.g., quadratically with speed), these methods suffer from grid collapse, where distinct discrete actions alias to the same boundary values. While action mapping [23] and jerk-bounded generators [1] address constrained action selection, our approach analytically computes a per-step feasible action region and adaptively re-discretizes the action grid onto it, preserving action resolution across varying vehicle speeds while guaranteeing feasible commands. 2.4 Default models and their failure modes PufferDrive provides two kinematics models by default. Classic commands an acceleration and a steering command directly from a fixed 7×137× 13 grid, applied unchanged. Nothing ties consecutive commands together: within one step of Δt=0.1s t=0.1\,s the policy may switch from full braking to full throttle or slew the steering across its whole range, producing jerks orders of magnitude beyond any human comfort envelope. As a result, Classic learns effective but kinematically implausible driving. The steering command comes in two parameterizations: the steering angle δt _t, as originally implemented, or the steering rate δ˙t δ_t, which integrates to the angle and therefore low-pass filters the steering without bounding anything. Every constrained model in this work commands the rate (Sec. 6), so we treat it as the default parameterization: Classic denotes the steer-rate model throughout, and the original is written Classic (steer angle). Jerk sits at the opposite end: the policy commands longitudinal and lateral jerk, which the environment integrates into accelerations clamped to the comfort band, so the limits hold by rule. It is not a separate dynamics model: the same kinematic bicycle model of Sec. 2.1 runs underneath, only re-parameterized one derivative up. The longitudinal jerk integrates to the speed, while the lateral jerk integrates to a lateral acceleration that is mapped back to a steering angle through the bicycle curvature relation δ=arctan(κL)δ= (κ L) with κ=alat/v2κ=a^lat/v^2. The disadvantage is that the action now operates on the second derivative of the velocity: credit assignment must propagate through j→a→v→xj→ a→ v→ x, and the same jerk action produces different motion depending on the accumulated acceleration state. A single action barely changes the trajectory, and exploration in jerk space is much harder. 3 Method We therefore introduce an action space in between: the policy commands an acceleration and a steering rate (at,δ˙t)(a_t, δ_t) — one derivative closer to the trajectory than jerk control — and the comfort limits are enforced inside the environment step by restricting each command to the set feasible from the current state (Sec. 3.1). Two variants of this restriction are compared (Sec. 3.2): clipping, which keeps Classic’s absolute action semantics, and adaptive re-discretization, which re-maps the action grid onto the feasible set at each step so that actions never collapse onto the limit values. 3.1 Per-step feasible set Combining the jerk and absolute limits yields per-axis target bands tlonA^lon_t and tlatA^lat_t. The longitudinal band directly bounds the commanded acceleration ata_t via vt+1=vt+atΔtv_t+1=v_t+a_t t. In contrast, the lateral band couples both control inputs: at+1lat(at,δ˙t)=vt+1(at)ωt+1(δ˙t),a^lat_t+1(a_t, δ_t)=v_t+1(a_t)\, _t+1( δ_t), where ωt+1(δ˙t)=vttan(δt+δ˙tΔt)L. _t+1( δ_t)= v_t ( _t+ δ_t t)L. (4) Thus, the realized lateral acceleration depends on both the next speed, determined by a, and the next steering angle, determined by δ˙ δ. The jointly feasible action set is ℱt=(at,δ˙t):at+1lon(at)∈t+1lon,at+1lat(at,δ˙t)∈t+1lat,F_t= \(a_t, δ_t):a^lon_t+1(a_t) ^lon_t+1,\ a^lat_t+1(a_t, δ_t) ^lat_t+1 \, (5) and is generally not separable into independent intervals. A key observation is that ωt+1 _t+1 depends only on the steering-rate command and the known current speed vtv_t. Consequently, for fixed δ˙t δ_t, at+1lata^lat_t+1 is linear in vt+1v_t+1, allowing the lateral band t+1latA^lat_t+1 to be inverted analytically into an interval on vt+1v_t+1. Constructing the feasible box. The box is a deterministic function of the current state. We first intersect the actuator-rate limits with the mechanical angle constraint |δt+δ˙tΔt|≤δmax| _t+ δ_t t|≤ _ . Sampling this entire interval uniformly is ineffective at speed: the lateral band permits only a narrow interval of rates. We therefore invert t+1latA^lat_t+1 at the nominal next speed vnom=vt+atlonΔtv_nom=v_t+a^lon_t t, which yields candidate rates δ˙t=arctan(Lat+1lat/(vtvnom))−δtΔt,at+1lat∈t+1lat. δ_t= \! (L\,a^lat_t+1/(v_tv_nom) )- _t t, a^lat_t+1 ^lat_t+1. The nominal-speed approximation only guides sampling: we widen the resulting rate window, intersect it with the hard rate and angle limits, and retain the full hard interval at low speed. We sample 1717 rates centered on the prior rate and test each against the exact reachable acceleration interval. For each sampled rate, ωt+1 _t+1 is known. Intersecting the speed interval reachable through the longitudinal band with the interval obtained by dividing t+1latA^lat_t+1 by ωt+1 _t+1 gives the exact feasible acceleration interval for that rate: a vertical slice of ℱtF_t (Fig. 3(a)). For every contiguous feasible run, we intersect its acceleration intervals to form a rectangle. The retained box [a¯t,a¯t]×[δ¯˙t,δ˙¯t][ a_t, a_t]×[ δ_t, δ_t] is the candidate with the largest normalized area; when the anchor sample is feasible, candidates must contain it. Its middle discrete action therefore repeats the last action whenever compatible with comfort. The box is an inner approximation of ℱtF_t, recomputed at every control step, and every command it contains is feasible by construction. 3.2 Clipped and adaptive action mapping Both variants draw from the same nominal 7×137× 13 grid of accelerations and steering rates; they differ in what an action index means (Fig. 3(b)). Clipped. Action indices keep fixed physical values, and the requested command is clipped componentwise into the box. Semantics are stable, but whenever the box is smaller than the grid span — the typical regime for steering at speed — many indices collapse onto the same box border: effectively only the extreme commands remain distinguishable, and the policy cannot tell from the action alone what will actually be executed. Adaptive. To ensure a fair comparison with the Classic model in PufferDrive, we use the same number of discrete actions (7×137× 13), while remapping them to acceleration and steering-rate commands. These discrete actions are mapped to acceleration and steering-rate commands and re-discretized onto the current feasible box at every step using a piecewise-linear anchored map: for an interval [ℓt,ht][ _t,h_t] and anchor ctc_t, the middle index maps exactly to ctc_t, the extreme indices to the borders, and intermediate indices interpolate linearly on each side. The anchor is chosen as the previous step’s realized action whenever it remains feasible under the current box; otherwise, the interval is mapped without an anchor. When used, the anchor gives the middle action the semantics “repeat what you just did”. All 7×137× 13 actions remain distinct and feasible by construction — nothing saturates — at the price of relative semantics: the same index denotes different physical commands in different states. ctc_t−-0.50.00.51.01.5−-0.04−-0.020.000.020.04ata_t [m/s2]δ˙t δ_t [rad/s]t+1lonA^lon_t+1t+1latA^lat_t+1ℱt+1F_t+1 (a) Feasible Box index kkcommandhth_tℓt _tcollapsedindicesmidanchor ctc_t (b) Action Resolution Overlay Figure 3: Construction and utilization of the feasible action space. (a) Per-step construction of the feasible action box (computed from an actual vehicle state; see text for parameters). The orange band denotes the longitudinal target interval t+1lonA^lon_t+1, while the blue region contains all (at,δ˙t)(a_t, δ_t) pairs satisfying the lateral target band t+1latA^lat_t+1. Their intersection (purple) is the feasible set ℱt+1F_t+1, from which the largest axis-aligned rectangle (black) is retained. (b) Comparison of action resolution strategies against the feasible interval [ℓt,ht][ _t,h_t]. In the Clipped approach (purple arrows and dark purple dots), fixed grid values saturate at the borders, causing multiple indices to collapse. In the Adaptive approach (teal line and dots), the grid is directly re-discretized onto the interval, locking the middle index to the anchor ctc_t (circled) and spreading extremes exactly to the borders so that all indices stay distinct and feasible. 3.3 Observations The feasible box depends on the previous realized accelerations and steering rate, which are not visible in the instantaneous pose — a policy without this information would face a partially observed problem, since identical observations could resolve the same action differently. Both variants therefore observe, in addition to the standard ego features, the previous realized action (the anchor) and the four box edges, normalized by the profile limits. The two variants share the identical observation layout, so their comparison isolates the action-mapping strategy alone. 4 Experiments We evaluate our comfort-bounded kinematic models against unconstrained and naively bounded baselines. Our experiments are designed to address two key questions: (1) Limit Compliance: does enforcing the comfort envelope actually eliminate envelope-violating motion? (2) Navigation Cost: what, if any, is the cost of these comfort constraints on navigation performance? 4.1 Experimental Setup We evaluate our models in PufferDrive, a data-driven 2D bird’s-eye view simulator built on the Waymo Open Motion Dataset (WOMD) [10] and parallelized via PufferLib [22]. All vehicles in a scene are controlled simultaneously for episodes of 9.19.1\,s (Δt=0.1 t=0.1\,s), with agents removed upon collision, leaving the drivable area, or reaching their goal. We train on 80,00080,000 WOMD scenarios and evaluate on the held-out validation split of 10,00010,000 scenarios. Policies are trained using Proximal Policy Optimization (PPO) [21] for 22 billion steps across three seeds, using a recurrent actor-critic network and simple reward terms such as collision, offroad and goal reaching. All configurations share identical hyperparameters and training budgets, differing only in their action spaces and bound enforcements. 4.2 Model Configurations We compare the six kinematic models introduced above. The three unconstrained baselines, Classic, Classic (steer angle) and Jerk (Section 2.4), check no action against the comfort envelope (Figure 2). The first two differ only in whether the policy commands the steering rate or the steering angle. The pair isolates the effect of that interface choice from the effect of enforcing the envelope. The other three enforce the envelope, in increasing order of how much of the feasible set they preserve: Jerk (bounded grid) restricts the jerk grid a priori with a single static conservative grid; Clipped Classic clips each command into the per-step feasible box axis by axis; and Adaptive Classic (ours) re-discretizes its grid onto that box, so every selectable action is feasible by construction (Section 3.2). Every Classic model except the one qualified as (steer angle) commands steering rate rather than angle (Section 6). The comfort envelope is a property of the simulator, not the model: each constrained model builds its feasible set from either the aggressive or the tighter normal band (Figure 2). Unless stated otherwise, results enforce the aggressive band. 4.3 Evaluation Protocol Each trained policy is evaluated on the validation split, and we report mean ± standard deviation over the three seeds. We report three navigation metrics (the share of agents reaching their goal, the share colliding, and the share leaving the drivable area) and, for each of the four kinematic quantities (alon,alat,jlon,jlata^lon,a^lat,j^lon,j^lat), the share of steps that violate the comfort envelope. Two properties of this tally matter for reading the results. First, we measure the realized quantity, not the commanded one: a model that bounds its inputs but still drives outside the envelope through the coupling terms is counted as violating, which is precisely the failure mode naive clamping exhibits. Second, the tally covers only the steps an agent actually drives (steps after removal or goal completion are excluded), so the rate is a share of driving rather than of episode length, which would otherwise reward a model for finishing early. 4.4 Qualitative Analysis and PufferDrive-Editor Figure 4: The PufferDrive-Editor user interface. The upper panel displays the map canvas and vehicle trajectory paths (e.g., analyzing the reactive policy in orange). The lower panel contains the real-time kinematics dashboard, showing the active passenger-comfort limits (green/amber boxes) for acceleration and jerk, the vehicle’s actual realized state trajectory (highlighted dots and trail), and target action re-discretization intervals. Aggregate metrics over the validation split average away the rare, safety-critical moments in which the limits actually bind. To inspect behavior near those limits we present PufferDrive-Editor, a browser-based scene editor and policy-debugging interface that runs locally on the map and scenario binaries loaded by PufferDrive. Although motivated by the analysis of our constraint-aware kinematic models, PufferDrive-Editor is a general tool for understanding learned driving policies in PufferDrive, enabling researchers to identify policy strengths, weaknesses, and failure modes beyond aggregate metrics. It combines three functions. A scenario and track editor loads road layouts from scenario binaries and lets researchers author stress tests by moving or inserting static obstacles and placing dynamic vehicles with custom poses, velocities, trajectories, and goals (e.g., narrow corridors or parked cars); this isolates skills such as slalom maneuverability and short-distance emergency braking. A real-time kinematics dashboard plots the realized metrics at every step against the OPM boundaries (Fig. 4), highlighting comfort exceedances in red so it is immediately visible whether a breach occurred in acceleration or jerk. Finally, multi-model auditing compares two rollouts either side by side, with synchronized timelines and feasible action boxes ℱ(s)F(s), or overlaid on a single map, making visible where a policy under naive clamping saturates its steering and leaves the lane while the adaptive model preserves the authority to negotiate the obstacle. 5 Results We now return to the two questions that motivated our experiments: whether enforcing the comfort envelope actually eliminates envelope-violating motion, and what it costs in navigation performance. We first report the aggregate numbers across all six kinematic models (Section 5.1), then turn to the stress-test scenes that expose the mechanism behind those numbers (Section 5.2). 5.1 Quantitative Performance on PufferDrive Table 1 reports navigation performance and the realized-kinematics violation rate for all six kinematic models over three seeds; Figure 5 resolves the violation columns into the full driving-zone distribution. Unconstrained control drives outside the envelope. The unconstrained Classic baselines achieve the highest goal completion performance, but only by driving outside the comfort limits. They exceed even the aggressive comfort envelope on far more than a quarter of their steps, mostly in lateral jerk, followed by longitudinal jerk and lateral acceleration, with maximum violations reaching 4513% for jerk and 93% for acceleration. In comparison, the constrained variants only exceed the bounds when e.g. the physical steering-angle limit prevents their enforcement, with maximum violations of just 0.26% for jerk and 9×10−49× 10^-4% for acceleration. Which steering command the policy holds barely matters for the performance metrics, the steer-rate and steer-angle variants are within noise of each other in goal completion and collisions. Commanding the rate integrates the steering and thereby reduces the violations that a single abrupt angle change can produce. While the unconstrained Jerk baseline removes acceleration violations by construction, it worsens jerk violations because the action space of the jerk model is already partially outside the aggressive bounds. Enforcing the envelope costs little. Both clipping and adaptive re-discretization reduce comfort violations to essentially zero (Fig. 5), while reducing goal completion by around two percentage points compared to the unconstrained Classic baseline. In contrast, the bounded-grid Jerk baseline achieves lower navigation performance, indicating that a single static action grid is overly restrictive across varying vehicle speeds. Among the per-step projection methods, adaptive re-discretization consistently performs best: it matches clipping’s comfort guarantees while achieving slightly higher goal completion and lower collision and off-road rates under the aggressive profile. The same trend is maintained under the tighter normal profile. The constraint does not make policies drive conservatively. Zero violations alone would be insufficient evidence if the policy simply avoided operating near the limits. Figure 5 shows the opposite: the adaptive model spends ≈74%≈ 74\% of driven steps in the aggressive longitudinal-acceleration band and roughly half in the aggressive jerk bands. It operates at the very edge of the safety envelope without crossing it; in contrast, the unconstrained baselines mostly fluctuate between normal operation and violation states for longitudinal and lateral jerk. Table 1: Navigation performance per dynamics model on the PufferDrive validation split, under the AGGRESSIVE and NORMAL comfort profiles, averaged over three seeds, in percent. Unconstrained baselines do not enforce comfort limits and are reported once. Navigation columns give mean ± std. Bold marks the best value per column within each outer group (Unconstrained, Aggressive, or Normal): the unconstrained baselines buy their navigation numbers with envelope-violating control, so comparing them against constrained models would be unfair. Dynamics model Goal ↑ Collisions ↓ Offroad ↓ Unconstrained Classic 98.74±0.0398.74± 0.03 0.48±0.020.48± 0.02 0.25±0.020.25± 0.02 Classic (steer angle) 98.79±0.0398.79± 0.03 0.49±0.020.49± 0.02 0.27±0.020.27± 0.02 Jerk 97.68±0.0297.68± 0.02 1.16±0.031.16± 0.03 0.53±0.040.53± 0.04 Aggressive Jerk (bounded grid) 96.39±0.0996.39± 0.09 1.79±0.041.79± 0.04 1.00±0.071.00± 0.07 Clipped Classic 96.77±0.0696.77± 0.06 1.68±0.021.68± 0.02 0.80±0.050.80± 0.05 Adaptive Classic 96.89±0.0496.89± 0.04 1.65±0.031.65± 0.03 0.71±0.020.71± 0.02 Normal Jerk (bounded grid) 92.32±0.0892.32± 0.08 2.39±0.062.39± 0.06 1.27±0.071.27± 0.07 Clipped Classic 92.51±0.0392.51± 0.03 2.12±0.022.12± 0.02 1.22±0.021.22± 0.02 Adaptive Classic 92.73±0.0492.73± 0.04 2.27±0.012.27± 0.01 1.15±0.011.15± 0.01 Figure 5: Driving-zone occupancy of the realized kinematics: the share of driven steps whose realized quantity falls in the normal, aggressive, or violating band of the comfort envelope, per kinematic model (mean over three seeds). The constrained models enforce the aggressive comfort profile (bounded constraints); the unconstrained baselines enforce no limits. The adaptive model fills the aggressive band without ever leaving it; the unconstrained baselines place a large share of their mass outside the envelope. The shaded column adds how far outside: the max penetration, i.e. the worst-case overshoot past the crossed bound relative to that bound (+100%+100\% = twice the limit). Its bars are proportional within a panel, each scaled to the axis printed below it. Tightening the envelope. Table 1 repeats the comparison under the tighter normal comfort profile, where the models may only command actions within the comfortable band. All families degrade by a similar margin (≈4≈ 4 points of goal reaching) as the feasible set shrinks, and adaptive re-discretization remains the strongest constrained driver: the adaptive steer-rate model reaches 92.73%92.73\% goal completion, the best of any constrained model under the normal profile, while keeping violations at zero because it preserves full action resolution inside whatever box remains rather than collapsing a fixed grid onto its boundary. 5.2 Qualitative Analysis in Stress-Test Scenes Aggregate metrics conceal what happens where the limits bind. Inspecting rollouts in the hand-authored stress-test scenes (Section 4.4) with PufferDrive-Editor’s dashboard (Fig. 4), the unconstrained configurations show acceleration and jerk spikes leaving the envelope just before sharp turns or collisions, while our adaptive model keeps inputs smooth and strictly inside the bounds. Figure 6 makes this concrete on a slalom under four models: the bounded-grid Jerk baseline deviates from the reference and leaves its goals unreached, whereas our adaptive model tracks the route through the slalom and reaches its goals cleanly within the comfort bounds. (a) Classic (steer angle). (b) Clipped. (c) bounded Jerk. (d) Adaptive (ours). Figure 6: The same stress-test scenes under four kinematic models, zoomed to the identical region around the policy-controlled route. Policy-controlled agents are orange (numbered 4–6, dashed routes; goals shown as numbered ringed dots); parked agents are blue. Our adaptive model (d) tracks the reference through the scene and reaches all goals within the comfort bounds, whereas the bounded-grid Jerk baseline (c) deviates from the route and leaves its goals unreached (empty rings); the unconstrained Classic (steer angle) (a) and clipped (b) variants are shown for reference. 6 Discussion The quantitative results and qualitative profiles validate our hypotheses regarding RL action spaces under constraints, yielding several insights. Comfort vs. navigability trade-off. Moving from the aggressive to the normal profile consistently drops goal completion by roughly 44 percentage points and raises collisions, as the tighter bounds curb aggressive braking and acceleration: passenger comfort directly constrains defensive reactivity. One might worry that bounding the action space removes the emergency maneuvers a policy needs to stay safe, but the envelope is a configurable profile rather than a fixed ceiling. The aggressive band sits at the extremes of human driving, so the policy keeps the full range a human would use and forgoes only motion beyond it, and a deployment wanting a wider margin can enforce a looser profile. Empirically the constraint rarely binds, costing under two points of goal completion against the unconstrained baseline while removing nearly all violations. Adaptive mapping beats clipping and jerk control. Adaptive mapping outperforms clipped steering and jerk control, mainly in safety-critical navigation. While all constrained methods achieve similar goal completion, adaptive re-discretization preserves steering resolution within the feasible range, improving control authority when avoiding collisions and off-road events. Clipping instead saturates the fixed action grid as the steering range shrinks at high speed. Jerk control is weakest in these hard-navigation settings because second-order actions introduce an additional integration chain (j→a→v→xj→ a→ v→ x), making the control problem harder to learn and credit assignment more difficult. Thus, while jerk control is simple to constrain, direct first-order control is better suited for precise obstacle avoidance and boundary recovery. Action-space design: rate vs. angle, hard vs. soft. We command steering rate (δ˙ δ) rather than steering angle (δ) by default, and all constrained Classic variants use it. Rate control provides a useful balance between responsiveness and smoothness: it avoids abrupt steering changes while retaining more control authority than jerk-based control, which introduces an additional integration chain. The two unconstrained variants are compared in Table 1 to isolate the effect of the steering action space where both perform comparably bad. Alternatively, comfort can be encouraged through reward penalties, as Gigaflow [6] does for harsh acceleration and jerk. Such a penalty prices comfort against other objectives through a tunable weight rather than ruling a maneuver out, so the optimal policy still accepts a violation whenever the payoff of an evasive maneuver exceeds its cost. Reward design therefore shapes driving preferences but cannot reliably define which actions should be considered unacceptable. Hard action constraints and reward shaping are complementary: the constraint makes out-of-envelope motion unreachable by construction and so guarantees compliance with the modeled comfort envelope [2, 7], while the reward can optimize driving style within the feasible region. 6.1 Conclusion We introduced a constraint-aware, adaptive action-mapping framework to address the reliance of learned driving policies on maneuvers no human driver would perform and no occupant would accept. By analytically inverting the lateral-jerk constraint in next-steering-angle space, our approach dynamically scales the control grid to span exactly the comfort envelope at each step. On WOMD and a hand-authored slalom stress test, the proposed adaptive steer-rate Classic model enforced comfort compliance (limiting comfort violations to below 0.01%0.01\%) and outperformed clipped and direct-jerk baselines in navigability. Because every action the policy can select now lies inside the enforced comfort envelope, the resulting safety metrics reflect policy quality rather than the simulator’s permissiveness, closing the gap that motivated this work. Interactive audits via PufferDrive-Editor confirmed that adaptive re-discretization preserves steering authority and prevents grid collapse, allowing the policy to negotiate the slalom’s tight turns and narrow gaps. Acknowledgement This work is part of BrainLinks-BrainTools which is funded by the Federal Ministry of Economics, Science and Arts of Baden-Württemberg within the sustainability program for projects of the excellence initiative I. Also, this research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 539134284, through EFRE (FEIH_2698644) and the state of Baden-Württemberg. References [1] S. A. Alizadeh Kolagar, M. Heydari Shahna, and J. Mattila (2024) Combining deep reinforcement learning with a jerk-bounded trajectory generator for kinematically constrained motion planning. arXiv preprint arXiv:2410.20907. Cited by: §2.3. [2] M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu (2018) Safe reinforcement learning via shielding. In AAAI, Cited by: §6. [3] I. Bae, J. Moon, J. Jhung, H. Suk, T. Kim, H. Park, J. Cha, J. Kim, D. Kim, and S. Kim (2019) Self-driving like a human driver instead of a robocar: personalized comfortable driving experience for autonomous vehicles. In Machine Learning for Autonomous Driving Workshop at the 33rd Conference on Neural Information Processing Systems (NeurIPS), Vancouver, Canada. Cited by: §1, Figure 2, Figure 2, §2.2, §2.2. [4] R. Cheng, G. Orosz, R. M. Murray, and J. W. Burdick (2019) End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In AAAI, p. 3387–3395. Cited by: §2.3. [5] H. Cui, T. Nguyen, F. Chou, T. Lin, J. Schneider, D. Bradley, and N. Djuric (2020) Deep kinematic models for kinematically feasible vehicle trajectory predictions. In IEEE Int. Conf. Robot. Autom. (ICRA), p. 10563–10569. Cited by: §2.3. [6] M. Cusumano-Towner, D. Hafner, A. Hertzberg, B. Huval, A. Petrenko, E. Vinitsky, E. Wijmans, T. Killian, S. Bowers, O. Sener, V. Koltun, and P. Krähenbühl (2025) Robust autonomy emerges from self-play. In ICML, Proceedings of Machine Learning Research (PMLR), Vol. 267, p. 11710–11737. Cited by: §2.3, §6. [7] G. Dalal, K. Dvijotham, M. Vecerik, T. Hester, C. Paduraru, and Y. Tassa (2018) Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757. Cited by: §2.3, §6. [8] K. N. de Winkel, T. Irmak, R. Happee, and B. Shyrokau (2023) Standards for passenger comfort in automated vehicles: acceleration and jerk. Applied Ergonomics 106, p. 103881. Cited by: §2.2. [9] M. Elbanhawi, M. Simic, and R. Jazar (2015) In the passenger seat: investigating ride comfort measures in autonomous cars. IEEE Intelligent Transportation Systems Magazine 7 (3), p. 4–17. Cited by: §2.2. [10] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp, C. R. Qi, Y. Zhou, Z. Yang, A. Chouard, P. Sun, J. Ngiam, V. Vasudevan, A. McCauley, J. Shlens, and D. Anguelov (2021) Large scale interactive motion forecasting for autonomous driving: the waymo open motion dataset. In ICCV, p. 9690–9699. Cited by: §4.1. [11] F. Feng, S. Bao, J. R. Sayer, C. Flannagan, M. Manser, and R. Wunderlich (2017) Can vehicle longitudinal jerk be used to identify aggressive drivers? an examination using naturalistic driving data. Accident Analysis & Prevention 104, p. 125–136. External Links: Document Cited by: §1, §2.2. [12] C. Gulino, J. Fu, W. Luo, G. Tucker, E. Bronstein, Y. Lu, J. Harb, X. Pan, Y. Wang, X. Chen, J. D. Co-Reyes, R. Agarwal, R. Roelofs, Y. Lu, N. Montali, P. Mougin, Z. Yang, B. White, A. Faust, R. McAllister, D. Anguelov, and B. Sapp (2023) Waymax: an accelerated, data-driven simulator for large-scale autonomous driving research. In NeurIPS, Cited by: §1, §2.3. [13] S. Kazemkhani, A. Pandya, D. Cornelisse, B. Shacklett, and E. Vinitsky (2025) GPUDrive: data-driven, multi-agent driving simulation at 1 million FPS. In ICLR, Cited by: §1, §2.3. [14] J. Kong, M. Pfeiffer, G. Schildbach, and F. Borrelli (2015) Kinematic and dynamic vehicle models for autonomous driving control design. In IEEE Intelligent Vehicles Symposium (IV), p. 1094–1099. Cited by: §1, §2.1, §2.3. [15] H. Krasowski, X. Wang, and M. Althoff (2020) Safe reinforcement learning for autonomous lane changing using set-based prediction. In IEEE Int. Conf. Intell. Transp. Syst. (ITSC), Cited by: §2.3. [16] D. Martin and D. Litwhiler (2008) An investigation of acceleration and jerk profiles of public transportation vehicles. In ASEE Annual Conference & Exposition, Cited by: §2.2. [17] N. Montali, J. Lambert, P. Mougin, A. Kuefler, N. Rhinehart, M. Li, C. Gulino, T. Emrich, Z. Yang, S. Whiteson, B. White, and D. Anguelov (2023) The waymo open sim agents challenge. In NeurIPS, Cited by: §2.2. [18] P. Polack, F. Altché, B. d’Andréa-Novel, and A. de La Fortelle (2017) The kinematic bicycle model: a consistent model for planning feasible trajectories for autonomous vehicles?. In IEEE Intelligent Vehicles Symposium (IV), p. 812–818. Cited by: §1, §2.1, §2.3. [19] R. Rajamani (2012) Vehicle dynamics and control. 2nd edition, Mechanical Engineering Series, Springer. External Links: Document Cited by: §2.1. [20] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone (2020) Trajectron++: dynamically-feasible trajectory forecasting with heterogeneous data. In ECCV, p. 683–700. Cited by: §2.3. [21] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §4.1. [22] J. Suárez (2023) PufferLib: making reinforcement learning libraries and environments play nice. In Agent Learning in Open-Endedness Workshop at the Conference on Neural Information Processing Systems (NeurIPS), Cited by: §4.1. [23] M. Theile, L. Dirnberger, R. Trumpp, M. Caccamo, and A. L. Sangiovanni-Vincentelli (2025) Action mapping for reinforcement learning in continuous environments with constraints. Reinforcement Learning Journal 6. Cited by: §2.3. [24] E. Vinitsky, N. Lichtlé, X. Yang, B. Amos, and J. Foerster (2022) Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real world. In NeurIPS, Cited by: §1, §2.3. [25] X. Wang, H. Krasowski, and M. Althoff (2021) CommonRoad-RL: a configurable reinforcement learning environment for motion planning of autonomous vehicles. In IEEE Int. Conf. Intell. Transp. Syst. (ITSC), Cited by: §2.3. [26] D. Xu, Y. Chen, B. Ivanovic, and M. Pavone (2023) BITS: bi-level imitation for traffic simulation. In IEEE Int. Conf. Robot. Autom. (ICRA), p. 2929–2936. Cited by: §2.2. [27] L. Zheng, S. Son, J. Liang, X. Wang, B. Clipp, and M. C. Lin (2024) Deep stochastic kinematic models for probabilistic motion forecasting in traffic. arXiv preprint arXiv:2406.01431. Cited by: §2.3.