Paper deep dive
Action-Slot: Structured Action-Centric Representation Learning for Multi-Agent Atomic Activity Understanding
Yu-Ho Chang, Chi-Hsi Kung, Yi-Hsuan Tsai, Yi-Ting Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/27/2026, 3:47:32 AM
Summary
The paper introduces Action-Slot, a structured action-centric representation learning framework for multi-agent atomic activity understanding in traffic scenes. It adapts slot attention by introducing category-aligned action slots, parallel spatio-temporal updating, and background/negative-slot regularization to disentangle concurrent activities. The framework also includes an attention-difference-based pseudo mask selection for weakly supervised localization. Additionally, the authors propose TACO, a synthetic dataset with pixel-level annotations, and demonstrate state-of-the-art performance on OATS, TACO, and nuScenes.
Entities (11)
Relation Signals (11)
Action-Slot → enables → weakly supervised localization
confidence 95% · enabling weakly supervised localization without dense annotations.
TACO → isconstructedusing → CARLA
confidence 95% · TACO, a large-scale synthetic dataset constructed using CARLA [27].
TACO → isusedfor → systematic evaluation
confidence 95% · To support systematic evaluation, we introduce TACO... Experiments on OATS, TACO, and annotated nuScenes show superior recognition
Action-Slot → proposescomponent → parallel spatio-temporal slot updating
confidence 95% · (2) parallel spatio-temporal slot updating for holistic video-level reasoning
Action-Slot → proposescomponent → background and negative-slot regularization
confidence 95% · (3) background and negative-slot regularization that enforces competition between foreground activities and irrelevant regions.
Action-Slot → proposescomponent → category-aligned action slots
confidence 95% · We reformulate slot learning... through three designs: (1) category-aligned action slots that anchor slots to predefined activity categories
Action-Slot → uses → Slot Attention
confidence 95% · We introduce Action-Slot... reformulate slot learning as structured activity decomposition... Slot attention is widely used for object-centric decomposition
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Atomic activity understanding aims to recognize and localize structured traffic behaviors that jointly encode motion patterns and their grounding in road topology. Unlike conventional action recognition, atomic activities are multi-agent, multi-label, and topology-aware: multiple activities co-occur while many agents remain inactive. We introduce Action-Slot, a structured action-centric representation learning framework. Slot attention is widely used for object-centric decomposition, but its permutation-invariant design and object-level inductive bias are misaligned with atomic activity semantics. We reformulate slot learning as structured activity decomposition through three designs: (1) category-aligned action slots that anchor slots to predefined activity categories, (2) parallel spatio-temporal slot updating for holistic video-level reasoning, and (3) background and negative-slot regularization that enforces competition between foreground activities and irrelevant regions. Together these establish an activity-centric inductive bias that disentangles concurrent and asynchronous activities directly from raw video. Beyond recognition, the learned representations encode transferable spatio-temporal grounding signals. We further propose an attention-difference-based pseudo mask selection framework that suppresses false positives by measuring attention changes before and after candidate region removal, enabling weakly supervised localization without dense annotations. To support systematic evaluation, we introduce TACO, a balanced synthetic dataset with full atomic activity coverage and pixel-level annotations. Experiments on OATS, TACO, and annotated nuScenes show superior recognition, strong sim-to-real transfer, and state-of-the-art weakly supervised localization.
Tags
Links
- Source: https://arxiv.org/abs/2609.24127v1
- Canonical: https://arxiv.org/abs/2609.24127v1
Trouble viewing inline? Open PDF directly →
Full Text
98,441 characters extracted from source content.
Expand or collapse full text
Action-Slot: Structured Action-Centric Representation Learning for Multi-Agent Atomic Activity Understanding Yu-Ho Chang∗ Chi-Hsi Kung∗ Yi-Hsuan Tsai Yi-Ting Chen† Abstract Atomic activity understanding aims to recognize and localize structured traffic behaviors that jointly encode motion patterns and their grounding in road topology. Unlike conventional action recognition, atomic activities are inherently multi-agent, multi-label, and topology-aware, where multiple activities may co-occur while many agents remain inactive. This setting calls for structured action-centric representation learning, requiring decomposing a scene into semantically meaningful activity components. We introduce Action-Slot, a structured action-centric representation learning framework. While slot attention has been widely used for object-centric decomposition, its permutation-invariant design and object-level inductive bias are misaligned with atomic activity semantics. We reformulate slot learning as structured activity decomposition through three key designs: (1) category-aligned action slots that impose semantic anchoring over predefined atomic activity categories, (2) parallel spatio-temporal slot updating for holistic video-level reasoning, and (3) background and negative-slot regularization that enforces competition between foreground activities and irrelevant regions. Together, these designs establish an activity-centric inductive bias that disentangles concurrent and asynchronous atomic activities directly from raw video. Beyond recognition, we show that the learned representations encode transferable spatial-temporal grounding signals. We further propose an attention-difference–based pseudo mask selection framework that suppresses false positives by measuring attention changes before and after candidate region removal, enabling effective weakly supervised localization. To support systematic evaluation, we introduce TACO, a balanced dataset with full atomic activity coverage and pixel-level annotations. Extensive experiments across synthetic and real-world datasets demonstrate superior recognition performance, cross-domain generalization, structured decomposition analysis, and weakly supervised localization. Overall, this work establishes a principled action-centric representation learning framework that unifies recognition and spatial reasoning for multi-agent atomic activity understanding. Index Terms: Multi-agent activity recognition and localization, atomic activity, slot attention, structured activity decomposition, and action-centric representation learning. Fig. 1: Overview of Action-Slot for Multi-Agent Atomic Activity Understanding. (a) A multi-agent traffic scene containing multiple concurrent atomic activities, illustrated by colored trajectories. For example, the red trajectory corresponds to the atomic activity Z1-Z4: C+, representing a group of vehicles turning left along a topologically defined path. (b) Action-Slot decomposes the scene into semantically aligned action slots, each capturing one atomic activity without relying on object proposals. (c) The resulting slot representations enable multi-label atomic activity recognition by predicting whether each atomic activity category is present in the scene. (d) The learned action slots provide spatial grounding signals that enable high-quality pseudo-mask generation for weakly supervised target atomic activity localization. I Introduction Understanding who is doing what, where, and when in complex dynamic scenes remains a fundamental challenge in computer vision. In multi-agent environments such as traffic intersections, multiple agents simultaneously execute distinct motion patterns under shared structural constraints such as road topology and traffic rules. Accurate recognition and localization of such structured activities are essential for downstream applications, including intention prediction [1, 2, 3], scenario retrieval [4, 5, 6, 7], and scenario-based safety evaluation [8, 9, 10]. Atomic activity, introduced in OATS [11], provides a structured abstraction of traffic behavior by grounding motion patterns in the underlying road topology. Unlike conventional action recognition, atomic activity understanding is inherently multi-agent, multi-label, and topology-aware, where multiple activities may co-occur within a single scene, while many agents remain inactive. Figure 1 illustrates the central challenge and motivation of this work. As shown in Fig. 1 (a), a traffic scene may contain multiple concurrent atomic activities, as well as inactive objects such as the standing pedestrian at the far-right corner. Rather than encoding such complex dynamics with a compact feature, we argue that effective atomic activity understanding must decompose a scene into semantically meaningful activity components, selectively attend to relevant spatio-temporal regions, and suppress irrelevant agents and background clutter. Existing approaches are not well aligned with these structural requirements. Video-level models based on 3D ConvNets [12, 13, 14, 15] or the Transformer [16, 17, 18, 19] aggregate global features but lack explicit mechanisms for disentangling concurrent activities. Object-aware methods [20, 21, 22, 23, 11] leverage proposal-based representations [24, 25], yet they rely on explicit object supervision and primarily encode object identity rather than activity semantics grounded in topology. These limitations reveal a gap: while object-centric decomposition has been extensively studied, activity-centric decomposition remains underexplored. This motivates a central question: Can we learn semantically structured, action-centric representations that decompose complex multi-agent activities without relying on object proposals? To address this question, we propose Action-Slot, a structured action-centric representation learning framework that reformulates slot attention as a mechanism for structured activity decomposition. Slot attention [26] provides a mechanism for extracting object-centric representations. However, directly applying it to atomic activities exposes three fundamental mismatches: (1) permutation invariance conflicts with fixed semantic activity categories; (2) frame-wise recurrent updates limit global temporal reasoning; and (3) object-centric inductive biases do not align with higher-order activity semantics. We introduce three principled designs. First, category-aligned action slots associate each slot with a predefined atomic activity category, transforming permutation-invariant slots into semantically anchored activity components. Second, parallel spatio-temporal updating enables holistic reasoning across the entire video rather than locally confined frame-level updates. Third, background and negative-slot regularization introduces explicit competition between foreground activity slots and background. This design enforces slot learning not attending to inactive regions. Collectively, these designs establish a new action-centric inductive bias for atomic activity understanding, as conceptualized in Fig. 1 (b). The learned representations encode spatial grounding signals beyond recognition. As illustrated in Fig. 1 (d), we extend Action-Slot to weakly supervised target atomic activity localization, where only video-level labels are available. Raw attention maps are often diffuse and contain false positives. We therefore propose an attention-difference–based pseudo mask selection framework that measures attention changes before and after candidate region removal to identify discriminative regions. The resulting selected pseudo masks provide effective supervision for localization. To facilitate systematic evaluation of atomic activity understanding, we introduce Traffic Activity Recognition Dataset (TACO), a large-scale synthetic dataset constructed using CARLA [27]. TACO provides balanced coverage of all defined atomic activity categories, including rare atomic activities, and includes pixel-level annotations for localization evaluation. It addresses two major gaps in OATS: class imbalance and the absence of dense spatial annotations. We further annotate atomic activities on nuScenes [28] to assess cross-domain generalization from synthetic to real-world data. Extensive experiments validate Action-Slot from multiple perspectives. First, it achieves superior recognition performance on both OATS [11] and TACO. Second, we show that pretraining on TACO improves cross-domain performance on a real-world dataset, and that validates the effectiveness of the collected large-scale, cost-efficient synthetic data and the learned action-centric representations. Third, we prove that action-centric decomposition is superior to explicit object-level guidance in capturing concurrent atomic activities. Finally, supervision with refined pseudo masks yields state-of-the-art weakly supervised target atomic activity localization performance [29, 30]. Overall, this work positions action-centric representation learning as a general framework for atomic activity understanding, bridging recognition and spatial reasoning under a unified representation paradigm. This work makes the following contributions: 1. We reformulate slot attention as structured activity decomposition via category-aligned action slots, parallel spatio-temporal updating, and background and negative- slot regularization for atomic activity understanding. 2. We unify recognition and weakly supervised target atomic activity localization through an attention-difference–based pseudo mask selection framework that generates reliable pseudo masks for supervision. 3. We establish TACO, a balanced and topology-aware benchmark with full atomic activity coverage and pixel-level annotations, enabling systematic evaluation of recognition and localization. 4. We demonstrate strong cross-domain generalization across synthetic and real-world datasets, and model architectures that validate the robustness and transferability of action-centric representations. This journal version substantially expands the CVPR 2024 paper [31]. Beyond recognition, we generalize Action-Slot into a structured action-centric representation learning framework, introduce dense annotations enabling localization evaluation, and provide detailed analysis of weakly supervised target atomic activity localization that are not explored in the original conference version. Our source code, dataset, and visualization videos are available at https://hcis-lab.github.io/Action-slot-TPAMI_project_page/. I Related Work I-A Video Action Recognition Recent advances in video action recognition have been driven by increasingly powerful spatio-temporal architectures, including 3D ConvNets [12, 15, 13, 14] and Transformer-based models [32, 33, 16, 17, 34]. Large-scale datasets such as Kinetics [35] have enabled effective pretraining, significantly improving transfer performance on downstream benchmarks, including HMDB [36], THUMOS [37], and Charades [38] datasets. Beyond single-label recognition, the community has increasingly focused on multi-label action recognition, supported by benchmarks such as MultiTHUMOS [39], Charades [38], and AVA [40]. Among related tasks, group activity recognition [41, 42] is particularly relevant, as it models multiple agents performing distinct actions within a shared scene. Atomic activity recognition in traffic environments differs in several fundamental aspects. First, traffic scenes exhibit extreme activity sparsity: most objects remain inactive, forming an explicit negative class that must be modeled rather than implicitly ignored. This sparsity characteristic is largely absent in conventional group activity datasets. Second, atomic activities are tightly grounded in the underlying road topology, requiring joint reasoning over object identities, spatial configurations, and motion patterns under shared environmental constraints. Third, existing spatio-temporal backbones primarily rely on holistic feature aggregation over the entire video and lack explicit mechanisms for structured activity decomposition under such sparsity constraints. As a result, they struggle to disentangle multiple concurrent atomic activities while suppressing inactive regions. A natural alternative is object-centric decomposition [20, 21, 22, 23, 11]. However, object-centric representations partition scenes according to appearance or identity rather than activity semantics. Therefore, object-level decomposition does not directly yield activity decomposition. Slot attention provides a promising foundation and motivates our reformulation toward structured activity decomposition. I-B Representation Learning via Slot Attention Slot attention [26] is an unsupervised framework that is designed to discover object-centric abstractions. The initial work demonstrates its capability on synthetic datasets [43]. Subsequent works have extended slot-based modeling to real-world dynamic scenes [44, 45]. These works incorporates auxiliary signals such as optical flow [46], motion segmentation [47], object locations [48], or depth [48] to stabilize object discovery and temporal tracking. Extending slot attention to learn action-centric representations for atomic activity poses several challenges. First, slots in the original formulation are permutation-invariant, which is desirable for unsupervised object discovery [26, 46, 48, 47, 49, 50, 51]. However, atomic activity recognition relies on a fixed and semantically defined activity vocabulary, where permutation invariance hinders consistent alignment between slots and predefined categories. Second, existing video-based slot architectures, such as SAVi [46] and Slot-VPS [49], are primarily designed for object tracking and segmentation. These methods update slots recurrently on a per-frame basis. In contrast, atomic activity understanding requires holistic temporal reasoning. Third, existing extensions leverage object-level cues (e.g., flow or depth) to stabilize slot learning. These cues are misaligned with activity semantics: many traffic participants are inactive (e.g., pedestrians strolling on the sidewalk and vehicles waiting at traffic lights), and motion alone does not inform atomic activity. As a result, slot learning may be biased toward saliency rather than structured activity decomposition. To overcome these limitations, we reformulate slot binding from object instances to activity semantics, introduce parallel spatio-temporal updating for global temporal reasoning, and incorporate background-aware regularization to handle activity sparsity. To the best of our knowledge, this work is among the first to systematically reformulate slot attention for structured activity decomposition and demonstrate the effectiveness in complex traffic scenarios. I-C Weakly-Supervised Action Localization Recent advances in weakly supervised action segmentation have shown that meaningful localization can emerge from pretrained visual representations and auxiliary structural cues, even without access to pixel-level annotations. For example, VSCR [30] combines heterogeneous contrastive cues, including CAMs, edges, motion, and semantic tags, to provide indirect supervision. More broadly, weakly supervised action localization and segmentation methods commonly follow a synthesize-and-refine paradigm [52, 53, 54, 55], where pseudo masks generated from superpixels [56] or Grad-CAM [57] are progressively refined. For example, Yan et al. [58] employ ranking SVMs together with conditional random fields to identify representative supervoxels, while Chen et al. [59] introduce a cut-and-paste strategy that preserves pseudo masks whose semantics remain consistent under a discriminator. Although effective for general action understanding benchmarks [60], these approaches are less suitable for atomic activity recognition, where activity semantics are strongly coupled with spatial configuration and scene context. For instance, C4-C1:P and C3-C2:P both describe a pedestrian moving in the same direction, yet correspond to distinct activities due to their relative positions within the intersection. Consequently, region-level perturbations such as cut-and-paste operations may alter the underlying atomic activity label and introduce ambiguous supervision signals. Moreover, these approaches rely on recognition models producing sufficiently discriminative attention maps to bootstrap segmentation learning. However, subtle spatial interactions in atomic activities often yield coarse and noisy activations. Without further verification, these methods struggle to effectively correct spurious responses, leading to unreliable pseudo masks. To address this, we exploit learned action-centric representations [31] to induce activity-aligned attention maps. To further eliminate residual spurious responses, we propose an attention-difference-based selection framework that evaluates each region’s causal effect by measuring attention variations upon candidate removal. TABLE I: Comparison of topology-aware traffic activity datasets. We introduce TACO, which provides balanced atomic activity categories with spatio-temporal localization annotations. The symbol † denotes that we extend nuScenes with atomic activity recognition annotation. Dataset # Videos # Labeled Activities # Classes Activity Labels Spatio-Temporal Localization for Atomic Activity Inner-City [61] 59000 (frames) 3.1k 25 High-level Action & Location ✗ LOKI [62] 644 28k 14 Intention ✗ ROAD [63] 22 10.8k 43 High-level Action & Location ✗ ROAD-Waymo [64] 8.5 (hours) 39k 43 High-level Action & Location ✗ OATS [11] 1k 6.6k 59 Atomic Activity ✗ ATARS [65] 39 235k (frames) 38 Atomic Activity ✗ nuScenes† [28] 426 933 51 Atomic Activity ✗ TACO (Ours) 6.8k 16.5k 64 Atomic Activity ✓ I-D Weakly-suervised Video Object Grounding Weakly-Supervised Video Object Grounding (WSVOG) aims to localize objects based on natural language descriptions without access to spatial annotations [66, 67]. A common approach in WSVOG formulates the task as a frame-level multiple instance learning (MIL) problem [68, 69, 29, 70]. MIL-based grounding methods treat spatial proposals as instances within a positive bag and encourage at least one instance to match the supervisory signal. However, under activity sparsity, a video may contain numerous inactive regions that correlate with contextual cues, leading to unstable atomic activity localization. In contrast, the proposed attention-difference pseudo framework exploits Action-Slot’s activity-aligned attention maps to selected pseudo masks, yielding substantially improved localization performance over MIL-based approaches. I-E Traffic Activity Understanding Datasets Traffic activity understanding supports the development of intelligent driving systems, such as scene analysis [71, 2, 3], scenario retrieval [4, 5, 6, 7], and safety-critical scenario generation [9, 72, 73, 74, 75]. Early traffic datasets [61, 71, 76, 77] primarily annotate high-level actions (e.g., left turn) for behavior recognition. However, such coarse labels fail to differentiate actions originating from distinct spatial contexts. For example, a left turn from the oncoming lane conveys different semantics than a left turn initiated from the ego lane. To address this limitation, datasets such as Inner-City [61], LOKI [62], ROAD [63] augment activity with location-aware (e.g., middle of the road) annotations. Recently, OATS [11] unifies action and spatial context through topology-aware description language called Atomic Activity, which decomposes interactive traffic scenarios into atomic activities defined by road-user types and motion patterns tied to road structures. These datasets involve topology-aware semantics and require explicit modeling of activity sparsity. Despite these advances, collecting real-world traffic data with comprehensive and balanced atomic activity distributions remains challenging, particularly for rare events. This limitation hinders systematic evaluation of atomic activity recognition and localization. To overcome these challenges, we construct the TACO dataset utilizing the CARLA simulator [27] to gather instances of all conceivable activity categories, ensuring a well-balanced distribution, and further provide human-labeled spatio-temporal annotations for localization evaluation. Table I compares TACO and the extended nuScenes with existing traffic activity datasets with topology-aware annotations. By establishing a controlled and open-sourced benchmark for topology-aware atomic activity recognition and localization, TACO facilitates systematic evaluation of structured representation learning methods and encourages future research on scalable auto-labeling strategies [78, 79]. Fig. 2: The top of the figure illustrates the proposed framework. Action-Slot takes video as input and uses a CNN encoder to extract feature patches. All patches are then processed with individual slots simultaneously to find the most relevant spatio-temporal patches corresponding to each action slot. The updated action slots are fed into a fully connected layer to predict the probability of the corresponding action class, excluding the background slot. The bottom of the figure depicts the attention maps of action and background slots. We propose to incorporate a background mask M_ bg to supervise the background slot. The design facilitates other action slots to capture action signals. Furthermore, we design a regularization for slots allocated to negative classes (e.g., Z3-Z4: C) using an all-zero mask M_ neg. I Method In this section, we first introduce the background of Slot Attention. We then present an overview of learning action-centric representations for multi-agent atomic activity recognition using our proposed method, Action-Slot, and describe the specific modifications made to adapt Slot Attention to our task. Finally, we introduce a pseudo mask selection framework that enables weakly supervised, query-based atomic activity localization within Action-Slot. I-A Preliminary: Slot Attention The Slot Attention module [26] can be viewed as a clustering algorithm that maps image patches input to a set of K output slots S∈ℝK×DS ^K× D_ slot, where D_ slot is the dimension of each slot. Specifically, the slots S are first initialized by randomly sampling K vectors from a Gaussian distribution, where the parameter K is usually defined as the maximum number of objects in an image. An input frame, i.e., image features F∈F∈ ℝH×W×DR^H× W× D_ in with size H×WH× W and dimension D_ in, is first flattened to patch tokens F′∈F^ ∈ ℝN×DR^N× D_ in, where N=H×WN=H× W. Then the tokens are mapped to the slots using the dot product attention module. That is, the attention weight can be calculated as A=1Dk(F′)⋅q(S)∈ℝN×KA= 1 Dk(F^ )· q(S) ^N× K, and q and k are linear transformations that map the input F and slots S to a common dimension D. We obtain the updated values for slots through U=A¯T⋅v(F′)∈ℝK×DU= A^T· v(F^ ) ^K× D, where A¯ A is the normalized attention weight calculated via the softmax operation and v is a linear transformation. Finally, the slots are updated via a GRU: S′=(S,U)S^ = GRU(S,U) [80]. To refine the slots, the updating process repeats M iterations [26, 46, 48]. Note that the key difference between classical attention [81] and slot attention is that the attention weights A are normalized with the softmax operation slot-wise instead of token-wise. This difference enables slots to compete with each other so that each slot attends to different relevant regions of input. To extend slot attention to video tasks, prior work [46, 48, 47] propagates slots recurrently via a GRU, i.e., St=(St−1,U)S^t= GRU(S^t-1,U). I-B Overview of Proposed Method for Recognition I-B1 Problem Formulation We supervise the model to learn action-centric representations through a multi-label atomic activity recognition task. Specifically, given a video clip ViV_i consisting of T frames Itit=1T\I_t^i\_t=1^T, the objective is to determine whether any atomic activities Y are present in the clip. The ground-truth label YiY^i for video ViV_i is a binary multi-label vector, i.e., Yi=ycc=1NY^i=\y_c\_c=1^N_ cl, where yc=1y_c=1 if the corresponding activity appears in the video and 00 otherwise, and N_ cl denotes the total number of activity classes. Notably, we do not impose temporal constraints on when an activity occurs; an activity ycy_c may appear in any frame of Itit=1T\I_t^i\_t=1^Tand may occur multiple times within a single video. I-B2 Overview of Action-Slot We propose Action-Slot, an action-centric, Slot-Attention-based framework for decomposing atomic activities in videos. Our key idea is to associate each slot with a specific atomic activity, enabling it to learn a dedicated action-centric representation; we refer to these as action slots. An overview of Action-Slot is illustrated in Figure 2. Unlike existing Slot Attention approaches designed for unsupervised object discovery and tracking [46, 48, 47], we introduce three key modifications. First, we allocate a fixed number of slots K equal to the number of atomic activity classes, N_ cl. Each action slot is supervised using the corresponding ground-truth label ycy_c, indicating whether activity c is present in the video. Second, we introduce an additional background slot to capture regions unlikely to contain any activity. By absorbing attention over irrelevant regions, the background slot allows the action slots to focus on activity-relevant areas. Third, we explicitly discourage inactive action slots—corresponding to activities absent from the video—from attending to any regions. Finally, instead of recurrently propagating slots over time as in prior work [46, 48, 47], we update all action slots in parallel by jointly considering all frames. I-C Action-Slot We now describe the details of our Action-Slot design for learning action-centric representations. Following [47], we initialize slots using learnable parameters. We extract image features Given a video clip, we first extract frame-level features F∈ℝT×H×W×DinF ^T× H× W× D_in using video encoder [82, 12, 14, 15] The features are then flattened into F′∈ℝN×DF^ ^N× D_ in, where N=T×H×WN=T× H× W. We further incorporate learnable 3D spatio-temporal positional embeddings E∈ℝT×H×WE ^T× H× W into the tokens F′F^ [32, 33, 16, 47, 17]. I-C1 Allocated Slot We define a set of slots Skk=1K\S_k\_k=1^K, where K=NK=N_ cl denotes the number of atomic activity classes. Unlike prior Slot Attention approaches [26, 46, 48, 49], which do not assign predefined semantics to individual slots and typically rely on Hungarian matching for bipartite alignment with objects, we explicitly associate each slot with a specific activity class, forming action slots. Each action slot is equipped with an independent binary classifier that predicts y^c y_c for the corresponding activity class c. We supervise these predictions using a binary cross-entropy (BCE) loss for each sample: L=∑c=1NBCE(y^c,yc)L_ act= _c=1^N_ cl BCE( y_c,y_c). I-C2 Parallel Updating We update slots across image frames in a parallel manner, as shown in Figure 3 (a). Specifically, we compute normalized attention weight A¯∈ℝN×K A ^N× K over the entire spatio-temporal volume, where N=T×H×WN=T× H× W. The K slots are then updated in a single pass using A¯ A. This differs from prior approaches[46, 48, 47], which update slots recurrently across frames. In those methods, slots are propagated over T iterations, where the previous slot states and the current frame features are used to compute the updated slot states at each time step, as illustrated in Figure 3 (b). Fig. 3: (a) Our parallel scheme considers updating slots based on the spatio-temporal features for all the frames. (b) The slots are updated recurrently along the temporal dimension, where each time the slot only considers a frame-wise feature. I-C3 Background Slot and Attention Guidance Providing direct supervision to action slots corresponding to positive classes (i.e., present atomic activities) is non-trivial, since objects are not continuously involved in activities and not all objects participate in actions. Therefore, we propose to use a background slot that is not supervised by any activity classes. To address this, we introduce a background slot that is not associated with any activity class. Due to the softmax normalization in Slot Attention, the background slot is encouraged to attend to regions irrelevant to activities, thereby allowing action slots to focus on activity-relevant areas (see the bottom of Figure 2). To further strengthen this mechanism, we supervise the background slot with a background mask M_ bg, which explicitly guides it toward non-activity regions. The mask is constructed by excluding pixels belonging to vehicles, pedestrians, drivable areas, crosswalks, and sidewalks. The background attention is optimized using a binary cross-entropy loss: L=BCE(A¯,M)L_ bg= BCE( A_ bg,M_ bg), where A¯ A_ bg denotes the normalized attention map of the background slot. I-C4 Regularization for Action-Slot As discussed earlier, directly guiding the attention of slots corresponding to positive classes is non-trivial. Instead, we introduce a regularization term that discourages action slots associated with negative classes (i.e., activities absent from the video) from attending to any regions. To this end, we construct a negative mask M_ neg whose elements are all zeros. We then define the following loss: L=∑c|yc=0BCE(A¯c,M)L_ neg= _\c|y_c=0\ BCE( A_c,M_ neg), where A¯c A_c denotes the normalized attention output of slot ScS_c for negative class c. This regularization suppresses attention from inactive action slots. Through the inherent competition induced by the softmax operation in Slot Attention, suppressing negative slots encourages slots associated with positive classes to focus more strongly on informative regions. Importantly, this regularization does not require any additional annotations. The overall training objective of Action-Slot is given by L=L+wL+wL_ all=L_ act+w_ bgL_ bg+w_ negL_ neg , where w_ bg and w_ neg balance the background and negative regularization terms (set to 0.5 and 1.0, respectively in our experiments). I-D Weakly-Supervised Target Atomic Activity Localization We extend Action-Slot from video-level recognition to spatio-temporal localization. Our goal is to localize a queried atomic activity using only video-level supervision. We exploit the attention maps extract from each slot as localization cues and refine them to generate pseudo labels for training. Specifically, given a queried atomic activity, we extract its corresponding attention map MkM_k from the associated action slot SkS_k. We perform a two-stage pseudo mask generating procedure – (1) Attention-to-Object Matching and (2) Attention-Difference-Based Pseudo Mask Selection – to obtain reliable pseudo masks, as depicted in Figure 4. Fig. 4: The Overview of the Proposed Pseudo Mask Selection Framework. First, we generate the initial pseudo mask by matching the attention map with nearby object masks. To alleviate the false positive resulting from the diffuse or ambiguous attention map, we inpaint the initial pseudo mask region and identify the important regions from the attention difference before and after inpainting, then re-associate with object masks to generate selected pseudo masks. I-D1 Attention-to-Object Matching To suppress noise in the raw attention map, we first cluster MkM_k and align the resulting clusters with object masks. Concretely, we apply thresholding to MkM_k to remove low-confidence attended regions and apply connected component analysis to obtain attention clusters Ciki=1I\C^k_i\_i=1^I, where the parameter I is the number of connected components. For the ithi^th cluster CikC_i^k, we first identify the cluster centroid (cx,ik,cy,ik)(c_x,i^k,c_y,i^k) with the object center (ox,j,oy,j)(o_x,j,o_y,j) of an object mask ObjjObj_j and determine if there is a match via the two criteria: (1) Minimum Distance, which selects the object whose center is closest to the cluster centroid; (2) Gravity-based Distance. The distance is defined below gravity(j)=area(Obj)d2+ϵ,gravity(j)= area(Obj_j)d^2+ε, where d=‖(cx,ik,cy,ik)−(ox,j,oy,j)‖2d=\|(c_x,i^k,c_y,i^k)-(o_x,j,o_y,j)\|_2. We use Gravity-based Distance for four-wheelers and Minimum Distance for two-wheelers and pedestrians. This mixture matching strategy aims to mitigate the small-object bias inherent in purely distance-based matching. Finally, each attention cluster is replaced by the corresponding matched object masks, producing initial pseudo mask MkinitM_k^init composed of refined clusters C′iki=1I\C ^k_i\_i=1^I. I-D2 Attention-Difference-Based Pseudo Mask Selection Although Attention-to-Object Matching provides object-level grounding, the resulting pseudo masks may still include inactive or irrelevant objects due to fragmented or noisy attention responses. To further refine the pseudo masks, we introduce an Attention-Difference-Based strategy that estimates the causal effect of each candidate object on the slot attention via counterfactual intervention. Specifically, we construct a counterfactual video by masking the regions corresponding to the refined clusters C′iki=1I\C ^k_i\_i=1^I in the initial pseudo mask MkM_k init. We then apply an off-the-shelf video inpainting model, ProPainter [83], to fill in the masked regions, producing an inpainted video V′V in which the candidate objects are removed. Next, we extract the attention map MkM inpaint_k from V′V and compute the attention difference: Mk=max(Mk−Mk,0),M diff_k= (M init_k-M inpaint_k,0 ), which captures the reduction in attention caused by removing the candidate objects. This difference isolates regions that have a genuine causal contribution to a queried atomic activity. Finally, we apply Attention-to-Object Matching again to MkM diff_k to obtain the refined pseudo label MkM final_k. I-D3 Atomic Activity Localization Model We adopt a Mask2Former-style localization architecture [84] to predict the spatio-temporal mask of a queried atomic activity. The model takes as input the refined attention map MkM_k corresponding to the queried activity, together with low-level and high-level spatio-temporal features extracted from a shared backbone encoder (e.g., the third and fifth stages of X3D/ResNet), denoted as FlF_l and FhF_h, respectively. Notably, the localization module does not directly use slot embeddings as features. Instead, we perform a cross-attention operation to integrate the slot-specific attention prior with the low-level features FlF_l: F=Attention(WqMk,WkFl,WvFl),F_ attn=Attention(W_qM_k,\;W_kF_l,\;W_vF_l), where WqW_q, WkW_k, and WvW_v are learnable projection matrices. This operation injects activity-specific priors into the fine-grained feature representation, and that enables the model to focus on regions relevant to the queried atomic activity. The resulting intermediate feature F_ attn is then upsampled and concatenated with the high-level semantic features FhF_h to form the fused representation F_ fusion. Finally, F_ fusion is fed into a lightweight 3D convolutional decoder to predict the spatial-temporal mask M^k M_k. The localization model is trained using a combination of binary cross-entropy (BCE) loss and Dice loss [85], defined as ℒ=ℒ+ℒ.L_ seg=L_ BCE+L_ Dice. IV Experiments We investigate the effectiveness of Action-Slot from four perspectives: (1) Recognition Performance. We compare Action-Slot against video-level and object-aware baselines on OATS and TACO, demonstrating the advantage of action-centric representations for multi-label atomic activity recognition. Ablation studies further justify key design choices. (2) Cross-Domain Generalization. We evaluate representation transfer by pretraining on TACO and testing on real-world datasets, demonstrating improved generalization and robustness across domains. (3) Structured Decomposition Analysis. We investigate whether action-centric decomposition is superior to explicit object-level guidance in capturing concurrent atomic activities. (4) Weakly Supervised Target Atomic Activity Localization Quality. We assess the quality of pseudo masks generated at different stages and show that Action-Slot provides stronger localization guidance. Supervising a simple localization model with our selected pseudo masks outperforms existing weakly supervised baselines. IV-A Datasets IV-A1 TACO We construct TACO, a balanced and topology-aware benchmark with full atomic activity coverage using the CARLA simulator [27]. Scenario Collection. We adopt three complementary approaches for data collection: auto-pilot, Scenario Runner [86], and automatic scenario augmentation [10]. (a) Auto-pilot. We use the built-in autopilot to control both an ego vehicle and surrounding road users. A scenario is automatically recorded when (1) an ego vehicle approaches an intersection and (2) at least one other road user is nearby. We randomly vary the number and spawn locations of road users to increase diversity. (b) Scenario Augmentation [10]. We leverage pre-recorded scenarios from RiskBench [10], including both interactive (an ego vehicle interacts with risky road users) and non-interactive scenarios. Following the augmentation pipeline of RiskBench, we automatically generate diverse variants by introducing additional road users and altering weather conditions. Compared to autopilot-collected data, these scenarios exhibit more human-like maneuvers and riskier interactions. This difference stems from the RiskBench data collection protocol, in which human participants manually control both the ego vehicle and interacting agents. (c) Scenario Runner [86]. The method is developed by CARLA [27] and it enables scripted scenario generation by specifying routes and behaviors for road users. Unlike auto-pilot and augmentation, it allows us to explicitly manipulate the distribution of queried atomic activities. A collection starts when an ego vehicle reaches a predefined trigger point and ends after all scripted actions are completed. To enhance diversity, we additionally spawn random road users around an ego vehicle. For all methods, we randomize weather and lighting, excluding night scenes due to poor visibility. Sensor Suites. We deploy a wide field-of-view camera (120 degrees) to record events taking place on the extreme left and right sides of the ego-vehicle. We collect images and instance segmentation. In addition, we collect instance segmentation from Bird’s eye view. Statistics. The TACO dataset contains 5,178 video clips, of which 1,148 are reserved for testing. In total, we annotate 16,521 video-level atomic activity labels for recognition and 1,939 spatio-temporal annotations for localization. The dataset includes RGB images and instance segmentation maps with an original resolution of 512 × 1536 pixels. All images are downsampled to 256 × 768 for model training. Additional dataset statistics are provided in the supplementary material of the CVPR paper [87]. IV-A2 OATS The OATS dataset [11] was collected in San Francisco using an instrumented vehicle. It consists of 1,026 labeled videos and is partitioned into three splits for cross-validation. OATS defines 59 traffic atomic activity categories. Following the original experimental protocol, we train and evaluate on the selected 35 categories. The original image resolution is 1200 × 1920 pixels, and we downsample images to 224 × 224, as in [11]. IV-A3 nuScenes We annotate video-level atomic activity labels on the train_valtrain\_val split of the nuScenes dataset [28], which contains 850 videos. Our annotation protocol produces 426 short clips, each consisting of 16 frames. The annotated nuScenes subset covers 42 atomic activity classes. IV-B Recognition Performance IV-B1 Comparison with SOTA Video-Level Models. We implement video-level baselines, including I3D [12], X3D [15], CSN [13], SlowFast [14], MViT [16], and VideoMAE [17], using the PyTorchVideo library [88]. All models are adapted for multi-label recognition by replacing their original classifiers with a single linear layer that outputs N_ cl channels. Object-Aware Models. For object-aware baselines, including ORN [20], ARG [21], and OATS [11], we first extract object features using an object tracker followed by RoIAlign [24]. We adopt OC-SORT [89] for tracking and set the maximum number of tracklets to 20 following OATS. To enhance contextual modeling in ORN and ARG [20, 21], we concatenate object features with global video features obtained by applying a 1×1×11× 1× 1 3D convolution to X3D features. Each proposal is classified using a linear layer with N+1N_ cl+1 output channels, where the additional class denotes the absence of any action. We do not include the OATS model [11] in our comparison due to the unavailability of its official implementation. Slot-Based Models. For slot-based models, including SAVi [46, 48], MO [47], Slot-VPS [49], and our Action-Slot, we use extracted features from backbone encoders such as ResNet [82] or X3D [15]. For X3D, we remove the final projection layer, resulting in slightly smaller model sizes for some slot-based variants, as reported in Table I. To ensure fair comparison, we re-implement prior slot-based approaches by explicitly assigning slots to atomic activity classes while preserving their original slot initialization strategies. Specifically, SAVi [46] samples slots from a learned distribution at each forward pass, whereas MO [47] and Slot-VPS [49] treat slots as learnable queries [90]. Regarding slot update strategies, SAVi and MO perform recurrent temporal updates, while Slot-VPS employs an additional self-attention module [81] to update slots across frames. For all methods, we use the final-frame slot representations as input to the classification head. Pretraining and Backbone Choice. We first follow the pretraining protocol of OATS [11] and implement models using a ResNet-50 [82] backbone pretrained on ImageNet [91]. To further investigate the impact of video pretraining, we additionally train models with video backbones [12, 14, 15] pretrained on Kinetics-400 [35] for experiments on both the OATS and TACO datasets. Unless otherwise specified, we adopt X3D [15] as the default backbone encoder due to its compact model size. IV-B2 Implementation Details Background Masks. For TACO, background masks are derived from the automatically annotated instance segmentation in CARLA. For OATS and nuScenes, we generate background masks using an off-the-shelf DeepLabV3+ model [92]. Metrics. Following standard practice in multi-label action recognition [39, 38, 40, 11], we report mean Average Precision (mAP). Visualization. To qualitatively evaluate whether Action-Slot learns meaningful action-centric representations, we visualize attention maps from the allocated action slots. For visualization, we retain pixels with attention scores greater than 0.5 on OATS and 0.2 on TACO. The thresholds are chosen empirically. TABLE I: Quantitative results on the OATS dataset. “Seq” denotes the input sequence length. The symbol ‡ denotes the re-implementation of slot-based methods with explicit slot-to-category assignment. S1, S2, and S3 correspond to the three predefined splits in the OATS dataset. Method Backbone Seq Pretrain S1 S2 S3 mAP CSN [13, 11] ResNet152 32 IG65M 12.1 12.6 12.9 12.5 TPN [93, 11] ResNet50 32 ImageNet 11.6 13.3 12.9 12.6 SlowOnly [14, 11] ResNet50 32 ImageNet 11.2 14.7 12.9 13.0 SlowFast [14, 11] ResNet50 32 None 10.8 15.1 14.5 13.5 I3D (NL) [94, 11] ResNet50 32 ImageNet 11.9 15.5 14.0 13.8 I3D [12, 11] ResNet50 32 ImageNet 11.8 14.3 16.8 14.3 ORN [20, 11] ResNet50 32 ImageNet 16.8 13.4 18.1 16.1 ARG [21, 11] Inceptionv3 32 ImageNet 20.2 21.3 19.3 20.3 OATS [11] Inceptionv3 32 ImageNet 24.3 28.6 27.2 26.7 SAVi‡ [46, 48] ResNet50 32 ImageNet 21.1 22.4 22.5 22.0 MO‡ [47] ResNet50 32 ImageNet 14.3 15.6 18.2 16.0 Slot-VPS‡ [49] ResNet50 32 ImageNet 15.7 17.8 17.2 16.9 Action-Slot (Ours) ResNet50 32 ImageNet 26.6 28.6 30.8 28.6 I3D [12] ResNet50 8 Kinetics-400 21.7 24.6 24.4 23.6 X3D [15] N/A 16 Kinetics-400 30.4 33.2 30.6 31.4 CSN [13] ResNet101 32 Kinetics-400 43.1 47.1 44.3 44.8 SlowFast [14] ResNet50 16 Kinetics-400 36.1 36.6 34.2 35.6 ORN [20] X3D 16 Kinetics-400 19.3 22.5 23.6 21.8 ARG [21] X3D 16 Kinetics-400 24.8 25.9 29.3 26.7 SAVi‡ [46, 48] X3D 16 Kinetics-400 19.0 22.1 21.6 20.9 MO‡ [47] X3D 16 Kinetics-400 25.3 25.0 24.2 24.8 Slot-VPS‡ [49] X3D 16 Kinetics-400 24.7 24.6 25.5 24.9 Action-Slot (Ours) X3D 16 Kinetics-400 48.1 47.7 48.8 48.2 TABLE I: Quantitative results on the TACO dataset. “Seq” refers to the input sequence length. C, K, P, C+, K+, and P+ denote activity groups involving different road user types. The symbol ‡ marks the re-implementation of slot-based methods with explicit slot-to-category assignment. Method Para. (M) Seq C K P C+ K+ P+ mAP I3D [12] 27.3 8 27.3 19.4 30.5 34.6 33.6 34.8 29.7 X3D [15] 3.0 16 37.5 20.3 34.6 56.3 51.5 38.8 38.3 CSN [13] 21.4 32 43.5 35.5 43.0 52.5 46.1 43.4 44.0 SlowFast [14] 33.7 16 33.9 19.7 36.7 39.9 42.6 41.0 35.2 MViT [16] 36.6 16 21.4 13.8 26.3 43.7 30.0 33.8 27.9 VideoMAE [17] 57.9 16 30.6 18.6 27.1 51.6 33.0 37.4 33.1 ORN [20] 4.8 16 25.5 15.8 24.6 31.6 19.8 13.9 22.2 ARG [21] 12.2 16 27.8 15.0 27.2 35.6 13.7 20.0 23.2 SAVi‡ [46, 48] 2.3 16 19.2 13.6 27.5 23.3 25.9 35.9 23.3 MO‡ [47] 2.3 16 34.2 24.9 39.2 38.3 39.4 37.7 35.3 Slot-VPS‡ [49] 3.5 16 31.9 21.3 32.0 51.9 44.7 31.3 36.0 Action-Slot (Ours) 2.3 16 48.1 41.2 49.2 70.1 62.6 52.8 54.4 IV-B3 Main Results OATS. Table I compares our method with video-level, object-aware, and slot-based approaches. In the top group, all models use an ImageNet-pretrained ResNet-50 backbone, following the evaluation protocol of OATS [11]. In the bottom group, we further compare methods equipped with stronger video backbones pretrained on Kinetics-400 [35]. Under both ImageNet and Kinetics pretraining settings, Action-Slot achieves state-of-the-art performance on OATS. These results validate the advantage of Action-Slot in identifying activity-relevant regions through the proposed structured action-centric representation learning framework, without relying on object proposals. TACO. We establish a new benchmark on the proposed TACO dataset to evaluate Action-Slot across a broader and more diverse set of atomic activities. In the top group of Table I, we additionally include two recent video transformers, MViT [16] and VideoMAE [17]. These models perform worse than other video-level baselines, which may be attributed to their data-intensive training requirements [95]. Object-aware models [20, 21, 11], shown in the middle group, achieve suboptimal results, particularly for activities involving groups of road users. This suggests that modeling pairwise object relations using MLPs or GCNs [96] is insufficient for capturing the holistic contextual and motion dynamics required for such activities. In the bottom group of Table I, we re-implement object-centric slot-based models with allocated slots (denoted by ‡ ). Action-Slot outperforms all competing methods by a large margin across all activity classes. The performance gains can be attributed to our parallel slot updating strategy and the proposed attention guidance mechanism. Notably, Action-Slot excels at activities involving groups of road users, highlighting its ability to model holistic action dynamics. Moreover, Action-Slot achieves strong performance while maintaining a compact model size. IV-B4 Ablation Study We conduct ablation studies in Table VI to evaluate the contributions of the proposed designs. Non-Allocated vs. Allocated. Introducing allocated slots significantly improves performance over the non-allocated variant, yielding a gain of 31.9% (ID 1 vs. 3). These findings underscore the necessity of explicit slot-to-category assignment in tasks with predefined activity taxonomies, where permutation-invariant slot attention may not be well aligned with the atomic activity recognition. Recurrent vs. Parallel. Updating action slots in parallel across all frames leads to a 17.9% performance improvement compared to recurrent updating (ID 2 vs. 3). This finding indicates that the parallel strategy better captures global spatio-temporal dependencies. Background Slot and Attention Guidance L_ bg. Simply introducing a non-allocated background slot without mask supervision slightly degrades performance (ID 3 vs. 4), which contrasts with observations in prior slot attention studies [47, 49]. We attribute this to potential attention distraction, where the background slot competes with action slots in the absence of explicit supervision. When supervised with the background mask loss L_ bg, the model achieves additional performance gains compared to using allocated slots alone (ID 3 vs. 5), demonstrating the effectiveness of attention guidance for stabilizing slot competition. TABLE IV: Ablation study of Action-Slot. The reported results are averaged across the three predefined splits of the OATS dataset. ID Allocated Update BG Slot L_ bg L_ neg mAP 1 parallel 10.8 2 ✓ recurrent 24.8 3 ✓ parallel 42.7 4 ✓ parallel ✓ 40.8 5 ✓ parallel ✓ ✓ 43.6 6 ✓ parallel ✓ ✓ 43.0 7 ✓ parallel ✓ ✓ ✓ 48.2 TABLE V: Comparison of backbone architectures for Action-Slot on the TACO dataset. mAP I3D 29.7 Action-Slot 37.6 (+7.9) X3D [15] 37.8 Action-Slot 54.4 (+16.6) SlowFast [14] 35.2 Action-Slot 46.7 (+11.5) TABLE VI: Comparison of Action-Slot and object-level guidance on TACO across different numbers of road users (N) per video. BG and Neg represent the background slot and the regularization term in our framework. N ≤ 5 5 << N ≤ 15 N >> 15 object 49.4 47.7 43.5 BG+Neg 55.2 50.9 46.3 Action Slots Regularization L_ neg. We regularize action slots with the L_ neg term to discourage action slots allocated to negative classes from attending any regions in spatio-temporal features. This design enhances the likelihood of other action slots allocated to positive classes identifying the region of interest more effectively. We demonstrate that the inclusion of the regularization term L_ neg further improves the one only using the allocated slot (ID 3 vs. 6). The final configuration (ID 7 in Table VI) consistently achieves the best performance on OATS, providing comprehensive validation of the proposed Action-Slot design. Action-Slot with Different Backbones. Table VI reports the performance of Action-Slot using different backbone encoders. We choose I3D [12], X3D [15], and SlowFast [14] for experiments. Action-Slot significantly improves all 3D CNN methods, showing great generalization ability. Among them, X3D achieves the most improvement. We hypothesize that this is because X3D’s features retain the original temporal dimension, affording Action-Slot greater spatio-temporal information to decompose atomic activities. In contrast, other backbone encoders downsample the temporal dimension; for instance, SlowFast takes a video sequence of length 16 as input and reduces the temporal dimension to 4. Action-Slot MO‡ [47] Fig. 5: Visualization of attention maps learned on OATS. Colored masks depict the attention of the activity slots Z4-Z3:C and C2-C1:P+. Note that, while MO successfully predicts the activity C2-C1:P+, the corresponding attention scores are very low. Action-Slot MO‡ [47] Fig. 6: Visualization of attention maps learned on TACO. Colored masks are from the allocated slots of activity Z4-Z1:K and Z1-Z2:K+. Note that Z4-Z1:K appears twice in the 1st st and 5th th frames independently. IV-C Cross-Domain Generalization To assess generalization, we perform transfer learning from TACO to real-world benchmarks, including OATS [11] and a newly annotated nuScenes dataset [28]. We compare pretraining on Kinetics, OATS, and TACO (Table VII), followed by fine-tuning on the respective target datasets. Pretraining on TACO. Due to its large scale and relatively balanced activity distribution, TACO pretraining consistently improves performance across all models on both real-world datasets. Sim-to-Real Transferability. Although TACO is collected in a synthetic environment, it demonstrates stronger cross-domain generalization than OATS. As shown in Table VII, OATS pretraining yields marginal improvements and even degrades performance for X3D, whereas TACO pretraining results in substantial gains across all three categories of models. These findings validate the effectiveness of large-scale, low-cost synthetic data in enhancing sim-to-real transfer for atomic activity recognition. TABLE VII: Comparisons of pretrained representations from OATS and TACO. We perform transfer learning on the real-world OATS and nuScenes dataset. We report the average mAP of the three splits in the OATS dataset. OATS nuScenes Kinetics +TACO Kinetics +OATS +TACO X3D [15] 31.4 34.9 (+3.5) 19.8 18.9 (-0.9) 27.8 (+8.0) ARG [21] 26.7 31.3 (+4.6) 12.2 12.7 (+0.5) 17.0 (+4.8) Action-Slot 48.2 59.5 (+11.3) 23.6 23.6 (+0.0) 32.3 (+8.7) IV-D Structured Decomposition Analysis To verify that Action-Slot truly performs activity-centric decomposition rather than object grouping, we compare against object-level guidance and visualize the differences. Action-Slot vs. Object-level Guidance. We conduct experiments on the TACO dataset to evaluate the effect of varying numbers of road users per video under two attention guidance mechanisms within the Action-Slot framework For object-level guidance, we use instance segmentation collected in TACO to obtain masks for vehicles and pedestrians. Hungarian matcher is used to assign object masks to slots, following MO [47]. As shown in Table VI, Action-Slot with strong ground-truth object-level supervision performs worse than our proposed attention guidance mechanism. This result suggests that relying solely on object-level cues is insufficient for our task, as not all objects participate in the same atomic activity at any given time. Qualitative Results. We visualize the attention maps learned using the re-implemented MO [47] and Action-Slot on the OATS and TACO datasets in Figure 5 and Figure 6, respectively. Action-Slot demonstrates the capability to decompose multiple atomic activities from intricate scenes on both datasets. In Figure 5, Action-Slot localizes the group activity C2-C1:P+, while MO [47] fails to attend any relevant regions. Moreover, Action-Slot recognizes two independent Z4-Z1:K activities, instead of predicting them as a group activity, i.e., Z4-Z1:K+ in Figure 6. The evidence justifies the capability of Action-Slot to decompose structured activities. More interestingly, Action-Slot attends to relevant regions when road users initiate an action and cease attention once the action is completed, as seen in the case of the bicyclists in the fourth frame of Figure 6. IV-E Weakly-Supervised Target Atomic Activity Localization Quality We investigate spatio-temporal localization to examine two central questions: (1) Do the action-centric representations learned by Action-Slot inherently capture spatial-temporal grounding signals suitable for pseudo-mask generation? (2) How to enhance the coarse spatial-temporal cues and enable reliable localization? TABLE VIII: Evaluation of pseudo masks across different generation stages. The source action recognition models are trained exclusively on recognition task. Stage Attention map Source mIoU oIoU tIoU mAP@tIoU Prec. Recall F1 Raw Attention/Grad-CAM X3D [15] 3.6 3.4 0.9 0 6.1 18.8 6.5 CSN [13] 2.0 2.5 0.5 0 2.9 9.3 3.3 Slot-VPS [49] 1.8 2.9 0.6 0 4.8 5.3 3.2 Action-Slot 4.6 5.6 1.7 0 9.1 15.7 8.0 Attention-Object Matching X3D [15] 16.1 21.0 15.7 3.4 22.5 31.4 22.3 CSN [13] 17.7 23.2 17.3 3.8 23.4 36.0 24.2 Slot-VPS [49] 14.6 21.3 14.6 3.5 30.1 19.6 20.1 Action-Slot 23.0 24.0 25.4 9.1 41.1 31.9 30.7 Pseudo Mask Selection X3D [15] 16.9 20.9 16.5 4.1 23.8 32.1 23.0 CSN [13] 17.8 23.3 17.4 3.8 23.3 36.1 24.2 Slot-VPS [49] 18.0 23.1 17.8 3.5 34.2 25.5 24.9 Action-Slot 26.8 29.8 28.8 9.6 47.0 36.9 35.9 IV-E1 Implementation Details To bridge the gap between video-level recognition and pixel-level localization, we leverage foreground object masks generated by Grounded-SAM-2 [79] as spatial proposals used in the matching process described in Sec. I-D1, yielding high-quality pseudo-masks for supervision. We then train the localization model with these pseudo-masks with the segmentation loss ℒL_ seg to fine-tune the entire Action-Slot framework. For the ablation study on attention-object matching in Sec. IV-E7, we implement the baseline pseudo-mask generation algorithms proposed by Chen et al. [59], which converting attention maps into initial binary masks M_ init using Otsu’s thresholding [97]. The baseline originally refine masks by merging superpixels [56]; however, we replace the superpixels with the predicted object masks for a fair comparison and merge those that yield an IoU ≥0.1≥ 0.1 with M_ init. IV-E2 Baselines To demonstrate the spatial-temporal transferability of Action-Slot, we compare the pseudo masks generated by other recognition models, i.e., X3D, CSN, and Slot-VPS. For X3D and CSN, we compute Grad-CAMs [57] from the last layer before the projection head and use them as pseudo masks. The Grad-CAM outputs are trilinearly interpolated to match the spatial-temporal resolution of Action-Slot’s attention maps. For Slot-VPS, we directly adopt its attention maps as pseudo masks. To compare the proposed weakly supervised localization model with the existing approaches, we re-implement the multiple instance learning (MIL) method proposed by Chen et al. [29]. Specifically, we replace the RPN-generated object proposals [98] with object masks generated by Grounded SAM 2 [79] and omit the unsupervised tracking stage. We also re-implement VSCR [30] by incorporating its CAM-aware contrastive loss while replacing the original CAM map with our generated initial pseudo mask. TABLE IX: Performance of localization models trained on different pseudo masks. Compared to existing weakly-supervised action localization baselines. Method mIoU oIoU tIoU mAP@tIoU Precision Recall F1 Chen et al. [29] 15.0 16.1 14.2 2.5 16.8 35.8 20.5 VSCR [30] 22.8 25.6 20.2 5.5 35.4 44.9 32.2 Ours w/o selection 22.7 25.7 20.3 5.7 34.6 45.3 31.2 Ours 24.7 28.1 23.5 6.8 38.9 44.0 34.8 TABLE X: False Positive (FP) rates (%) for absent-action queries across different actor types. A sample is counted as an FP if the number of activated pixels exceeds an actor-specific threshold. Method FP(C/C+) FP(K/K+) FP(P/P+) Ours w/o Selection 53.4 59.2 76.8 Ours w/ Selection 25.2 32.8 49.4 IV-E3 Metrics We evaluate localization performance using mean IoU (mIoU), overall IoU (oIoU), and temporal IoU (tIoU). The mIoU computes the average IoU across actions, while oIoU aggregates all predicted and ground-truth pixels over the dataset before computing IoU, emphasizing larger samples. Since our model outputs spatio-temporal masks rather than per-frame confidence scores, we redefine tIoU using frame-wise spatial IoU as a proxy for confidence. Frames with spatial IoU greater than 0.5 are considered true positives, and tIoU is computed as the ratio of true positive frames to the total number of frames containing either predicted or ground-truth pixels. In addition, we report pixel-level precision, recall, and F1 score to provide a more comprehensive evaluation of mask quality. IV-E4 Pseudo Mask Quality We evaluate the quality of pseudo masks generated by models trained exclusively on the recognition task. As shown in Table VIII, masks derived from raw attention maps and Grad-CAM [57] are highly inaccurate, yielding coarse localization. After applying the attention-to-object matching introduced in Sec. I-D1, all methods exhibit significant improvements across all evaluation metrics. This indicates that existing action recognition models fail to offer fine object localization cues. To further refine these masks leverages the important regions identified through attention differences (described in Sec. I-D2), leading to consistent gains across all metrics. These results validate the effectiveness of our selection strategy in isolating the most relevant spatio-temporal regions associated with a specific action. Notably, at every stage, pseudo masks generated by Action-Slot consistently outperform those from other models, suggesting that its action-centric representation provides crucial action semantics enabling strong spatio-temporal localization. Frame Z3-Z1:C+ Ours w/o Selection Ours Frame Z2-Z1:K+ Ours w/o selection Ours Fig. 7: Qualitative localization results trained with different pseudo masks. Rows from top to bottom: camera frames, ground-truth masks, predictions without pseudo mask selection, and predictions with pseudo mask selection. IV-E5 Localization Model Performance We evaluate our localization model with other weakly-supervised baselines. As shown in Table IX, the MIL-based approach [29] performs the worst across all spatial-temporal metrics. This indicates that MIL loss alone is insufficient for localizing atomic activities under weak supervision. In contrast, incorporating the contrastive loss from VSCR [30] yields only marginal improvements over our base model (22.8% vs. 22.7% mIoU), which is trained on initial pseudo masks without selection. This is because VSCR’s contrastive objective heavily relies on appearance cues, which offer limited discrimination for atomic activities where actors share highly similar, overlapped visual features. Moreover, without a mechanism to filter out spatial noise, pulling these unrefined regions closer in the feature space can mistakenly entangle the true target with background distractors, leaving the model vulnerable to spurious attention responses. Overall, our localization model, explicitly trained using pseudo masks refined by the selection stage, achieves the best performance. This validates the effectiveness of our attention difference selection strategy: by leveraging counterfactual intervention to eliminate spurious attention responses, it yields highly precise pseudo-supervision for downstream localization training. Qualitative examples in Figure 7 illustrate localization results of querying different atomic activities in cluttered traffic scenes with multiple co-occurring activities. Without the selection stage, initial predictions roughly capture the regions of interest but often include irrelevant areas, leading to significant false positives. With the refined pseudo masks fine-tuning, localization becomes more precise, with improved boundary alignment and more effective suppression of irrelevant regions. These improvements are particularly evident in cases where multiple agents perform similar activities. As illustrated in Figure 7, when querying Z2-Z1:K+, the selection stage enables the model to correctly distinguish between two bicycles and a red van driving the same pattern, whereas the initial predictions erroneously include the van in the mask. Such results highlight the advantage of our selection framework in producing sharper and more discriminative localization. IV-E6 Robustness of Absent-Action Queries We assess the false positive rate when querying atomic activity categories that are absent in videos. For each video in the testing set, we randomly sample a query of an absent actor category (e.g., pedestrian). A sample is treated as a false positive if the number of activated pixels is larger than a threshold. To account for varying object sizes, we apply actor-type-specific thresholds on the number of activated pixels. Specifically, we group the actions into three main categories based on the actor type: vehicle (C and C+), two-wheeler (K and K+), and pedestrian (P and P+). The threshold for each actor type is determined by the average number of pixels in a video, calculated from the ground truth annotations. Our results reported in Table X show that the model trained with refined pseudo masks exhibits a substantial reduction in false positives across all actor types, even without any explicit optimization for handling non-present actions. Importantly, this improvement is not simply due to globally suppressing activations during the process of identifying important regions at the selection stage; when querying present activities, the selection stage increases recall and IoU (Table IX), demonstrating that it selectively removes spurious activations while preserving true activity coverage. TABLE XI: Ablation study on attention-object matching strategies and comparison with the overlap-based approach by Chen et al. [59]. Method C C+ K K+ P P+ mIoU Overlap-based [59] 1.5 1.4 0.7 0.5 0.2 0.4 0.9 Distance 31.7 27.3 29.9 22.3 23.6 12.3 25.8 Gravity 41.7 27.4 28.4 24.3 17.7 11.3 27.1 Mixed 41.7 27.4 29.9 22.3 23.6 12.3 28.9 IV-E7 Ablation Study Attention-Object Matching. We ablate the attention-object matching strategy by comparing our mixture matching introduced in Sec. I-D1 with the overlap-based pseudo mask generating algorithm [59]. As shown in Table XI, this threshold-and-overlap paradigm fails drastically in our experiment. The primary cause of this failure is its heavy reliance on spatial overlap with an initial binary mask MinitM_init. In complex driving scenes, applying hard thresholding to continuous attention heatmaps inevitably yields a highly noisy, fragmented, or overly dilated MinitM_init. When the initial mask is severely degraded, the overlap-based matching criterion completely breaks down—yielding near-zero overlap for valid small actors or erroneously merging adjacent distractors. In contrast, our proposed matching metrics bypass this fragile pixel-level overlap. Interestingly, we observe that the Gravity and Distance metrics perform comparably on two-wheelers (K and K+). We attribute this to the intermediate and highly variable relative scale of two-wheelers in real-world traffic: they act as smaller objects when compared to four-wheelers, but as larger objects when compared to pedestrians. This dual nature of their relative scale causes both distance-based and area-weighted (Gravity) metrics to be similarly effective. Overall, by leveraging the robust geometric properties (centers and areas) rather than fragile overlap conditions, our proposed mixture matching approach yields the best average performance across all categories in terms of mIoU and justifies our design choice. TABLE XII: Ablation study on query representations of the atomic activity for localization. Selection denotes whether the mask selection mechanism is applied. Input Selection mIoU oIoU tIoU mAP@tIoU One-hot ✗ 11.7 15.0 7.5 0.8 Slot features 12.7 13.8 7.9 0.9 Attention 22.7 25.7 20.3 5.7 One-hot ✓ 12.4 13.6 7.2 0.7 Slot features 13.7 14.0 9.0 1.1 Attention 24.7 28.1 23.5 6.8 Query Representation. We study the query representation by comparing conventional one-hot class embeddings and the Action-Slot’s representations—class-specified slot features and the corresponding attention. Table XII shows that one-hot embeddings and slot features fail to provide sufficient hints for localization, yielding only 12.4% and 13.7% mIoU, respectively. In contrast, using the attention map significantly improves the performance to 24.7%. This indicates that attention maps, inherently derived from feature-level dot-products, retain much richer spatial cues about the target activity. While slot features encode action semantics, they can barely recover the spatial-temporal cues. Finally, we observe that applying our selection mechanism (detailed in Sec. I-D2) consistently improves the performance across all representations by filtering out irrelevant objects via counterfactual intervention. Fig. 8: Action-Slot attention visualization in a failure case involving occlusion. Distinct mask colors represent attention maps for different atomic activity classes. The yellow bus partially occludes the white car emerging from the right roadway (Z2). Action-Slot correctly predicts Z3-Z1:C and Z2-Z1:C, but fails to detect Z2-Z3:C, which is performed by the occluded white car (indicated by the black arrow). Input Features. We examine whether Action-Slot provides stronger input representations for localization by comparing them with features from video models. As shown in Table XIII, using Action-Slot features yields substantial overall improvements compared to their raw backbone counterparts. This demonstrates that Action-Slot learns more structured, action-centric representations that are significantly more informative for atomic activity localization than standard video backbones. TABLE XIII: Ablation study on input features for localization, comparing raw features from standard video backbones against those enhanced by the Action-Slot. Input Features mIoU oIoU tIoU mAP@tIoU I3D [12] 13.2 15.4 11.5 2.5 I3D w/ Action-Slot 13.8 19.9 12.5 3.5 X3D [15] 14.3 15.2 12.9 2.2 X3D w/ Action-Slot 24.7 28.1 23.5 6.8 V Conclusions In this work, we present a structured action-centric representation learning framework that reformulates slot attention for multi-agent atomic activity understanding. We show that the learned action-centric representations serve as a unified foundation for two complementary aspects of atomic activity understanding: multi-label recognition and spatio-temporal localization. Extensive experiments demonstrate that Action-slot achieves state-of-the-art performance on OATS and the proposed TACO dataset, while significantly improving cross-domain generalization through balanced, topology-aware pretraining. Beyond recognition, we demonstrate that the structured slot representations encode meaningful spatial grounding signals. Leveraging these signals, we develop a weakly supervised localization framework that refines slot attention into high-quality pseudo masks. The resulting model achieves superior localization accuracy and substantially reduces false positives under absent queries, without requiring dense annotations. Overall, our findings suggest that action-centric structured representations provide a principled and transferable foundation for multi-agent video understanding. By bridging recognition and localization within a unified representation paradigm, this work validates the potential of structured decomposition as a scalable alternative to annotation-intensive approaches for complex dynamic scenes. More broadly, this unified perspective opens a new direction for structured video representation learning, where semantically aligned decomposition enables scalable recognition and localization under weak supervision. Limitation. Although Action-Slot demonstrates favorable quantitative and qualitative performance, we observe that it is still challenging to handle cases when two activities occlude with each other, as shown in Figure 8. In these cases, the corresponding action slots may be confused about where they should attend. This observation is closely relevant to the tracking task under occlusion, where frequent ID switches are often observed. Moreover, the evaluation of atomic activity localization still relies heavily on manually labeled data. To advance the field, we invite the community to explore automatic or semi-automatic labeling strategies that can enable more comprehensive and efficient evaluation. References [1] A. Jain, H. S. Koppula, B. Raghavan, S. Soh, and A. Saxena (2015) Car that Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models. In IEEE International Conference on Computer Vision (ICCV), Cited by: §I. [2] B. Liu, E. Adeli, Z. Cao, K. Lee, A. Shenoi, A. Gaidon, and J. C. Niebles (20202020) Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction. In IEEE Robotics and Automation Letters, Vol. 5, p. 3485–3492. Cited by: §I, §I-E. [3] Z. Xiao, A. Yuille, and Y. Chen (2022) Learning Road Scene-level Representations via Semantic Region Prediction. In Conference on Robot Learning (CoRL), Cited by: §I, §I-E. [4] D. Lin, S. Fidler, C. Kong, and R. Urtasun (2014) Visual Semantic Search: Retrieving Videos via Complex Textual Queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I, §I-E. [5] A. Taha, Y. Chen, T. Misu, A. Shrivastava, and L. Davis (2020) Boosting Standard Classification Architectures Through a Ranking Regularizer. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Cited by: §I, §I-E. [6] S. Segal, E. Kee, W. Luo, A. Sadat, E. Yumer, and R. Urtasun (2020) Universal Embeddings for Spatio-Temporal Tagging of Self-Driving Logs. In Conference on Robot Learning (CoRL), Cited by: §I, §I-E. [7] M. Naphade, S. Wang, D. C. Anastasiu, Z. Tang, M. Chang, Y. Yao, L. Zheng, M. S. Rahman, M. S. Arya, A. Sharma, Q. Feng, V. Ablavsky, S. Sclaroff, P. Chakraborty, S. Prajapati, A. Li, S. Li, K. Kunadharaju, S. Jiang, and R. Chellappa (2023) The 7th AI City Challenge. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Cited by: §I, §I-E. [8] (2022) CARLA Autonomous Driving Challenge. Note: https://carlachallenge.org/ Cited by: §I. [9] C. Xu, W. Ding, W. Lyu, Z. Liu, S. Wang, Y. He, H. Hu, D. Zhao, and B. Li (2022) SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehicles. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, Cited by: §I, §I-E. [10] C. Kung, C. Yang, P. Pao, S. Lu, P. Chen, H. Lu, and Y. Chen (2024) RiskBench: a scenario-based benchmark for risk identification. IEEE Conference on Robotics and Automation (ICRA). Cited by: §I, item (b), item (b), §IV-A1. [11] N. Agarwal and Y. Chen (2023) Ordered Atomic Activity for Fine-grained Interactive Traffic Scenario Understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 8624–8636. Cited by: §I, §I, §I, §I-A, §I-E, TABLE I, §IV-A2, §IV-B1, §IV-B1, §IV-B2, §IV-B3, §IV-B3, §IV-C, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I, TABLE I. [12] J. Carreira and A. Zisserman (2017) Quo vadis, Action Recognition? A New Model and the Kinetics Dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 6299–6308. Cited by: §I, §I-A, §I-C, §IV-B1, §IV-B1, §IV-B4, TABLE XIII, TABLE I, TABLE I, TABLE I. [13] D. Tran, H. Wang, L. Torresani, and M. Feiszli (2019) Video Classification with Channel-separated Convolutional Networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 5552–5561. Cited by: §I, §I-A, §IV-B1, TABLE I, TABLE I, TABLE I, TABLE VIII, TABLE VIII, TABLE VIII. [14] C. Feichtenhofer, H. Fan, J. Malik, and K. He (2019) SlowFast Networks for Video Recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 6202–6211. Cited by: §I, §I-A, §I-C, §IV-B1, §IV-B1, §IV-B4, TABLE I, TABLE I, TABLE I, TABLE I, TABLE VI. [15] C. Feichtenhofer (2020) X3D: Expanding Architectures for Efficient Video Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 203–213. Cited by: §I, §I-A, §I-C, §IV-B1, §IV-B1, §IV-B1, §IV-B4, TABLE XIII, TABLE I, TABLE I, TABLE VI, TABLE VII, TABLE VIII, TABLE VIII, TABLE VIII. [16] H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer (2021) Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 6824–6835. Cited by: §I, §I-A, §I-C, §IV-B1, §IV-B3, TABLE I. [17] Z. Tong, Y. Song, J. Wang, and L. Wang (2022) VideoMAE: masked autoencoders are data-efficient learners for self-supervised video pre-training. In Advances in Neural Information Processing Systems, Cited by: §I, §I-A, §I-C, §IV-B1, §IV-B3, TABLE I. [18] J. Zhao, Y. Zhang, X. Li, H. Chen, B. Shuai, M. Xu, C. Liu, K. Kundu, Y. Xiong, D. Modolo, et al. (2022) Tuber: Tubelet Transformer for Video Action Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 13598–13607. Cited by: §I. [19] A. A. Gritsenko, X. Xiong, J. Djolonga, M. Dehghani, C. Sun, M. Lucic, C. Schmid, and A. Arnab (2024) End-to-End Spatio-Temporal Action Localisation with Video Transformers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 18373–18383. Cited by: §I. [20] F. Baradel, N. Neverova, C. Wolf, J. Mille, and G. Mori (2018) Object Level Visual Reasoning in Videos. In Proceedings of the European Conference on Computer Vision (ECCV), Cited by: §I, §I-A, §IV-B1, §IV-B3, TABLE I, TABLE I, TABLE I. [21] J. Wu, L. Wang, L. Wang, J. Guo, and G. Wu (2019) Learning Actor Relation Graphs for Group Activity Recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Cited by: §I, §I-A, §IV-B1, §IV-B3, TABLE I, TABLE I, TABLE I, TABLE VII. [22] J. Pan, S. Chen, M. Z. Shou, Y. Liu, J. Shao, and H. Li (2021) Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 464–474. Cited by: §I, §I-A. [23] A. Arnab, X. Xiong, A. Gritsenko, R. Romijnders, J. Djolonga, M. Dehghani, C. Sun, M. Lučić, and C. Schmid (2022) Beyond Transfer Learning: Co-finetuning for Action Localisation. arXiv preprint arXiv:2207.03807. Cited by: §I, §I-A. [24] K. He, G. Gkioxari, P. Dollár, and R. Girshick (2017) Mask R-CNN. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 2961–2969. Cited by: §I, §IV-B1. [25] R. Girshick (2015) Fast r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), p. 1440–1448. Cited by: §I. [26] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf (2020) Object-centric Learning with Slot Attention. Advances in Neural Information Processing Systems (NeurIPS) 33, p. 11525–11538. Cited by: §I, §I-B, §I-B, §I-A, §I-A, §I-C1. [27] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun (2017) CARLA: An Open Urban Driving Simulator. In Proceedings of the 1st Annual Conference on Robot Learning (CoRL), p. 1–16. Cited by: §I, §I-E, item (c), §IV-A1. [28] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom (2020) nuscenes: A Multimodal Dataset for Autonomous Driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I, TABLE I, §IV-A3, §IV-C. [29] W. Chen, G. Li, X. Zhang, S. Wang, L. Li, and Q. Huang (2023) Weakly Supervised Text-based Actor-Action Video Segmentation by Clip-level Multi-instance Learning. ACM Transactions on Multimedia Computing, Communications and Applications 19 (1), p. 1–22. Cited by: §I, §I-D, §IV-E2, §IV-E5, TABLE IX. [30] B. Duan, H. Tang, C. Sun, Y. Zhu, and Y. Yan (2024) Mining and Unifying Heterogeneous Contrastive Relations for Weakly-Supervised Actor-Action Segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 494–503. Cited by: §I, §I-C, §IV-E2, §IV-E5, TABLE IX. [31] C. Kung, S. Lu, Y. Tsai, and Y. Chen (2024) Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I, §I-C. [32] G. Bertasius, H. Wang, and L. Torresani (2021) Is space-time attention all you need for video understanding?. In International Conference on Machine Learning (ICML), Cited by: §I-A, §I-C. [33] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid (2021) ViViT: A Video Vision Transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 6836–6846. Cited by: §I-A, §I-C. [34] C. Kung, F. Ramirez, J. Ha, Y. Chen, D. Crandall, and Y. Tsai (2025) What changed and what could have changed? state-change counterfactuals for procedure-aware video representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Cited by: §I-A. [35] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, et al. (2017) The Kinetics Human Action Video Dataset. arXiv preprint arXiv:1705.06950. Cited by: §I-A, §IV-B1, §IV-B3. [36] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre (2011) HMDB: A Large Video Database for Human Motion Recognition. In 2011 International Conference on Computer Vision (ICCV), p. 2556–2563. Cited by: §I-A. [37] Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar (2014) THUMOS Challenge: Action Recognition with a Large Number of Classes. Note: http://crcv.ucf.edu/THUMOS14/ Cited by: §I-A. [38] G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta (2016) Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding. In Proceedings of the European Conference on Computer Vision (ECCV), p. 510–526. Cited by: §I-A, §IV-B2. [39] S. Yeung, O. Russakovsky, N. Jin, M. Andriluka, G. Mori, and L. Fei-Fei (2017) Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos. International Journal of Computer Vision (IJCV) 126 (2–4), p. 375–389. Cited by: §I-A, §IV-B2. [40] C. Gu, C. Sun, D. A. Ross, C. Vondrick, C. Pantofaru, Y. Li, S. Vijayanarasimhan, G. Toderici, S. Ricco, R. Sukthankar, et al. (2018) AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 6047–6056. Cited by: §I-A, §IV-B2. [41] W. Choi, K. Shahid, and S. Savarese (2009) What are they doing?: collective activity classification using spatio-temporal relationship among people. In IEEE International Conference on Computer Vision Workshops (ICCV-W), p. 1282–1289. Cited by: §I-A. [42] M. S. Ibrahim, S. Muralidharan, Z. Deng, A. Vahdat, and G. Mori (2016) A hierarchical deep temporal model for group activity recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 1971–1980. Cited by: §I-A. [43] J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick (2017) CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 2901–2910. Cited by: §I-B. [44] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. (2020) Scalability in Perception for Autonomous Driving: Waymo Open Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I-B. [45] A. Geiger, P. Lenz, and R. Urtasun (2012) Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I-B. [46] T. Kipf, G. F. Elsayed, A. Mahendran, A. Stone, S. Sabour, G. Heigold, R. Jonschkowski, A. Dosovitskiy, and K. Greff (2022) Conditional Object-Centric Learning from Video. In International Conference on Learning Representations (ICLR), Cited by: §I-B, §I-B, §I-A, §I-B2, §I-C1, §I-C2, §IV-B1, TABLE I, TABLE I, TABLE I. [47] Z. Bao, P. Tokmakov, A. Jabri, Y. Wang, A. Gaidon, and M. Hebert (2022) Discovering Objects that Can Move. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11789–11798. Cited by: §I-B, §I-B, §I-A, §I-B2, §I-C2, §I-C, Fig. 5, Fig. 6, §IV-B1, §IV-B4, §IV-D, §IV-D, TABLE I, TABLE I, TABLE I. [48] G. Elsayed, A. Mahendran, S. van Steenkiste, K. Greff, M. C. Mozer, and T. Kipf (2022) SAVi++: Towards End-to-End Object-centric Learning from Real-world Videos. Conference on Neural Information Processing Systems (NeurIPS). Cited by: §I-B, §I-B, §I-A, §I-B2, §I-C1, §I-C2, §IV-B1, TABLE I, TABLE I, TABLE I. [49] Y. Zhou, H. Zhang, H. Lee, S. Sun, P. Li, Y. Zhu, B. Yoo, X. Qi, and J. Han (2022) Slot-VPS: Object-centric Representation Learning for Video Panoptic Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 3093–3103. Cited by: §I-B, §I-C1, §IV-B1, §IV-B4, TABLE I, TABLE I, TABLE I, TABLE VIII, TABLE VIII, TABLE VIII. [50] M. S. M. Sajjadi, D. Duckworth, A. Mahendran, S. van Steenkiste, F. Pavetic, M. Lucic, L. J. Guibas, K. Greff, and T. Kipf (2022) Object Scene Representation Transformer. In Advances in Neural Information Processing Systems, p. 9512–9524. Cited by: §I-B. [51] O. Biza, S. van Steenkiste, M. S. Sajjadi, G. F. Elsayed, A. Mahendran, and T. Kipf (2023) Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames. In International Conference on Machine Learning (ICML), Cited by: §I-B. [52] J. Ahn and S. Kwak (2018) Learning Pixel-Level Semantic Affinity with Image-Level Supervision for Weakly Supervised Semantic Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 4981–4990. Cited by: §I-C. [53] L. Chen, W. Wu, C. Fu, X. Han, and Y. Zhang (2020) Weakly Supervised Semantic Segmentation with Boundary Exploration. In European Conference on Computer Vision (ECCV), p. 347–362. Cited by: §I-C. [54] T. Wu, J. Huang, G. Gao, X. Wei, X. Wei, X. Luo, and C. H. Liu (2021) Embedded Discriminative Attention Mechanism for Weakly Supervised Semantic Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 16765–16774. Cited by: §I-C. [55] K. Sun, H. Shi, Z. Zhang, and Y. Huang (2021) ECS-Net: Improving Weakly Supervised Semantic Segmentation by Using Connections Between Class Activation Maps. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 7283–7292. Cited by: §I-C. [56] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk (2012) SLIC Superpixels Compared to State-Of-The-Art Superpixel Methods. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) 34 (11), p. 2274–2282. Cited by: §I-C, §IV-E1. [57] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra (2017) Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), p. 618–626. Cited by: §I-C, §IV-E2, §IV-E4. [58] Y. Yan, C. Xu, D. Cai, and J. J. Corso (2017) Weakly Supervised Actor-Action Segmentation via Robust Multi-Task Ranking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 1298–1307. Cited by: §I-C. [59] J. Chen, Z. Li, J. Luo, and C. Xu (2020) Learning a weakly-supervised video actor-action segmentation model with a wise selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 9901–9911. Cited by: §I-C, §IV-E1, §IV-E7, TABLE XI, TABLE XI. [60] C. Xu, S. Hsieh, C. Xiong, and J. J. Corso (2015) Can humans fly? action understanding with multiple classes of actors. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 2264–2273. Cited by: §I-C. [61] C. Chen, W. Choi, and M. Chandraker (2016) Atomic Scenes for Scalable Traffic Scene Recognition in Monocular Videos. In IEEE Winter Conference on Applications of Computer Vision (WACV), p. 1–9. Cited by: §I-E, TABLE I. [62] H. Girase, H. Gang, S. Malla, J. Li, A. Kanehara, K. Mangalam, and C. Choi (2021) LOKI: Long Term and Key Intentions for Trajectory Prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 9803–9812. Cited by: §I-E, TABLE I. [63] G. Singh, S. Akrigg, M. Di Maio, V. Fontana, R. J. Alitappeh, S. Khan, S. Saha, K. Jeddisaravi, F. Yousefi, J. Culley, et al. (2022) ROAD: The ROad event Awareness Dataset for Autonomous Driving. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) 45 (1), p. 1036–1054. Cited by: §I-E, TABLE I. [64] S. Khan, I. Teeti, R. J. Alitappeh, M. C. Stoian, E. Giunchiglia, G. Singh, A. Bradley, and F. Cuzzolin (2024) ROAD-Waymo: Action Awareness at Scale for Autonomous Driving. arXiv preprint arXiv:2411.01683. Cited by: TABLE I. [65] Z. Chen, H. Wu, C. Kung, Y. Chen, and Y. Peng (2025) ATARS: An Aerial Traffic Atomic Activity Recognition and Temporal Segmentation Dataset. arXiv preprint arXiv:2503.18553. Cited by: TABLE I. [66] H. Yu and J. Siskind (2017) ISentence Directed Video Object Codiscovery. International Journal of Computer Vision (IJCV) (3), p. 312–334. Cited by: §I-D. [67] D. Huang*, S. Buch*, L. Dery, A. Garg, L. Fei-Fei, and J. C. Niebles (2018) Finding “It”: Weakly-Supervised, Reference-Aware Visual Grounding in Instructional Videos. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I-D. [68] L. Zhou, N. Louis, and J. J. Corso (2018) Weakly-supervised video object grounding from text by loss weighting and object interaction. British Machine Vision Conference (BMVC). Cited by: §I-D. [69] J. Shi, J. Xu, B. Gong, and C. Xu (2019) Not all frames are equal: weakly-supervised video grounding with contextual similarity and visual clustering losses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 10444–10452. Cited by: §I-D. [70] W. Wang, J. Gao, and C. Xu (2022) Weakly-Supervised Video Object Grounding via Causal Intervention. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) 45 (3), p. 3933–3948. Cited by: §I-D. [71] V. Ramanishka, Y. Chen, T. Misu, and K. Saenko (2018) Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 7699–7707. Cited by: §I-E. [72] D. Rempe, J. Philion, L. J. Guibas, S. Fidler, and O. Litany (2022) Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior. In Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I-E. [73] Y. Cao, C. Xiao, A. Anandkumar, D. Xu, and M. Pavone (2022) AdvDO: Realistic Adversarial Attacks for Trajectory Prediction. In European Conference on Computer Vision (ECCV), Cited by: §I-E. [74] D. Rempe, Z. Luo, X. B. Peng, Y. Yuan, K. Kitani, K. Kreis, S. Fidler, and O. Litany (2023) Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion. In Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §I-E. [75] P. Chen, C. Kung, C. Chang, W. Chiu, and Y. Chen (2026) Controllable Collision Scenario Generation via Collision Pattern Prediction. IEEE International Conference on Robotics and Automation (ICRA). Cited by: §I-E. [76] M. G. Li, B. Jiang, Z. Che, X. Shi, M. Liu, Y. Meng, J. Ye, and Y. Liu (2019) DBUS: Human Driving Behavior Understanding System. In ICCV Workshops, p. 2436–2444. Cited by: §I-E. [77] S. Malla, B. Dariush, and C. Choi (2020) TITAN: Future Forecast using Action Priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 11186–11196. Cited by: §I-E. [78] Z. Chen, L. Zhang, P. Hu, H. Lu, and Y. He (2024) MaskTrack: Auto-Labeling and Stable Tracking for Video Object Segmentation. IEEE Transactions on Neural Networks and Learning Systems. Cited by: §I-E. [79] N. Ravi, V. Gabeur, Y. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C. Wu, R. Girshick, P. Dollár, and C. Feichtenhofer (2024) SAM 2: segment anything in images and videos. External Links: 2408.00714, Link Cited by: §I-E, §IV-E1, §IV-E2. [80] K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio (2014) Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: §I-A. [81] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. Conference on Neural Information Processing Systems 30. Cited by: §I-A, §IV-B1. [82] K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 770–778. Cited by: §I-C, §IV-B1, §IV-B1. [83] S. Zhou, C. Li, K. C. Chan, and C. C. Loy (2023) ProPainter: improving propagation and transformer for video inpainting. In Proceedings of IEEE International Conference on Computer Vision (ICCV), Cited by: §I-D2. [84] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar (2022) Masked-Attention Mask Transformer for Universal Image Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 1290–1299. Cited by: §I-D3. [85] C. H. Sudre, W. Li, T. Vercauteren, S. Ourselin, and M. Jorge Cardoso (2017) Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In International Workshop on Deep Learning in Medical Image Analysis, p. 240–248. Cited by: §I-D3. [86] (2023) ScenarioRunner for CARLA. Note: https://github.com/carla-simulator/scenario_runner Cited by: item (c), §IV-A1. [87] C. Kung, S. Lu, Y. Tsai, and Y. Chen (2024) Supplementary material action-slot: visual action-centric representations for multi-label atomic activity recognition in traffic scenes. Cited by: §IV-A1. [88] H. Fan, T. Murrell, H. Wang, K. V. Alwala, Y. Li, Y. Li, B. Xiong, N. Ravi, M. Li, H. Yang, J. Malik, R. Girshick, M. Feiszli, A. Adcock, W. Lo, and C. Feichtenhofer (2021) PyTorchVideo: A Deep Learning Library for Video Understanding. In Proceedings of the 29th ACM International Conference on Multimedia, Note: https://pytorchvideo.org/ Cited by: §IV-B1. [89] G. Maggiolino, A. Ahmad, J. Cao, and K. Kitani (2023) Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification. arXiv preprint arXiv:2302.11813. Cited by: §IV-B1. [90] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko (2020) End-to-end Object Detection with Transformers. In European Conference on Computer Vision (ECCV), p. 213–229. Cited by: §IV-B1. [91] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) Imagenet: A Large-scale Hierarchical Image Database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 248–255. Cited by: §IV-B1. [92] L. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), p. 801–818. Cited by: §IV-B2. [93] C. Yang, Y. Xu, J. Shi, B. Dai, and B. Zhou (2020) Temporal Pyramid Network for Action Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 591–600. Cited by: TABLE I. [94] X. Wang, R. Girshick, A. Gupta, and K. He (2018) Non-local Neural Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 7794–7803. Cited by: TABLE I. [95] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021) An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations (ICLR), Cited by: §IV-B3. [96] T. N. Kipf and M. Welling (2017) Semi-supervised Classification with Graph Convolutional Networks. International Conference on Learning Representations (ICLR). Cited by: §IV-B3. [97] N. Otsu et al. (1975) A threshold selection method from gray-level histograms. Automatica 11 (285-296), p. 23–27. Cited by: §IV-E1. [98] S. Ren, K. He, R. Girshick, and J. Sun (2015) Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems (NeurIPS) 28. Cited by: §IV-E2.