Paper deep dive
De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shift
Mengyuan Liu, Yuhang Wen, Yi Zhang, Songtao Wu, Hong Liu, Junsong Yuan, Beichen Ding
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Skeleton sequences can represent both individual actions and multi-entity interactions, encompassing human bodies, hands, objects, and robots. Existing approaches to recognize skeleton-based actions and interactions usually adopt a late fusion strategy, which expects individuals are independent and identically distributed to train a robust weight-shared entity encoder. However, observed entity bias in various skeletal data violates this assumption, leading to suboptimal optimization of backbone models that might produce wrong recognition results. This bias arises from the world coordinate system's initial configuration, where the choice of origin often creates bias in representation. To this end, we propose a Convex Hull Adaptive Shift based normalization method to reduce Entity bias (CHASE), improving performance across a variety of skeleton-based action and interaction recognition tasks. To adaptively apply plausible shifts to the input skeletons, we formulate a plug-and-play parameterized network that ensures the relocated world origin lies within the skeleton convex hull, which avoids non-convergence by limiting the search space. To further minimize entity bias, we incorporate an auxiliary objective that leverages pair-wise distribution distances to guide network optimization. To support both single- and multi-entity actions, we propose a sub-entity strategy that offers a consistent formulation for both scenarios. Moreover, CHASE demonstrates compatibility with various intra-skeleton modalities, such as bones and velocities, highlighting its adaptability. Essentially, our method works as a normalization approach to reduce entity bias, enabling subsequent classifiers to achieve improved recognition performance across diverse settings. Extensive experiments on 7 datasets verify our approach by seamlessly integrating with various backbones and significantly boosting their performance.
Tags
Links
- Source: https://arxiv.org/abs/2609.32454v1
- Canonical: https://arxiv.org/abs/2609.32454v1
Trouble viewing inline? Open PDF directly →
Full Text
94,489 characters extracted from source content.
Expand or collapse full text
De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shift Mengyuan Liu Affiliation: State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China Yuhang Wen Email: wenyh29@mail2.sysu.edu.cn Affiliation: School of Intelligent Systems Engineering, Sun Yat-sen University, China Yi Zhang Affiliation: City University of Hong Kong Songtao Wu Affiliation: R&D Center, Sony China Ltd., China Hong Liu Affiliation: State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China Junsong Yuan Affiliation: University at Buffalo SUNY, USA Beichen Ding Email: dingbch@mail.sysu.edu.cn Affiliation: School of Advanced Manufacturing & Southern Marine Science and Engineering Guangdong Laboratory (Zhuhai), Sun Yat-sen University, China Abstract Skeleton sequences can represent both individual actions and multi-entity interactions, encompassing human bodies, hands, objects, and robots. Existing approaches to recognize skeleton-based actions and interactions usually adopt a late fusion strategy, which expects individuals are independent and identically distributed to train a robust weight-shared entity encoder. However, observed entity bias in various skeletal data violates this assumption, leading to suboptimal optimization of backbone models that might produce wrong recognition results. This bias arises from the world coordinate system’s initial configuration, where the choice of origin often creates bias in representation. To this end, we propose a Convex Hull Adaptive Shift based normalization method to reduce Entity bias (CHASE), improving performance across a variety of skeleton-based action and interaction recognition tasks. To adaptively apply plausible shifts to the input skeletons, we formulate a plug-and-play parameterized network that ensures the relocated world origin lies within the skeleton convex hull, which avoids non-convergence by limiting the search space. To further minimize entity bias, we incorporate an auxiliary objective that leverages pair-wise distribution distances to guide network optimization. To support both single- and multi-entity actions, we propose a sub-entity strategy that offers a consistent formulation for both scenarios. Moreover, CHASE demonstrates compatibility with various intra-skeleton modalities, such as bones and velocities, highlighting its adaptability. Essentially, our method works as a normalization approach to reduce entity bias, enabling subsequent classifiers to achieve improved recognition performance across diverse settings. Extensive experiments on seven datasets, including NTU RGB+D, NTU RGB+D 120, H2O, Assembly101, Collective Activity, Volleyball, and HARPER, consistently verify our approach by seamlessly integrating with various backbones and significantly boosting their performance. Our code is publicly available at https://github.com/Necolizer/CHASE. keywordsAction Recognition, Interactive Action, Skeleton †equal-contributors: These authors contributed equally to this work.†equal-contributors: These authors contributed equally to this work. 1 Introduction Skeleton-based action recognition has been studied extensively for individual subjects, and many well-performing models exist for single-person action datasets (Reilly and Das, 2024; Rajasegaran et al., 2023; Li et al., 2023b). Extending these models to multi-entity settings is more difficult, as interactions can occur between human bodies (Wang et al., 2014; Liu et al., 2017a), hands (Sener et al., 2022; Ohkawa et al., 2023), objects (Kwon et al., 2021; Garcia-Hernando et al., 2018), and robots (Avogaro et al., 2024; Sampieri et al., 2022), each requiring the model to jointly reason over multiple entities with different roles. This breadth of scenarios supports applications in scene understanding (Ren et al., 2024; Jiang et al., 2024; Plizzari et al., 2024), human foundation models (Tang et al., 2023; Ci et al., 2023; Khirodkar et al., 2024; Wang et al., 2025), and human-robot collaboration (Du et al., 2024; Jahangard et al., 2024). Skeletal representations offer a compact and appearance-invariant encoding of spatiotemporal pose (Yang et al., 2024a; Wanyan et al., 2025; Yang et al., 2024b), which makes them a natural choice for this task. This paper addresses a data-level problem that has so far received little attention in the literature: the raw skeletal data for multi-entity actions carries inherent distributional asymmetries between entities, and these asymmetries systematically degrade the optimization of standard backbone models before any architectural choice is made. The predominant strategy for handling multiple entities in skeleton-based recognition is late fusion (Chen et al., 2021; Chi et al., 2022; Lee et al., 2023; Shi et al., 2020; Qiu et al., 2023; Myung et al., 2024). A weight-shared backbone independently encodes each entity, and the resulting feature vectors are averaged to produce the final representation. This approach is convenient because it reuses existing single-entity models without modification. It rests on the assumption that all entities are independently and identically distributed (IID), so that a single shared encoder can be expected to perform equally well on each of them. As we show in this paper, this assumption is frequently and substantially violated in practice. Figure 1: Overview. (a) Existing methods train their models on raw skeletal data with inherent entity bias, resulting in unsatisfactory performance. (b) Our proposed CHASE works as a normalization method to de-bias the skeleton sequences. (c) It can significantly reduce the inter-entity distribution discrepancy and improve recognition performance. (d) CHASE help backbones achieve state-of-the-art results across a variety of action and interaction learning tasks. We identify a fundamental issue in skeletal data for multi-entity actions, which we term Entity Bias. Entity bias refers to significant distributional discrepancies between skeleton sequences of individuals with different entity indices, a characteristic that is particularly evident in raw skeleton data. This bias originates from the initial configuration of the world coordinate system, where the choice of origin introduces inherent asymmetries into the representation. A representative example is the standard preprocessing of NTU RGB+D 120 (Liu et al., 2020a), one of the most widely used benchmarks. The coordinate origin is conventionally placed at the spine-base joint (pelvis) of the first person, a choice motivated by its stability compared to extremity joints. This convention works well for single-entity recognition. In multi-entity settings, however, it systematically favors the first entity: the first person is always centered at the origin while subsequent persons are spatially displaced, and their skeletal distributions diverge by amounts proportional to their physical separation from the reference entity. More broadly, any fixed choice of reference joint or reference entity during data collection or preprocessing will produce the same kind of asymmetry. As illustrated in Fig. 1 (a), entities with different indices are visualized in blue and orange, with their respective distributions (based on 10410^4 mutual action samples from NTU120 (Liu et al., 2020a)) projected onto a 2D plane. The results reveal notable discrepancies in both the mean and covariance of these two entity distributions. The late fusion strategy relies on the IID assumption to train a robust weight-shared classifier, and entity bias directly violates this assumption, resulting in suboptimal recognition performance. This raises the question: Is there a way to mitigate entity bias in skeleton sequences? To address the above problem, we propose CHASE, a Convex Hull Adaptive Shift based normalization method that reduces entity bias across a wide range of skeleton-based action and interaction recognition tasks. The core idea is to adaptively relocate skeleton sequences so that their per-entity distributions become approximately IID, as illustrated in Fig. 1 (b). CHASE consists of four components. A learnable plug-and-play module applies sample-adaptive shifts to the input skeletons, with the relocated coordinate origin constrained to lie within the skeleton convex hull. An auxiliary objective based on pairwise distribution distances guides the network toward distributional alignment during training. A sub-entity strategy extends CHASE to single-entity settings by decomposing one entity into structurally comparable sub-entities, so the same distribution alignment objective applies consistently. A modality-agnostic formulation ensures that CHASE operates on any skeletal representation without requiring modality-specific design choices. In essence, CHASE works as a normalization step that produces de-biased, nearly IID skeleton samples, improving the optimization of subsequent weight-shared backbone models. As shown in Fig. 1 (c)(d), experiments on 7 datasets confirm consistent improvements across diverse interaction types and backbone architectures. Our main contributions are three-fold: • The observed entity bias in skeleton sequences undermines the effectiveness of weight-shared backbones and results in suboptimal performance. To the best of our knowledge, we are the first to investigate the issue of the entity bias. Our main idea is to address this bias by adaptively re-centering the coordinate system for each sample, thereby significantly improving the performance of the subsequent backbone models. • We proposed CHASE, a novel de-biasing method based on convex hull adaptive shift, serving as an additional normalization step for the subsequent backbone. Specifically, CHASE consists of a learnable network and an auxiliary objective, which learns origin offsets within the skeleton convex hull and guides the bias minimization. • Extensive experiments on 7 diverse datasets consistently verify our proposed method by improving single-entity backbones to achieve state-of-the-art performance across a variety of recognition tasks. This paper is an extension of our conference paper (Wen et al., 2024). The new contributions of this paper include: • In (Wen et al., 2024), we formulated CHASE for multi-entity actions. Compared to the previous work, in this paper, we propose the Sub-Entity Strategy to extend CHASE to any-entity skeleton sequences, which includes both single- and multi-entity actions. This strategy offers a unified approach by conceptualizing a single entity as a collection of sub-entities derived from its components. Specifically, we define the criteria for selecting sub-entities, incorporating both subgraph and isomorphism conditions. Additionally, we illustrate through examples how sub-entities can be identified using heuristic methods, such as leveraging the physical topology of the entity graph in the skeleton sequences. • In (Wen et al., 2024), we only applied CHASE to joint modalities. Compared to the previous work, in this paper, we extend CHASE to accommodate diverse widely-used skeletal representations, including joints, bones, velocities, and JBF (Myung et al., 2024). This extension involves reformulating the learnable network, auxiliary objectives, and the sub-entity strategy for these modalities, thereby demonstrating the adaptability of CHASE in reducing bias and improving performance across diverse intra-skeleton modalities. • In (Wen et al., 2024), we conducted experiments with 4 backbones and 6 benchmarks, including NTU RGB+D (Shahroudy et al., 2016), NTU RGB+D 120 (Liu et al., 2020a), H2O (Kwon et al., 2021), Assembly101 (Sener et al., 2022), Collective Activity (Choi et al., 2009), and Volleyball (Ibrahim et al., 2016). Compared to the previous work, in this paper, we evaluate our method with more recently-proposed baseline backbones and also in genuinely new scenarios. In particular, we include HARPER (Avogaro et al., 2024), a human-robot interaction benchmark that introduces a fundamentally different skeleton topology: human and quadruped robot skeletons have distinct joint structures and dramatically different morphologies, making the entity bias both more severe and structurally heterogeneous than the human-human settings in prior work. CHASE achieves a significant improvement on HARPER, demonstrating its applicability in this challenging new domain where existing approaches are especially ill-suited. In addition, we provide extensive comparisons against more state-of-the-art methods. Further experiments are conducted to show that the proposed method can be effectively applied to single-entity actions and various skeletal representations. Notably, in NTU RGB+D and NTU RGB+D 120 benchmarks, CHASE significantly improves the ensemble accuracy of the state-of-the-art DeGCN (Myung et al., 2024). 2 Related Work 2.1 Distribution Alignment and Normalization Reducing distributional discrepancies between different subpopulations or domains is a long-studied problem. Domain adaptation methods align feature distributions across domains through statistical divergence minimization (Long et al., 2015) or adversarial training (Ganin et al., 2016). These methods are designed for cross-domain settings where the shift occurs between distinct datasets or collection conditions and where domain labels are explicitly available. Normalization layers address a complementary problem: stabilizing optimization by reducing internal covariate shift during training. Batch Normalization (Ioffe and Szegedy, 2015b) standardizes activations over each mini-batch. Instance Normalization (Ulyanov et al., 2016) operates per sample, Layer Normalization (Ba et al., 2016) per feature dimension, and Group Normalization (Wu and He, 2018) over channel groups to handle settings where batch statistics are unreliable. These techniques operate on intermediate feature representations and are independent of the geometry of the input data. The entity bias examined in this paper sits outside both of these frameworks. It arises in the spatial coordinate representation of skeleton sequences at the input level, prior to any feature extraction. It is not a cross-domain shift but a within-dataset asymmetry produced by a fixed convention for choosing a coordinate origin during data preprocessing. Standard feature normalization does not address this spatial asymmetry, and domain adaptation methods require explicit domain partitions that are unavailable in multi-entity interaction settings. CHASE is designed to close this gap. It learns a sample-adaptive spatial shift in the input space, constrained geometrically within the skeleton convex hull, and guided during training by a distribution alignment objective that operates over entities within each mini-batch. 2.2 Skeleton-based Action Recognition Models. A significant body of work focuses on designing neural network architectures to improve skeleton-based action recognition. Early approaches primarily relied on recurrent architectures to capture long-term temporal contexts (Liu et al., 2016; Zhu et al., 2016; Liu et al., 2017b; Zhang et al., 2017; Liu et al., 2018; Zhang et al., 2019). Subsequently, Graph Convolution Network (GCN)-based methods emerged as the dominant paradigm and remain central to this field today (Yan et al., 2018; Li et al., 2019; Shi et al., 2019; Liu et al., 2020b; Chen et al., 2021; Chi et al., 2022; Lee et al., 2023; Hao et al., 2021; Duan et al., 2022; Liu et al., 2023; Zhou et al., 2024; Zhang et al., 2024; Myung et al., 2024; Xie et al., 2025). To address specific challenges, various enhancements to GCNs have been proposed. InfoGCN (Chi et al., 2022) employs a novel objective to learn compact latent representations. BlockGCN (Zhou et al., 2024) proposes an efficient refinement to graph convolutions to mitigate redundancy issues. DeGCN (Myung et al., 2024) introduces deformable sampling locations on spatiotemporal graphs, improving the perception of discriminative receptive fields. More recently, self-attention mechanisms have been incorporated into spatiotemporal modeling for skeletons, offering new perspectives on sequence encoding and interaction modeling (Shi et al., 2020; Qiu et al., 2023; Zhou et al., 2022b; Wang et al., 2023; Long, 2023; Mao et al., 2023; Trivedi and Sarvadevabhatla, 2022; Do and Kim, 2024). These approaches explore diverse tokenization strategies, attention designs, and training pipelines to enhance model performance. For instance, STSA-Net (Qiu et al., 2023) employs a spatiotemporal segment encoding strategy to fuse joint relations across frames. Hyperformer (Zhou et al., 2022b) introduces a novel self-attention mechanism on hypergraphs, capturing higher-order relations intrinsic to skeleton data. To align with pretraining paradigms used in transformers, MAMP (Mao et al., 2023) performs masked self-component reconstruction on human joints, enabling effective feature representation for 3D action recognition. Representations. Recent advancements in skeleton-based action recognition increasingly exploit various intra-skeleton modalities (i.e., skeletal representations) and their ensembles (Chen et al., 2021; Chi et al., 2022; Lee et al., 2023; Shi et al., 2020; Qiu et al., 2023; Myung et al., 2024), including joints, bones, and velocities. PSUMNet (Trivedi and Sarvadevabhatla, 2022), for instance, integrates multiple intra-skeleton modalities into a unified framework, demonstrating the effectiveness of such a holistic approach. Objectives. There has also been interest in probing alternative or auxiliary optimization objectives to enhance robustness (Chi et al., 2022; Huang et al., 2023), incorporate supplementary textual descriptions (Xu et al., 2023; Xiang et al., 2023; He et al., 2024), and address challenging scenarios (Peng et al., 2024; Liu et al., 2024a). However, these methods mainly focus on individual actions and circumvent multi-entity interaction problems. Taking late fusion strategy for adaptation relies on the assumption that entities are IID (Chen et al., 2021; Chi et al., 2022; Lee et al., 2023; Shi et al., 2020; Qiu et al., 2023; Shi et al., 2019). Our proposed approach can enhance these existing methods noticeably by addressing the entity bias problem. 2.3 Skeleton-based Multi-Entity Action Recognition Human-Human Interactions. Since the introduction of two-person interaction datasets (Shahroudy et al., 2016; Liu et al., 2020a; Yun et al., 2012), the research community has recognized the importance of interaction modeling and has made significant efforts to develop more comprehensive benchmarks (Guo et al., 2022; Yin et al., 2023; Xu et al., 2024; Liang et al., 2024) and advanced models (Ji et al., 2014; Pang et al., 2022; Li et al., 2023a; Perez et al., 2022; Wen et al., 2023b; Liu et al., 2025; Pang et al., 2025). For example, LSTM-IRN (Perez et al., 2022) incorporated relational reasoning to model interactions by capturing diverse relationships between human joints. GDCN (Li et al., 2023a) introduced a graph diffusion convolutional network to uncover intrinsic local-global clues in two-person activities. IGFormer (Pang et al., 2022) leveraged prior knowledge of human body structure to design co-attention mechanisms for interaction recognition. Similarly, me-GCN (Liu et al., 2025) extracted adjacency matrices for individual entities and modeled mutual constraints between them to enhance performance. However, these approaches often rely on strong priors and intricate architectures that are specifically designed to handle interactions between exactly two human bodies, limiting their generalizability to broader scenarios. Hand-Object Interactions. Many existing works has explored egocentric hand-hand (Sener et al., 2022; Wen et al., 2023a; Ohkawa et al., 2023; Shamil et al., 2024) and hand-object (Kwon et al., 2021; Garcia-Hernando et al., 2018; Tekin et al., 2019; Shamil et al., 2024; Cho et al., 2023; Mucha and Kampel, 2024; Zhu et al., 2025) interactions. TA-GCN (Kwon et al., 2021) employs a topology-aware graph convolutional network to model hand-object relationships, where the graph dependencies of hands are predefined as priors. EffHandEgoNet (Mucha and Kampel, 2024) introduces a transformer-based architecture for action recognition that utilizes 2D poses of hands and objects. Similarly, HandFormer (Shamil et al., 2024) capitalizes on the unique characteristics of hand poses by temporally factorizing hand modeling and representing each joint through its short-term trajectories, achieving both efficiency and high accuracy. While these studies have made notable progress, they remain confined to this specific sub-domain by embedding hand and object priors directly into their model designs, which restricts their applicability to broader interaction scenarios. Human-Robot Interactions (HRI). In recent years, the skeleton-based Human-Robot Interaction datasets have been introduced (Avogaro et al., 2024; Sampieri et al., 2022). HARPER (Avogaro et al., 2024) is the first dataset to capture physical dyadic interactions between humans and quadruped robots, including unexpected collisions. In contrast, Chico (Sampieri et al., 2022) focuses on industrial settings, providing a smaller-scale dataset with multi-view videos and skeletons of operators and cobots engaging in collaborative assembly tasks. Notably, these two datasets are the only available resources that offer both human and robot 3D skeletons along with sufficient interaction categories for model training. Despite their introduction, to the best of our knowledge, no prior work has specifically addressed the skeleton-based HRI recognition task, leaving a critical gap in this domain of research. Group Activities. Group activities (Lan et al., 2012; Duan et al., 2023; Wang et al., 2024; Chang et al., 2024; Tamura, 2024) involve assigning semantic labels to motions that often include a large number of entities, some of which may perform irrelevant individual actions (Choi et al., 2009; Ibrahim et al., 2016). Methods tailored to this scenario (Azar et al., 2019; Wu et al., 2019; Gavrilyuk et al., 2020; Li et al., 2021; Yuan and Ni, 2021; Tamura et al., 2022; Han et al., 2022; Thilakarathne et al., 2022; Zhou et al., 2022a; Yuan et al., 2021) typically rely on multi-modality fusion or highly complex model architectures to achieve competitive performance. For example, COMPOSER (Zhou et al., 2022a) employed extremely intricate multi-scale design alongside a series of specialized learning objectives to train a heavy model for only 10-frame group activity learning. The above approaches excel in their respective sub-domains by leveraging two main strategies: incorporating domain-specific priors or employing intricate model designs. In contrast, we demonstrate that simple models, such as those initially designed for single-entity actions, can achieve competitive performance across these diverse scenarios when integrated with our CHASE method. Importantly, CHASE enables such models to perform effectively with minimal reliance on domain-specific priors, showcasing its adaptability. Figure 2: The overall framework of the proposed CHASE for skeleton-based action recognition, including individual actions, person-person interactions, hand-object interactions, human-robot interactions, and group activities. For clarity, we offer an example of person-person interaction skeleton sequence for pipeline illustration. Given a skeleton sequence as input, CHASE leverages Convex Hull Adaptive Shift, a learnable normalization module, to generate plausible and sample-adaptive offset for each input. It also collects pair-wise shifted skeletons within mini-batches, effectively addressing entity bias by introducing an auxiliary objective. Sub-Entity Strategy allows CHASE to be applied in both individual and multi-entity actions. Moreover, CHASE can be adopted in various widely-used skeletal modalities defined using joints and physical topology for de-biasing. 3 CHASE Fig. 2 presents the overall CHASE framework. The method is organized around four components, each building on the previous. Section 3.1 establishes the notation used throughout. Section 3.2 presents the Convex Hull Adaptive Shift, the core normalization module, and Section 3.3 introduces the auxiliary Entity Bias Minimization objective that guides its training. Together these two components form the foundation for multi-entity settings. The remaining two sections describe the journal-specific extensions. Section 3.4 develops the sub-entity strategy, which reformulates the single-entity setting so that the same MPMMD objective remains applicable when only one entity is present. Section 3.5 examines how entity bias manifests across different skeletal representations and shows that CHASE operates without any modality-specific assumptions. 3.1 Preliminaries Skeleton Sequences. Suppose that E entities (e.g. persons) engage in a purposeful act during a period of time T, and the pose of each entity is indicated by J joints with C Cartesian coordinates. We can define the skeleton sequence of an action as X∈ℝC×T×J×EX ^C× T× J× E. For clarity, we denote the total number of points (across all frames, joints, and entities) as U=T×J×EU=T× J× E. Skeleton-based Action Recognition. We define the task as finding the optimal estimator ℰθE_θ of the mapping ℰ:X↦YE:X Y, where X is the skeleton sequence of an action and Y is its corresponding label. Late Fusion Strategy. A common practice named late fusion strategy is widely-adopted when single-entity action recognition models meet complicated skeletal data (e.g., with multiple entities) (Chen et al., 2021; Chi et al., 2022; Lee et al., 2023; Shi et al., 2020; Qiu et al., 2023; Myung et al., 2024). This strategy begins with separating each entity. Subsequently, weight-shared backbones (e.g., GCN and Transformer) learn spatiotemporal features of each entity. Finally, an averaging operator applies to these features, whose result serves as the overall feature of this skeleton sequence. Notably, this common practice uses weight-shared encoders, which implicitly relies on an empirical assumption that all entities are independent and identically distributed. 3.2 Convex Hull Adaptive Shift The bias observed in skeleton sequences arises from the initial configuration of the world coordinate system. A representative example is the standard preprocessing of NTU RGB+D 120 (Liu et al., 2020a), one of the most widely used benchmarks, in which the coordinate origin is translated to the spine-base joint (pelvis) of the first person. This convention is chosen for its stability as a reference point, and works well in single-entity recognition. In multi-entity settings, however, it introduces a systematic asymmetry: the first entity is always centered at the origin while all other entities are displaced by amounts proportional to their physical separation, producing inter-entity distributional discrepancies that scale with that separation. More broadly, any fixed choice of reference joint or reference entity during data collection or preprocessing will embed such asymmetries into the representation. To address this distribution discrepancy, we propose the Convex Hull Adaptive Shift, a learnable parameterized network characterized by two key features. First, the offset is adaptively tailored to each individual skeleton sequence. Second, the shift vector is inherently plausible, as it is implicitly constrained within the skeleton’s convex hull. It avoids non-convergence by limiting the search space. By applying the Convex Hull Adaptive Shift, each sample is relocated to ensure that entities are rendered approximately IID. For a skeleton sequence X∈ℝC×UX ^C× U, shifting all points by a common vector p∗→∈ℝC×1 p^* ^C× 1 can be written compactly as: X^=X−p∗→J1,U, X=X- p^*J_1,U, (1) where J1,U∈ℝ1×UJ_1,U ^1× U is a row vector of ones that broadcasts p∗→ p^* across all U points, and X X is the shifted sequence. Making p∗→ p^* depend on the input via p∗→=XW p^*=XW introduces adaptability, but leaves p∗→ p^* unconstrained in ℝ3R^3, which can cause non-convergence. We address this by constraining p∗→ p^* to lie within the convex hull of X (Rockafellar, 1996) — the unique minimal convex set containing all points of X, equivalently the set of all convex combinations of those points. Figure 3: Skeleton convex hulls. A skeleton convex hull, denoted using dark blue dashed lines, is defined by all the spatial and temporal keypoints in a skeleton sequence. All feasible new origins remain within the convex hull, which limits the search space for shift vector and avoids non-convergence. The shift vector p∗→ p^* is formulated as: p∗→=Xsoftmax(W), p^*=Xsoftmax(W), (2) where X∈ℝC×UX ^C× U, W∈ℝU×1W ^U× 1, and p∗→∈ℝC×1 p^* ^C× 1. Let α~i=eαi/∑jeαj α_i=e _i/ _je _j denote the i-th component of softmax(W)softmax(W), where αi _i is the i-th element of W. Since α~i∈(0,1) α_i∈(0,1) and ∑iα~i=1 _i α_i=1, we have p∗→=∑i=1Uα~ip→i p^*= _i=1^U α_i p_i, which satisfies the definition of a convex combination. Therefore, p∗→ p^* represents a convex combination of points in X, lying within the minimal convex set encompasses X, as shown in Fig. 3. Convex Hull Adaptive Shift (CHAS). Combining Formula (1) and (2), we propose the Convex Hull Adaptive Shift as: X^=X(I−softmax(W)J1,U), X=X(I-softmax(W)J_1,U), (3) where I∈ℝU×UI ^U× U is the identity matrix. This formulation restricts the search space of p∗→ p^* from the entire ℝ3R^3 to the open convex hull, ensuring both adaptivity and plausibility. Coefficient Learning Block (CLB). With a fixed W, the gradient ∂X^/∂X=I−JU,1softmax(WT)∂ X/∂ X=I-J_U,1softmax(W^T) is constant, meaning all inputs share the same shift coefficients. True sample-adaptivity therefore requires W to depend on X through a learnable mapping ψ:ℝC×U↦ℝU×1ψ:R^C× U ^U× 1. As illustrated in Fig. 2, we implement this as a lightweight Coefficient Learning Block: W=ψ(X)=W3δ(W2ϕ(W1X+b)),W=ψ(X)=W_3δ(W_2φ(W_1X+b)), (4) where W1∈ℝC1×C,W2∈ℝC2×C1,W3∈ℝU×C2W_1 ^C_1× C,W_2 ^C_2× C_1,W_3 ^U× C_2 are weight matrices, b is a bias, ϕφ is a squeeze operator (Hu et al., 2018), and δ is an activation function. We set U≥C1>C2U≥ C_1>C_2 following the bottleneck design (Hu et al., 2018; Srivastava et al., 2015). 3.3 Objective for Entity Bias Minimization To optimize CHAS, we introduce an auxiliary objective that minimizes pairwise distributional discrepancies between entities, thereby preventing the learned shifts from collapsing into trivial solutions. Enforcing alignment between the feature distributions of different entities within each training mini-batch incrementally drives the network toward a canonical, shared coordinate space — mitigating entity bias without requiring global distribution matching. We build this objective on Maximum Mean Discrepancy (MMD), a kernel-based metric that measures the distance between two distributions P(x)P(x) and Q(y)Q(y) as: MMD(P,Q)=sup‖f‖ℋ≤1([f(x)]−[f(y)]),MMD(P,Q)= _\|f\|_H≤ 1 (E[f(x)]-E[f(y)] ), (5) where the supremum is over all functions f in the reproducing kernel Hilbert space ℋH. For E entity distributions PiP^i, the average pairwise MMD is: r(z)[MMD(z)]=∑i=1E−1∑j=i+1EMMD(Pi,Pj)/C(E,2),E_r(z)[MMD(z)]= _i=1^E-1 _j=i+1^EMMD(P^i,P^j)/C(E,2), (6) where z=(Pi,Pj)z=(P^i,P^j), r(z)r(z) is the probability density over pairs, and C(E,2)C(E,2) counts all entity pairs. Two approximations make this tractable: expectations are estimated over mini-batches, and rather than enumerating all C(E,2)C(E,2) pairs (complexity O(n!)O(n!)), we sample M pairs uniformly, giving: r(z)[MMD(z)]≈1M∑m=1MMMD(zm).E_r(z)[MMD(z)]≈ 1M _m=1^MMMD(z_m). (7) We refer to this as the Mini-batch Pairwise Maximum Mean Discrepancy Loss ℒmpmmdL_mpmmd. The total training loss is: ℒ=ℒCLS+λℒmpmmd,L=L_CLS+ _mpmmd, (8) where λ is a trade-off factor and ℒCLSL_CLS is for classification. The pseudo-code implementation of CHASE is detailed in Algorithm 1. For the current mini-batch, we apply a learnable normalization module to the input skeletal data x, and guide the bias minimization with the proposed MPMMD loss. Algorithm 1 Pseudo code of CHASE in a PyTorch-like style. ⬇ # backbone: encoder networks for single entity # in_channels: number of coordinates (C) # num_frame: number of frames (T) # num_point: number of joints per entity (J) # num_entity: number of entities (E) # pooling_seg, c1, c2: projection dimensions # lambda: trade-off factor # initialize num_list = [num_frame, num_point, num_entity] out_channel = prod(num_list) seg = prod(pooling_seg) seg_num_list = round(hd(num_list, pooling_seg)) seg_num = prod(seg_num_list) shift = n.Sequential( n.Conv3d(in_channels, c1, 1), n.AdaptiveAvgPool3d(pooling_seg), n.Conv3d(c1, c2, 1, bias=False), n.ReLU(inplace=True), n.Conv3d(c2, outchannel, 1, bias=False), ) for x, label in loader: # load a minibatch N, C, T, V, M = x.size() # Formula (7) sf = shift(x).view(N, T*M*V, -1) tx = rearrange(x) # Formula (5) sf = (tx @ sf.softmax(dim=1)).unsqueeze(-1) sf = rearrange(sf) x = x - sf # Formula (10) pairs = MiniBatchSampling(x) aux_loss = MPMMD(pairs) out = backbone(x) # Formula (11) total_loss = ce(out, label)+lambda*aux_loss prod: Product of all elements; hd: Hadamard Division; ce: Cross-Entropy. 3.4 Any-Entity Generalization via Sub-Entity Strategy Figure 4: Overview of sub-entity strategy. Black circles denote joints. A single entity can be viewed as many sub-entities of its own parts (denoted in different colors), which satisfy the subgraph and isomorphism conditions. Examples on human body skeletons are 2-part (2.P.) and 5-part (5.P.) partitions. So far, we have discussed CHASE under the condition E≥2E≥ 2. But what happens when E=1E=1? Single-entity actions are a critical area in spatiotemporal motion understanding. However, without multiple entities, defining pair-wise distribution discrepancies becomes infeasible. To address this, we introduce a sub-entity strategy that extends CHASE to single-entity actions. This approach adapts multi-entity definitions, providing a unified framework for handling any-entity actions. Sub-Entity Strategy. In this context, the CHASE framework treats a single entity as composed of multiple sub-entities. Accordingly, in the formulations that follow (Equations (3) and related), U denotes the number of these sub-entities, rather than the total number of temporal and spatial points. The sub-entity strategy is equivalent to selecting sub-entities w.r.t. a given graph G=(V,E)G=(V,E), such that the chosen sub-entities satisfy the following conditions: (1) Each sub-entity is a subgraph of the original graph; (2) All sub-entities are isomorphic. These conditions are detailed as follows: 1) Subgraph: Let G=(V,E)G=(V,E) be the original graph, and G′=(V′,E′)G =(V ,E ) be a sub-entity. Then, G′G is a subgraph of G if and only if: V′⊆V,E′⊆E,and∀(u,v)∈E′,u,v∈V′.V V,E E, ∀(u,v)∈ E ,u,v∈ V . (9) This means that G′G consists of a subset of the nodes V′V and a subset of the edges E′E , where every edge in E′E connects nodes in V′V . 2) Isomorphism: Let G1=(V1,E1)G_1=(V_1,E_1) and G2=(V2,E2)G_2=(V_2,E_2) be two sub-entities. Then, G1G_1 and G2G_2 are isomorphic if there exists a bijection g:V1→V2g:V_1→ V_2 such that for all pairs of nodes u,v∈V1u,v∈ V_1, the following holds: (u,v)∈E1⇔(g(u),g(v))∈E2.(u,v)∈ E_1 (g(u),g(v))∈ E_2. (10) This means there is a one-to-one correspondence g between the node sets of G1G_1 and G2G_2, such that the adjacency relationships (edges) are preserved under this mapping. Sub-entities are derived from physical topology. Figure 4 shows two examples: 2-part (2.P.) and 5-part (5.P.) decompositions based on natural body structure (Xiang et al., 2023). Sub-entities may overlap and exhibit 3D chirality; they can be proper or improper subgraphs. The primary motivation of this strategy differs fundamentally from conventional part-based modeling in skeleton-based recognition. Part-based methods decompose the body to improve feature extraction or capture local motion patterns in the backbone. The sub-entity strategy, by contrast, is not concerned with how features are extracted. Its purpose is to construct a set of structurally comparable units from a single entity so that the MPMMD loss can compute meaningful pairwise distribution discrepancies, an operation that is otherwise undefined when only one entity is present. The isomorphism condition is the critical formal requirement that makes this possible: it guarantees that all sub-entities share the same graph structure, and are therefore exchangeable under the distribution matching objective. Without this constraint, comparing distributions across structurally heterogeneous parts would be ill-defined. For single-entity actions (e.g., jumping), we treat an entity as multiple sub-entities corresponding to its body parts. This also enables smaller, weight-shared backbones by reducing the joint count per sub-entity, improving both efficiency and performance. The mechanism by which CHASE reduces entity bias among body parts follows the same principle as in the multi-entity setting. Even within a single person, anchoring the coordinate origin to a fixed reference point introduces distributional asymmetries between different body regions: parts that are spatially far from the reference accumulate larger positional offsets than parts that are close to it, so their joint coordinate distributions diverge in location and spread across samples. CHASE learns a sample-adaptive shift that brings these intra-body distributions closer together, guided by the MPMMD objective operating on the isomorphic sub-entities. The intra-body asymmetry is smaller in magnitude than the inter-person asymmetry, because body parts of the same person are spatially closer to each other than different persons typically are, which explains why the performance gains in single-entity settings are consistent but more modest than those in multi-entity settings. 3.5 A Modality-Agnostic De-biasing Framework The above sections formulate CHASE for joint modality (physical joints or keypoints). However, skeleton-based action recognition increasingly uses multiple intra-skeleton modalities and their ensembles (Chen et al., 2021; Chi et al., 2022; Lee et al., 2023; Shi et al., 2020; Qiu et al., 2023; Myung et al., 2024), such as bones and velocities. Here, we extend CHASE to these modalities. The definitions of the commonly used skeletal modalities (i.e., representations) are listed below: • Joint: A joint in a skeleton sequence is a point in the Cartesian coordinate system, which can be viewed as a vector p→i,t∈ℝC×1 p_i,t ^C× 1, where 1≤i≤J1≤ i≤ J and 1≤t≤T1≤ t≤ T. • Bone: A bone in a skeleton sequence is a differential vector p→i,j,t=p→i,t−p→j,t p_i,j,t= p_i,t- p_j,t, connecting two joints p→i,t p_i,t and p→j,t p_j,t. Here, 1≤i,j≤J1≤ i,j≤ J, 1≤t≤T1≤ t≤ T, and the connection between the joints is determined by a predefined graph (e.g., the bone structure of the human body). • Velocity: A velocity vector is a differential vector p→i,t,s=p→i,t−p→i,s p_i,t,s= p_i,t- p_i,s, representing the movement of the same joint (or bone) across different timestamps t and s, where typically t=s+1t=s+1 and 1≤s≤T−11≤ s≤ T-1. • JBF (Myung et al., 2024): A JBF vector is defined as the concatenation of a joint and a bone, represented as p→jbf∈ℝ2C×1 p_jbf ^2C× 1. Since vectors are points, the above definitions yield a set of points X′∈ℝC′×U′X ^C × U , where dimensions depend on the skeletal modality. CHASE only requires point sets as input, enabling straightforward extension to any modality. We reformulate Formula (3) as: X^=X′(I−softmax(W)J1,U′). X=X (I-softmax(W)J_1,U ). (11) Formulas (4) and (7) remain valid by replacing X with X′X . For bone representations, the sub-entity strategy preserves the Subgraph and Isomorphism conditions using the line graph L(G)L(G), the edge-to-vertex dual of G. In L(G)L(G), each edge of G becomes a vertex, and edges connect vertices representing edges in G that share a common vertex. Sub-entities are then selected as subgraphs of L(G)L(G). Table 1: Details of Benchmarks Used in Experiments Benchmark Skeleton Sequences #Category #Joint #Entity #Participant Type Example(s) Body Hand Object Robot NTU60 (Shahroudy et al., 2016) ✓ 60 25(×2) 1 or 2 40 3D hugging, walking towards NTU120 (Liu et al., 2020a) ✓ 120 25(×2) 1 or 2 106 3D exchange things, whisper H2O (Kwon et al., 2021) ✓ ✓ 36 21×3 3 4 3D squeeze lotion out from bottle Assembly101 (Sener et al., 2022) ✓ 1,380 21×2 2 53 3D screw track with hand CAD (Choi et al., 2009) ✓ 4 17×5.22 ≈ 5.22 Uncontrolled 2D queuing, walking, talking VD (Ibrahim et al., 2016) ✓ ✓ 8 17×12+1 13 Uncontrolled 2D left pass, right spike HARPER (Avogaro et al., 2024) ✓ ✓ 15 21+23 2 17 3D circular follow + crash 4 Experiments 4.1 Datasets & Evaluation In experiments, we evaluate the methods on a variety of skeletal datasets, including human actions & interactions (Shahroudy et al., 2016; Liu et al., 2020a), hand-object interactions (Kwon et al., 2021), bi-hand interactions (Sener et al., 2022), human-robot interactions (Avogaro et al., 2024), and group activities (Choi et al., 2009; Ibrahim et al., 2016). Details of these datasets are listed in Table 1. NTU RGB+D (NTU60) (Shahroudy et al., 2016) is a widely-used dataset for human activity recognition, structured into two distinct subsets: NTU49, which focuses on actions involving a single individual, and NTU11, dedicated to interactions between two people. In our experiments, we adopt the commonly applied X-Sub and X-View evaluation protocols. Table 2: Comparisons with Related Methods on Multi-Entity Action Recognition Datasets. Our proposed CHASE can improve diverse single-entity backbones to achieve state-of-the-art performance across a variety of benchmarks. Method* Venue NTU26(%) NTU11(%) H2O(%) ASB101(%) CAD(%) VD(%) HARPER(%) X-Sub X-Set X-Sub X-View Multi-Entity Backbones LSTM-IRN (Perez et al., 2022) TMM’22 77.70 79.60 90.50 93.50 - - - - - IGFormer (Pang et al., 2022) ECCV’22 85.40 86.50 93.60 96.50 - - - - - SkeleTR (Duan et al., 2023) ICCV’23 87.80 88.30 - - - - - - - ISTA-Net (Wen et al., 2023b) IROS’23 90.56 91.72 - - 89.09 28.01 87.16 91.40 79.63 H2OTR (Cho et al., 2023) CVPR’23 - - - - 90.90 - - - - GDCN (Li et al., 2023a) TPAMI’23 85.80 92.10 - - - - - - - AHNet-Large (Wang et al., 2024) PR’24 86.43 86.64 90.85 93.38 - - 89.32 84.31 - ME-Former (Hu and Liu, 2024) Biomi.’24 90.84 91.33 95.37 97.60 - - - - - EffHandEgoNet (Mucha and Kampel, 2024) arXiv’24 - - - - 91.32 - - - - HandFormer (Shamil et al., 2024) ECCV’24 - - - - 57.44 28.80 - - - me-GCN (Liu et al., 2025) THMS’25 90.00 90.00 95.50 98.20 - 27.20 - - - ASEA (Pang et al., 2025) ACMMM’25 90.52 91.77 - - - - - - - Single-Entity Backbones (+ CHASE) CTR-GCN (Chen et al., 2021) ICCV’21 89.32 90.19 95.94 98.32 81.68 27.83 80.45 92.66 78.24 + CHASE (Ours) - 91.30 92.34 96.45 98.83 91.05 28.03 89.61 92.89 86.11 InfoGCN (Chi et al., 2022) CVPR’22 90.22 91.13 95.51 97.76 76.24 27.18 83.07 91.77 80.56 + CHASE (Ours) - 91.86 92.41 96.35 98.25 83.47 27.36 84.18 92.00 83.33 STSA-Net (Qiu et al., 2023) Neuro.’23 88.41 90.19 95.96 98.47 92.29 27.70 80.20 92.52 78.71 + CHASE (Ours) - 89.77 91.54 96.63 98.73 94.77 27.81 85.93 92.78 88.89 HD-GCN (Lee et al., 2023) ICCV’23 88.25 90.08 95.58 97.93 72.73 27.31 76.93 91.32 80.10 + CHASE (Ours) - 90.81 92.06 96.22 98.31 81.61 27.50 82.39 92.00 89.82 DeGCN (Myung et al., 2024) TIP’24 90.84 90.36 95.94 97.95 80.16 28.76 83.66 93.74 83.79 + CHASE (Ours) - 91.61 92.77 96.82 98.75 86.50 28.91 86.41 93.97 94.90 Hyper-GCN (Zhou et al., 2025) ICCV’25 86.90 88.46 94.97 97.67 80.44 19.35 65.51 79.40 72.69 + CHASE (Ours) - 89.01 90.29 96.12 98.68 90.22 21.04 70.74 81.41 84.26 • * The best result of each benchmark is in bold. NTU RGB+D 120 (NTU120) (Liu et al., 2020a), an expanded version of NTU60, offers a larger and more diverse collection of human activity samples. It is also divided into two non-overlapping subsets: NTU94 for individual actions and NTU26 for mutual activities involving two participants. For evaluation, we utilize the X-Sub and X-Set benchmarks as described in (Liu et al., 2020a). H2O (Kwon et al., 2021) contributes to the field of 3D vision by offering annotated human hand poses and bounding boxes for manipulated objects. These resources support the modeling of both hand-object and hand-hand interactions. We utilize the training, validation, and test divisions as defined in (Kwon et al., 2021). Assembly101 (ASB101) (Sener et al., 2022), a comprehensive dataset for 3D manual procedural activities, features 1,380 distinct categories of interactive actions. Following (Sener et al., 2022), we adopt its predefined splits for training, validation, and testing. The labels used in our evaluations include detailed verb and noun pairs to capture fine-grained action semantics. Collective Activity Dataset (CAD) (Choi et al., 2009) documents pedestrian activities in public spaces using camera footage. It organizes group behaviors into four categories and assigns individual action labels. Our implementation follows the same classification schema and train-test configuration described in (Zhou et al., 2022a), but relying solely on 2D joint coordinate data. Volleyball Dataset (VD) (Ibrahim et al., 2016) contains match footage from volleyball games, where group activities are categorized into eight classes aligned with volleyball terminology. For our experiments, we adhere to the original splits proposed in (Zhou et al., 2022a), but relying solely on 2D joint coordinate data. HARPER (Avogaro et al., 2024) is a pioneering dataset centered on human-robot interaction, showcasing 15 types of dyadic interactions between humans and a Boston Dynamics Spot quadruped robot. Performed by 17 individuals, these interactions predominantly involve physical contact. We use the training and testing splits recommended in (Avogaro et al., 2024) for our evaluations. Figure 5: Qualitative results of CHASE. Different entity distributions are denoted by blue and orange. By incorporating CHASE, entity distributions become more aligned in terms of both mean and covariance. CHASE effectively mitigates the inherent entity bias, demonstrating its clear effectiveness across a range of data scales and a variety of benchmarks. 4.2 Implementation Details Experiments are conducted on the GeForce RTX 3070 GPUs. We select a diverse set of baseline models, encompassing both GCN-based, hyper-GCN-based, and transformer-based architectures. Specifically, our GCN-based baselines include CTR-GCN (Chen et al., 2021), InfoGCN (Chi et al., 2022) (k=1), HD-GCN (Lee et al., 2023) (CoM=1), and DeGCN (Myung et al., 2024). Hyper-GCN (large) (Zhou et al., 2025) is chosen as the hyper-GCN-based baseline. STSA-Net (Qiu et al., 2023) is included as our transformer-based representative. For training on NTU26 dataset, we follow data pre-processing and augmentations adopted in (Chen et al., 2021; Wen et al., 2023b). The classification loss is implemented by the cross entropy with label smoothing factor 0.1. SGD optimizer is used with Nesterov momentum of 0.9, a initial learning rate of 0.1 and a decay rate 0.1. Batch size is 64. Each training process was terminated after 110 epochs. As the hyper-parameters may vary across different benchmarks, please refer to the benchmark-specific configurations in our code repository. Table 3: Comparison with Other Alternatives Method (w/ CTR-GCN) Acc (%) Δ (%) Vanilla 89.32 - S2CoM 88.66 -0.67 BatchNorm 89.06 -0.27 ER 89.34 +0.02 Aug 89.72 +0.40 Per-Entity Learnable Offset 89.91 +0.59 Per-Joint Learnable Offset 89.46 +0.14 Global Learnable Offset 90.02 +0.70 S2CoM†/STD 90.29 +0.97 S2CoM† 90.79 +1.47 CHASE (Ours) 91.30 +1.98 4.3 Quantitative Results Recognition Performance. Table 2 summarizes the experimental results on 7 multi-entity action recognition datasets. For fair comparisons (Pang et al., 2022; Wen et al., 2023b), only the joint modality is used in this experiment. Top-1 accuracy serves as the evaluation metric. The results reveal two key observations: 1) We compare the proposed CHASE framework with 5 baseline single-entity backbone models (indicated by the light blue background). By employing CHASE to mitigate entity bias in skeletal data, we consistently enhance the performance of all baseline models. For instance, CHASE significantly improves the state-of-the-art single-entity backbone DeGCN (Myung et al., 2024), demonstrating a notable performance boost across various settings. The performance improvements differ across baseline models and benchmarks, primarily due to variations in the extent of entity bias, which is influenced by factors such as skeleton type and backbone architecture. 2) We further compare CHASE-augmented models with several state-of-the-art multi-entity encoders (indicated by the light yellow background), which often feature complex designs and limited applicability. CHASE enables simpler models employing a late fusion strategy to perform effectively across diverse scenarios, even surpassing these advanced methods. Notable examples include outperforming ISTA-Net (Wen et al., 2023b), me-GCN (Liu et al., 2025), ASEA (Pang et al., 2025), and HandFormer (Shamil et al., 2024). It suggests that CHASE generalizes well to various recognition tasks, regardless of dataset complexity. Figure 6: More qualitative results. (a) UMAP (McInnes et al., 2018) visualizations of skeleton sequence representations on the test split of NTU26 X-Sub. Compared with Vanilla, our proposed CHASE differentiate similar actions and interactions better by assisting backbones to learn more distinctive representations. (b) Category-level accuracy comparison between the vanilla baseline and CHASE. It demonstrates the effectiveness of CHASE by improving recognition accuracy for most categories. (c) Ablation study on λ for MPMMD objective. Table 4: Analysis of Inter-entity Distribution Discrepancies Set Method Avg KLD ↓ JSD ↓ BD ↓ HD ↓ MMD ↓ I Vanilla 1.07 0.19 0.25 0.46 0.94 CHASE 0.39 0.08 0.10 0.30 0.05 I Vanilla 1.00 0.18 0.23 0.45 1.03 CHASE 0.45 0.10 0.11 0.32 0.07 I Vanilla 0.72 0.14 0.17 0.39 1.25 CHASE 0.41 0.08 0.10 0.30 0.05 IV Vanilla 0.75 0.14 0.17 0.40 1.15 CHASE 0.41 0.08 0.09 0.30 0.04 Table 5: Analysis of Key Components in CHASE. * indicates non-convergence due to large search space of shift vector. CHAS MPMMD LR Acc (%) Δ (%) AS CHC CLB ✓ ✓ ✓ ✓ 0.1 91.30 - ✓ ✓ ✓ 0.1 22.65* -68.65 ✓ ✓ ✓ 0.01 86.99 -4.32 ✓ ✓ ✓ 0.1 91.20 -0.10 ✓ ✓ 0.1 22.75* -68.56 ✓ ✓ 0.01 23.51* -67.79 ✓ ✓ ✓ 0.1 20.42* -70.88 ✓ ✓ ✓ 0.1 91.17 -0.13 0.1 89.50 -1.81 Comparison with Alternatives. We evaluate our proposed CHASE framework against several alternative methods: 1) Vanilla: Directly utilize raw world or pixel coordinates. 2) S2CoM: Relocate each local origin to the spatiotemporal center of mass for individual entities. 3) BatchNorm (Ioffe and Szegedy, 2015a): Introduce an additional Batch Normalization step immediately after batches of samples are fed into the model. 4) ER (Entity Rearrangement) (Wen et al., 2023b): A strategy designed to disrupt the inherent order of entities to better model interactions. 5) Aug: Enhance data through random shifts applied to skeleton sequences as an augmentation technique. 6) Per-Entity Learnable Offset: Each entity is assigned a dedicated learnable 3D offset vector that is subtracted from all its joints, directly targeting entity-level bias with static learned parameters. 7) Per-Joint Learnable Offset: Each joint is assigned a learnable 3D offset applied uniformly across all samples, targeting joint-level spatial bias. 8) Global Learnable Offset: A single learnable 3D offset applied to all samples uniformly, representing the simplest form of learnable bias correction. 9) S2CoM†: Reposition the global origin to the overall spatiotemporal center of mass. 10) S2CoM†/STD: Normalize data by scaling channels based on their standard deviations after applying S2CoM†. Table 3 demonstrates that CHASE consistently achieves the highest accuracy improvements among all these methods. Notably, while the three learnable offset baselines (Per-Entity, Per-Joint, Global) yield modest improvements over vanilla, they fall substantially short of CHASE. This demonstrates that CHASE’s gains come from the sample-adaptive shift mechanism and the convex hull constraint, rather than simply learning bias correction parameters: static offsets per entity or joint cannot capture the within-class diversity of entity configurations that CHASE handles dynamically for each input sample. Entity Bias. Table 4 presents metrics evaluating the entity bias on the test sets I-IV following (Wen et al., 2024)’s settings. We measure the pair-wise distributions of sampled data points from different entities using Averaged Kullback-Leibler Divergence (Avg KLD), Jensen-Shannon Divergence (JSD), Bhattacharyya Distance (BD), Hellinger Distance (HD) and MMD. Table 4 demonstrates that CHASE significantly minimizes discrepancies across all evaluation metrics, thereby benefiting backbone learning for each entity. 4.4 Qualitative Results Fig. 5 illustrates how CHASE effectively mitigates the entity bias across various data scales. By incorporating CHASE, entity distributions become more aligned in terms of both mean and covariance. Additionally, Fig. 6 (a) compares the UMAP visualization of representations learned by the vanilla CTR-GCN and its CHASE-enhanced counterpart. It demonstrates that CHASE enables the backbone to learn more distinctive feature representations, further enhancing its ability to differentiate between similar categories. Furthermore, category-level accuracy scores, presented in Fig. 6 (b), further highlight that CHASE improves the backbone’s ability to recognize complex actions. For instance, actions such as wield knife (+8.50%), shoot with gun (+8.00%), whisper (+5.39%), and punch/slap (+4.37%) show significant gains with CHASE integration. 4.5 Ablation Study Table 6: Sub-Entity Strategy on Single-Entity Actions Method Setting* # Param. NTU94(%) X-Sub X-Set CTR-GCN Full Body 1.46M 82.69(±0.10) 85.04(±0.04) CTR-GCN 2 Parts 1.44M 83.22(±0.13) 85.81(±0.03) + CHASE 2 Parts 1.46M 83.60(±0.08) 85.85(±0.01) CTR-GCN 5 Parts 1.44M 81.70(±0.10) 84.08(±0.05) + CHASE 5 Parts 1.45M 81.99(±0.07) 84.44(±0.03) In this section, we investigate the effectiveness and efficiency of CHASE by answering a series of questions with ablation studies. Unless specified otherwise, we report the experimental results on NTU26 X-Sub (Liu et al., 2020a) criterion using CTR-GCN (Chen et al., 2021) as the baseline model, which is a well-established evaluation protocol for ablation (Pang et al., 2022; Wen et al., 2023b; Liu et al., 2025). Do all key components contribute to overall performance? The effectiveness of each component is analyzed in Table 5. Removing the convex hull constraint (CHC) leads to a substantial accuracy reduction of over 60% at initial learning rates of 0.1 and 0.01. This sharp decline underscores the pivotal role of CHC in facilitating adaptive shift learning. Furthermore, substituting Adaptive Shift (AS) with X^=XWJ1,U X=XWJ_1,U significantly impairs accuracy, revealing that merely introducing a comparable number of trainable parameters without incorporating an adaptive shift mechanism is insufficient. Lastly, Table 5 highlights that CHASE also gains improvements from the inclusion of CLB and MPMMD. Table 7: Intra-Skeleton Modalities and Ensemble Modality Method Acc (%) Joint Vanilla 87.07 CHASE 87.42 Bone Vanilla 88.31 CHASE 88.81 Joint Velocity Vanilla 83.38 CHASE 83.50 JBF Vanilla 89.33 CHASE 89.65 Ensemble Vanilla 90.55 CHASE 90.86 Is the sub-entity strategy effective for adapting CHASE to single-entity skeletons? Table 6 reports the results on NTU94, a subset of NTU120 focusing on individual actions. We evaluate three settings: Full Body (F.B.), 2 Parts (2.P.), and 5 Parts (5.P.), with the latter two representing possible implementations of the sub-entity strategy. The results demonstrate that selecting a reasonable approach to define sub-entities, such as 2.P., leads to improvements in both accuracy and model efficiency. The improvements remain consistent across independent training runs with different seed initializations, as confirmed by the standard deviations reported in Table 6. Furthermore, incorporating CHASE into the baseline models further enhances recognition accuracy while introducing only a minimal increase in model size. We note that the performance gains in the single-entity setting are more modest compared to multi-entity scenarios. This is expected: in the multi-entity setting, different persons occupy distinct spatial locations, resulting in large inter-entity distribution discrepancies that CHASE can effectively reduce. In the single-entity setting under the sub-entity strategy, the entities are body parts of the same person, whose spatial extents are much closer to each other within the same skeleton. The inherently smaller distribution gap among sub-entities limits the room for de-biasing, leading to smaller but still consistent improvements. To verify that distributional discrepancies exist among sub-entities and that CHASE reduces them, we measure pairwise distribution distances between sub-entities on NTU94, following the same protocol as Table 4. We use the 5-part decomposition for this diagnostic analysis because it yields C(5,2) = 10 pairs per sample, providing a more statistically stable estimate than the 2-part setting which produces only a single pair. As shown in Table 8, CHASE consistently reduces all five metrics compared to the Vanilla baseline, confirming that intra-body distributional asymmetries are present and that CHASE mitigates them through the same coordinate-shifting mechanism as in the multi-entity setting. The absolute reductions are smaller than those in Table 4 because the inter-sub-entity gap has a different composition from the inter-person gap. In the multi-entity setting, the gap is dominated by coordinate origin offsets between different persons and is largely removable by coordinate normalization. Among sub-entities of the same skeleton, the gap also includes an intrinsic structural component arising from anatomical differences between body parts, such as the distinct motion patterns of arms and legs, which coordinate normalization cannot eliminate. CHASE addresses the removable coordinate origin portion in both cases, and the residual gap in Table 8 reflects this irreducible structural heterogeneity. Table 8: Inter-Sub-Entity Distribution Discrepancies on NTU94 Method Avg KLD ↓ JSD ↓ BD ↓ HD ↓ MMD ↓ Vanilla 1.898 0.300 0.473 0.567 0.914 CHASE 1.887 0.276 0.416 0.545 0.901 Table 9: Number of Learnable Parameters CTR-GCN InfoGCN STSA-Net HD-GCN DeGCN Vanilla 1.44M 1.54M 4.13M 1.65M 1.36M CHASE 1.46M 1.57M 4.16M 1.68M 1.39M Δ +1.83% +1.96% +0.60% +1.60% +1.93% Can CHASE improve models with other intra-skeleton modalities as input? We verify this on NTU120 X-Sub benchmark by training the state-of-the-art model, DeGCN (Myung et al., 2024), using different skeletal modalities as input, followed by obtaining the ensemble result. As shown in Table 7, CHASE-wrapped encoders consistently achieve better recognition accuracies compared to their vanilla counterparts across various intra-skeleton modalities, including joints (+0.35%), bones (+0.50%), joint velocities (+0.12%), and JBF (Myung et al., 2024) (+0.32%). Notably, CHASE also improves the ensemble result by 0.31%, underscoring its versatility and effectiveness in enhancing recognition performance across multiple modalities. The varying magnitude of improvements across modalities reflects how entity bias manifests differently in different representations. Bones and JBF, which encode absolute structural relationships directly dependent on coordinate origin choices, benefit more substantially from adaptive coordinate shifting. Joints also demonstrate clear benefits as they represent absolute spatial positions. In contrast, joint velocities, which capture relative motion between consecutive frames, experience smaller performance gains because this modality is less directly affected by static coordinate offsets. These results demonstrate that CHASE’s de-biasing strategy effectively addresses entity bias in a manner proportional to each modality’s vulnerability to coordinate system effects. Is CHASE lightweight and efficient? CHASE is designed as a flexible wrapper compatible with various backbones. As shown in Table 9, it introduces approximately 26.37k trainable parameters, amounting to only a 1%-2% increase over the backbone’s original parameters, depending on the base model size. The total trainable parameters can be estimated as (U+1+C2)×C1+C2×U(U+1+C_2)× C_1+C_2× U. In terms of computational cost, CHASE adds around 2.50M FLOPs. These results confirm that it achieves a balance between effective and lightweight design, making it well-suited for enhancing skeleton-based learning. Table 10: Mixed Recognition of Single- & Multi-Entity Actions. * indicates this model is implemented and trained using the public code. Method Venue NTU120(%) NTU60(%) X-Sub X-Set X-Sub X-View CTR-GCN ICCV’21 88.90 90.60 92.40 96.80 InfoGCN CVPR’22 89.80 91.20 93.00 97.10 STSA-Net Neuro.’23 88.50 90.70 92.70 96.70 HD-GCN ICCV’23 90.10 91.60 93.40 97.20 SkateFormer ECCV’24 89.80 91.40 93.50 97.80 BlockGCN CVPR’24 90.30 91.50 93.10 97.00 MMCL M’24 90.30 91.70 93.50 97.40 Shap-Mix IJCAI’24 90.40 91.70 93.70 97.10 DeGCN* TIP’24 90.55 91.83 93.32 97.14 + CHASE - 90.86 91.89 93.39 97.20 Can CHASE improve performance in mixed single- and multi-entity action recognition? Table 10 presents the recognition performance for mixed single- and multi-entity actions across the entire NTU120 and NTU60 datasets. The results highlight the comparative effectiveness of the proposed CHASE framework when integrated with the state-of-the-art DeGCN (Myung et al., 2024). CHASE-augmented DeGCN achieves competitive or superior performance compared to other related methods, outperforming MMCL (Liu et al., 2024b) and Shap-Mix (Zhang et al., 2024) on more challenging NTU120 benchmark. What are the optimal hyperparameters for MPMMD? Fig. 6 (c) illustrates the impact of varying the trade-off weight factor λ. The best performance is achieved when λ=0.1λ=0.1, indicating that MPMMD functions as an auxiliary objective complementing the primary recognition task objective. Additionally, we experimented with different values of M but observed insignificant performance differences. Therefore, we set M=1M=1 to ensure computational efficiency. Table 11: Test-Time Skeleton Noises and Masking Test-Time + Noise (%) + Mask (%) σ=10−3σ=10^-3 σ=10−2σ=10^-2 pm=10−2p_m=10^-2 pm=10−1p_m=10^-1 CTR-GCN 88.55 80.72 81.15 56.37 + CHASE 91.24 82.53 88.57 60.65 Is the CHASE-wrapped model robust to test-time noise and occlusions? To assess robustness, we introduce intentional corruption to skeleton sequences during inference, simulating potential occlusions or errors in skeleton estimation. The added noise Xn∼(μ,σ2)X_n (μ,σ^2) follows a normal distribution with a mean of μ=0μ=0 and standard deviations of σ=10−3σ=10^-3 and 10−210^-2. For masking, skeleton sequences are randomly masked with probabilities pm=10−2p_m=10^-2 and 10−110^-1. When tested on noisy and occluded inputs in Table 11, CTR-GCN integrated with CHASE delivers better performance compared to the vanilla version, demonstrating enhanced robustness. 5 Conclusion This paper presents CHASE, a normalization method based on Convex Hull Adaptive Shift to minimize Entity bias for skeleton-based action and interaction recognition. To the best of our knowledge, our proposed approach is the first to address the observed entity bias in various skeleton sequences. Our core idea to de-bias is adaptively re-centering the world origin of each sample, thereby unbiasing the subsequent backbones and boosting their performance. CHASE works as a plug-and-play normalization module to reduce entity bias and helps the subsequent classifier gain improved recognition performances across diverse settings. For evaluations, we conduct extensive experiments using 5 baseline backbones on 7 datasets across various numbers and types of entities. The experimental results consistently indicate that CHASE can help reach state-of-the-art performances across a variety of action and interaction recognition tasks, contributing to developing efficient and effective skeleton-based learning models. Declarations 5.1 Funding This work was supported by National Natural Science Foundation of China (No. 62473007), Guangdong Outstanding Youth Fund (No. 2026B1515020015), Shenzhen Innovation in Science and Technology Foundation for The Excellent Youth Scholars (No. RCYX20231211090248064), Southern Marine Science and Engineering Guangdong Laboratory (Zhuhai) (SML2024SP007) and Research Project (No. HDHDW59B010202). 5.2 Competing Interests The authors have no competing interests to declare that are relevant to the content of this article. 5.3 Data Availability The authors declare that the datasets used in this paper are available at the following links: 1. NTU RGB+D (Shahroudy et al., 2016) and NTU RGB+D 120 (Liu et al., 2020a): https://rose1.ntu.edu.sg/dataset/actionRecognition/ 2. H2O (Kwon et al., 2021): https://h2odataset.ethz.ch/ 3. Assembly101 (Sener et al., 2022): https://assembly-101.github.io/ 4. Collective Activity Dataset (Choi et al., 2009): https://cvgl.stanford.edu/projects/collective/collectiveActivity.html 5. Volleyball Dataset (Ibrahim et al., 2016): https://github.com/mostafa-saad/deep-activity-rec 6. HARPER (Avogaro et al., 2024): https://github.com/intelligolabs/HARPER References Avogaro et al. (2024) A. Avogaro, A. Toaiari, F. Cunico, X. Xu, H. Dafas, A. Vinciarelli, E. Li, and M. Cristani Exploring 3d human pose estimation and forecasting from the robot’s perspective: the harper dataset. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. , p. . Cited by: 3rd item, §1, §2.3, Table 1, §4.1, §4.1, item 6. Azar et al. (2019) S. M. Azar, M. G. Atigh, A. Nickabadi, and A. Alahi Convolutional relational machine for group activity recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.3. Ba et al. (2016) J. L. Ba, J. R. Kiros, and G. E. Hinton Layer normalization. arXiv preprint arXiv:1607.06450. Cited by: §2.1. Chang et al. (2024) C. Chang, D. Li, D. Patel, P. Goel, H. Zhou, S. Moon, S. S. Sohn, S. Yoon, V. Pavlovic, and M. Kapadia M3Act: learning from synthetic human group activities. In Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.3. Chen et al. (2021) Y. Chen, Z. Zhang, C. Yuan, B. Li, Y. Deng, and W. Hu Channel-wise topology refinement graph convolution for skeleton-based action recognition. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 13359–13368. Cited by: §1, §2.2, §2.2, §2.2, §3.1, §3.5, §4.2, §4.5, Table 2. Chi et al. (2022) H. Chi, M. H. Ha, S. Chi, S. W. Lee, Q. Huang, and K. Ramani InfoGCN: representation learning for human skeleton-based action recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 20154–20164. Cited by: §1, §2.2, §2.2, §2.2, §2.2, §3.1, §3.5, §4.2, Table 2. Cho et al. (2023) H. Cho, C. Kim, J. Kim, S. Lee, E. Ismayilzada, and S. Baek Transformer-based unified recognition of two hands manipulating objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 4769–4778. Cited by: §2.3, Table 2. Choi et al. (2009) W. Choi, K. Shahid, and S. Savarese What are they doing? : collective activity classification using spatio-temporal relationship among people. In IEEE International Conference on Computer Vision Workshops, ICCV Workshops, Vol. , p. 1282–1289. Cited by: 3rd item, §2.3, Table 1, §4.1, §4.1, item 4. Ci et al. (2023) Y. Ci, Y. Wang, M. Chen, S. Tang, L. Bai, F. Zhu, R. Zhao, F. Yu, D. Qi, and W. Ouyang UniHCP: a unified model for human-centric perceptions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 17840–17852. Cited by: §1. Do and Kim (2024) J. Do and M. Kim SkateFormer: skeletal-temporal transformer for human action recognition. In European Conference on Computer Vision (ECCV), Cited by: §2.2. Du et al. (2024) W. Du, Q. Lyu, J. Shan, Z. Qi, H. Zhang, S. Chen, A. Peng, T. Shu, K. Lee, B. Dariush, and C. Gan Constrained human-ai cooperation: an inclusive embodied social intelligence challenge. In Annual Conference on Neural Information Processing Systems (NeurIPS), Cited by: §1. Duan et al. (2022) H. Duan, J. Wang, K. Chen, and D. Lin Pyskl: towards good practices for skeleton action recognition. In ACM International Conference on Multimedia (ACMMM), p. 7351–7354. Cited by: §2.2. Duan et al. (2023) H. Duan, M. Xu, B. Shuai, D. Modolo, Z. Tu, J. Tighe, and A. Bergamo SkeleTR: towards skeleton-based action recognition in the wild. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 13634–13644. Cited by: §2.3, Table 2. Ganin et al. (2016) Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky Domain-adversarial training of neural networks. Journal of machine learning research 17 (59), p. 1–35. Cited by: §2.1. Garcia-Hernando et al. (2018) G. Garcia-Hernando, S. Yuan, S. Baek, and T. Kim First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §2.3. Gavrilyuk et al. (2020) K. Gavrilyuk, R. Sanford, M. Javan, and C. G. M. Snoek Actor-transformers for group activity recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.3. Guo et al. (2022) W. Guo, X. Bie, X. Alameda-Pineda, and F. Moreno-Noguer Multi-person extreme motion prediction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 13053–13064. Cited by: §2.3. Han et al. (2022) M. Han, D. J. Zhang, Y. Wang, R. Yan, L. Yao, X. Chang, and Y. Qiao Dual-ai: dual-path actor interaction learning for group activity recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 2990–2999. Cited by: §2.3. Hao et al. (2021) X. Hao, J. Li, Y. Guo, T. Jiang, and M. Yu Hypergraph neural network for skeleton-based action recognition. IEEE Transactions on Image Processing 30 (), p. 2263–2275. Cited by: §2.2. He et al. (2024) T. He, Y. Chen, X. Gao, L. Wang, T. Hu, and H. Cheng Enhancing skeleton-based action recognition with language descriptions from pre-trained large multimodal models. IEEE Transactions on Circuits and Systems for Video Technology (), p. 1–1. Cited by: §2.2. Hu et al. (2018) J. Hu, L. Shen, and G. Sun Squeeze-and-excitation networks. In Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 7132–7141. Cited by: §3.2. Hu and Liu (2024) Q. Hu and H. Liu Multi-modal enhancement transformer network for skeleton-based human interaction recognition. Biomimetics 9 (3). Cited by: Table 2. Huang et al. (2023) X. Huang, H. Zhou, B. Feng, X. Wang, W. Liu, J. Wang, H. Feng, J. Han, E. Ding, and J. Wang Graph contrastive learning for skeleton-based action recognition. In International Conference on Learning Representations (ICLR), Cited by: §2.2. Ibrahim et al. (2016) M. S. Ibrahim, S. Muralidharan, Z. Deng, A. Vahdat, and G. Mori A hierarchical deep temporal model for group activity recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 1971–1980. Cited by: 3rd item, §2.3, Table 1, §4.1, §4.1, item 5. Ioffe and Szegedy (2015a) S. Ioffe and C. Szegedy Batch normalization: accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 37, p. 448–456. Cited by: §4.3. Ioffe and Szegedy (2015b) S. Ioffe and C. Szegedy Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Proceedings of Machine Learning Research, Vol. 37, p. 448–456. Cited by: §2.1. Jahangard et al. (2024) S. Jahangard, Z. Cai, S. Wen, and H. Rezatofighi JRDB-social: a multifaceted robotic dataset for understanding of context and dynamics of human interactions within social groups. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1. Ji et al. (2014) Y. Ji, G. Ye, and H. Cheng Interactive body part contrast mining for human interaction recognition. In IEEE International Conference on Multimedia and Expo Workshops (ICMEW), Vol. , p. 1–6. Cited by: §2.3. Jiang et al. (2024) N. Jiang, Z. Zhang, H. Li, X. Ma, Z. Wang, Y. Chen, T. Liu, Y. Zhu, and S. Huang Scaling up dynamic human-scene interaction modeling. arXiv:2403.08629. Cited by: §1. Khirodkar et al. (2024) R. Khirodkar, T. Bagautdinov, J. Martinez, S. Zhaoen, A. James, P. Selednik, S. Anderson, and S. Saito Sapiens: foundation for human vision models. In European Conference on Computer Vision (ECCV), Cited by: §1. Kwon et al. (2021) T. Kwon, B. Tekin, J. Stühmer, F. Bogo, and M. Pollefeys H2O: two hands manipulating objects for first person interaction recognition. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 10138–10148. Cited by: 3rd item, §1, §2.3, Table 1, §4.1, §4.1, item 2. Lan et al. (2012) T. Lan, Y. Wang, W. Yang, S. N. Robinovitch, and G. Mori Discriminative latent models for recognizing contextual group activities. IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (8), p. 1549–1562. Cited by: §2.3. Lee et al. (2023) J. Lee, M. Lee, D. Lee, and S. Lee Hierarchically decomposed graph convolutional networks for skeleton-based action recognition. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 10444–10453. Cited by: §1, §2.2, §2.2, §2.2, §3.1, §3.5, §4.2, Table 2. Li et al. (2019) M. Li, S. Chen, X. Chen, Y. Zhang, Y. Wang, and Q. Tian Actional-structural graph convolutional networks for skeleton-based action recognition. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 3590–3598. Cited by: §2.2. Li et al. (2023a) S. Li, X. He, W. Song, A. Hao, and H. Qin Graph diffusion convolutional network for skeleton based semantic recognition of two-person actions. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (7), p. 8477–8493. Cited by: §2.3, Table 2. Li et al. (2021) S. Li, Q. Cao, L. Liu, K. Yang, S. Liu, J. Hou, and S. Yi GroupFormer: group activity recognition with clustered spatial-temporal transformer. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 13668–13677. Cited by: §2.3. Li et al. (2023b) Z. Li, X. Gong, R. Song, P. Duan, J. Liu, and W. Zhang SMAM: self and mutual adaptive matching for skeleton-based few-shot action recognition. IEEE Transactions on Image Processing 32 (), p. 392–402. Cited by: §1. Liang et al. (2024) H. Liang, W. Zhang, W. Li, J. Yu, and L. Xu InterGen: diffusion-based multi-human motion generation under complex interactions. International Journal of Computer Vision. Cited by: §2.3. Liu et al. (2017a) C. Liu, Y. Hu, Y. Li, S. Song, and J. Liu PKU-mmd: a large scale benchmark for skeleton-based human action understanding. In Proceedings of the Workshop on Visual Analysis in Smart and Connected Communities, VSCC ’17, p. 1–8. Cited by: §1. Liu et al. (2023) D. Liu, P. Chen, M. Yao, Y. Lu, Z. Cai, and Y. Tian TSGCNeXt: dynamic-static multi-graph convolution for efficient skeleton-based action recognition with long-term learning potential. arXiv:2304.11631. Cited by: §2.2. Liu et al. (2024a) H. Liu, Y. Li, T. Mu, and S. Hu Recovering complete actions for cross-dataset skeleton action recognition. In Annual Conference on Neural Information Processing Systems (NeurIPS), Cited by: §2.2. Liu et al. (2024b) J. Liu, C. Chen, and M. Liu Multi-modality co-learning for efficient skeleton-based action recognition. In ACM International Conference on Multimedia (ACMMM), Cited by: §4.5. Liu et al. (2020a) J. Liu, A. Shahroudy, M. Perez, G. Wang, L. Duan, and A. C. Kot NTU rgb+d 120: a large-scale benchmark for 3d human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence 42 (10), p. 2684–2701. Cited by: 3rd item, §1, §2.3, §3.2, Table 1, §4.1, §4.1, §4.5, item 1. Liu et al. (2016) J. Liu, A. Shahroudy, D. Xu, and G. Wang Spatio-temporal lstm with trust gates for 3d human action recognition. In European Conference on Computer Vision (ECCV), p. 816–833. Cited by: §2.2. Liu et al. (2018) J. Liu, G. Wang, L. Duan, K. Abdiyeva, and A. C. Kot Skeleton-based human action recognition with global context-aware attention lstm networks. IEEE Transactions on Image Processing 27 (4), p. 1586–1599. Cited by: §2.2. Liu et al. (2017b) J. Liu, G. Wang, P. Hu, L. Duan, and A. C. Kot Global context-aware attention lstm networks for 3d action recognition. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 3671–3680. Cited by: §2.2. Liu et al. (2025) M. Liu, C. Chen, S. Wu, F. Meng, and H. Liu Learning mutual excitation for hand-to-hand and human-to-human interaction recognition. IEEE Transactions on Human-Machine Systems (), p. 1–10. Cited by: §2.3, §4.3, §4.5, Table 2. Liu et al. (2020b) Z. Liu, H. Zhang, Z. Chen, Z. Wang, and W. Ouyang Disentangling and unifying graph convolutions for skeleton-based action recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 140–149. Cited by: §2.2. Long et al. (2015) M. Long, Y. Cao, J. Wang, and M. Jordan Learning transferable features with deep adaptation networks. In International conference on machine learning (ICML), p. 97–105. Cited by: §2.1. Long (2023) N. H. B. Long STEP catformer: spatial-temporal effective body-part cross attention transformer for skeleton-based action recognition. arXiv:2312.03288. Cited by: §2.2. Mao et al. (2023) Y. Mao, J. Deng, W. Zhou, Y. Fang, W. Ouyang, and H. Li Masked motion predictors are strong 3d action representation learners. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 10181–10191. Cited by: §2.2. McInnes et al. (2018) L. McInnes, J. Healy, N. Saul, and L. Grossberger UMAP: uniform manifold approximation and projection. The Journal of Open Source Software 3 (29), p. 861. Cited by: Figure 6. Mucha and Kampel (2024) W. Mucha and M. Kampel In my perspective, in my hands: accurate egocentric 2d hand pose and action recognition. arXiv:2404.09308. Cited by: §2.3, Table 2. Myung et al. (2024) W. Myung, N. Su, J. Xue, and G. Wang DeGCN: deformable graph convolutional networks for skeleton-based action recognition. IEEE Transactions on Image Processing 33 (), p. 2477–2490. Cited by: 2nd item, 3rd item, §1, §2.2, §2.2, 4th item, §3.1, §3.5, §4.2, §4.3, §4.5, §4.5, Table 2. Ohkawa et al. (2023) T. Ohkawa, K. He, F. Sener, T. Hodan, L. Tran, and C. Keskin AssemblyHands: towards egocentric activity understanding via 3d hand pose estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 12999–13008. Cited by: §1, §2.3. Pang et al. (2025) C. Pang, X. Lu, Q. Zhou, and L. Lyu Learning adaptive node selection with external attention for human interaction recognition. In Proceedings of the 33rd ACM International Conference on Multimedia, M ’25, New York, NY, USA, p. 7297–7306. External Links: ISBN 9798400720352 Cited by: §2.3, §4.3, Table 2. Pang et al. (2022) Y. Pang, Q. Ke, H. Rahmani, J. Bailey, and J. Liu IGFormer: interaction graph transformer for skeleton-based human interaction recognition. In European Conference on Computer Vision (ECCV), p. 605–622. Cited by: §2.3, §4.3, §4.5, Table 2. Peng et al. (2024) K. Peng, C. Yin, J. Zheng, R. Liu, D. Schneider, J. Zhang, K. Yang, M. S. Sarfraz, R. Stiefelhagen, and A. Roitberg Navigating open set scenarios for skeleton-based action recognition. AAAI Conference on Artificial Intelligence 38 (5), p. 4487–4496. Cited by: §2.2. Perez et al. (2022) M. Perez, J. Liu, and A. C. Kot Interaction relational network for mutual action recognition. IEEE Transactions on Multimedia 24 (), p. 366–376. Cited by: §2.3, Table 2. Plizzari et al. (2024) C. Plizzari, G. Goletto, A. Furnari, S. Bansal, F. Ragusa, G. M. Farinella, D. Damen, and T. Tommasi An outlook into the future of egocentric vision. International Journal of Computer Vision, p. 1573–1405. Cited by: §1. Qiu et al. (2023) H. Qiu, B. Hou, B. Ren, and X. Zhang Spatio-temporal segments attention for skeleton-based action recognition. Neurocomputing 518, p. 30–38. External Links: ISSN 0925-2312 Cited by: §1, §2.2, §2.2, §2.2, §3.1, §3.5, §4.2, Table 2. Rajasegaran et al. (2023) J. Rajasegaran, G. Pavlakos, A. Kanazawa, C. Feichtenhofer, and J. Malik On the benefits of 3d pose and tracking for human action recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 640–649. Cited by: §1. Reilly and Das (2024) D. Reilly and S. Das Just add π! pose induced video transformers for understanding activities of daily living. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 18340–18350. Cited by: §1. Ren et al. (2024) H. Ren, Y. Zhou, H. FU, Y. Huang, X. LIN, J. Song, and B. Cheng SpikePoint: an efficient point-based spiking neural network for event cameras action recognition. In International Conference on Learning Representations (ICLR), Cited by: §1. Rockafellar (1996) R. T. Rockafellar Convex analysis. Princeton Landmarks in Mathematics and Physics, Princeton University Press, Princeton, NJ (en). Cited by: §3.2. Sampieri et al. (2022) A. Sampieri, G. M. D. di Melendugno, A. Avogaro, F. Cunico, F. Setti, G. Skenderi, M. Cristani, and F. Galasso Pose forecasting in industrial human-robot collaboration. In European Conference on Computer Vision (ECCV), p. 51–69. Cited by: §1, §2.3. Sener et al. (2022) F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao Assembly101: a large-scale multi-view video dataset for understanding procedural activities. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 21064–21074. Cited by: 3rd item, §1, §2.3, Table 1, §4.1, §4.1, item 3. Shahroudy et al. (2016) A. Shahroudy, J. Liu, T. Ng, and G. Wang NTU rgb+d: a large scale dataset for 3d human activity analysis. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 1010–1019. Cited by: 3rd item, §2.3, Table 1, §4.1, §4.1, item 1. Shamil et al. (2024) M. S. Shamil, D. Chatterjee, F. Sener, S. Ma, and A. Yao On the utility of 3d hand poses for action recognition. In European Conference on Computer Vision (ECCV), Cited by: §2.3, §4.3, Table 2. Shi et al. (2019) L. Shi, Y. Zhang, J. Cheng, and H. Lu Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 12018–12027. Cited by: §2.2, §2.2. Shi et al. (2020) L. Shi, Y. Zhang, J. Cheng, and H. Lu Decoupled spatial-temporal attention network for skeleton-based action-gesture recognition. In 15th Asian Conference on Computer Vision (ACCV), p. 38–53. Cited by: §1, §2.2, §2.2, §2.2, §3.1, §3.5. Srivastava et al. (2015) R. K. Srivastava, K. Greff, and J. Schmidhuber Training very deep networks. In Twenty-eighth Conference on Neural Information Processing Systems (NeurIPS), Vol. 28. Cited by: §3.2. Tamura et al. (2022) M. Tamura, R. Vishwakarma, and R. Vennelakanti Hunting group clues with transformers for social group activity recognition. In European Conference on Computer Vision (ECCV), p. 19–35. Cited by: §2.3. Tamura (2024) M. Tamura Design and analysis of efficient attention in transformers for social group activity recognition. International Journal of Computer Vision 132 (10), p. 4269–4288. Cited by: §2.3. Tang et al. (2023) S. Tang, C. Chen, Q. Xie, M. Chen, Y. Wang, Y. Ci, L. Bai, F. Zhu, H. Yang, L. Yi, R. Zhao, and W. Ouyang HumanBench: towards general human-centric perception with projector assisted pretraining. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 21970–21982. Cited by: §1. Tekin et al. (2019) B. Tekin, F. Bogo, and M. Pollefeys H+o: unified egocentric recognition of 3d hand-object poses and interactions. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. , p. 4506–4515. Cited by: §2.3. Thilakarathne et al. (2022) H. Thilakarathne, A. Nibali, Z. He, and S. Morgan Pose is all you need: the pose only group activity recognition system (POGARS). Machine Vision and Applications 33 (6), p. 95. Cited by: §2.3. Trivedi and Sarvadevabhatla (2022) N. Trivedi and R. K. Sarvadevabhatla PSUMNet: unified modality part streams are all you need for efficient pose-based action recognition. In European Conference on Computer Vision (ECCV), p. 211–227. Cited by: §2.2, §2.2. Ulyanov et al. (2016) D. Ulyanov, A. Vedaldi, and V. Lempitsky Instance normalization: the missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022. Cited by: §2.1. Wang et al. (2024) G. Wang, M. Liu, H. Liu, P. Guo, T. Wang, J. Guo, and R. Fan Augmented skeleton sequences with hypergraph network for self-supervised group activity recognition. Pattern Recognition 152, p. 110478. Cited by: §2.3, Table 2. Wang et al. (2025) H. Wang, W. Weng, J. Wang, F. Zhao, G. Xie, X. Geng, and L. Wang Foundation model for skeleton-based human action understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §1. Wang et al. (2014) J. Wang, X. Nie, Y. Xia, Y. Wu, and S. Zhu Cross-view action modeling, learning, and recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), p. 2649–2656. Cited by: §1. Wang et al. (2023) Q. Wang, S. Shi, J. He, J. Peng, T. Liu, and R. Weng IIP-transformer: intra-inter-part transformer for skeleton-based action recognition. In IEEE International Conference on Big Data (BigData), Vol. , p. 936–945. Cited by: §2.2. Wanyan et al. (2025) Y. Wanyan, X. Yang, W. Dong, and C. Xu A comprehensive review of few-shot action recognition. International Journal of Computer Vision 133 (10), p. 6832–6859. External Links: ISSN 1573-1405 Cited by: §1. Wen et al. (2023a) Y. Wen, H. Pan, L. Yang, J. Pan, T. Komura, and W. Wang Hierarchical temporal transformer for 3d hand pose estimation and action recognition from egocentric rgb videos. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 21243–21253. Cited by: §2.3. Wen et al. (2024) Y. Wen, M. Liu, S. Wu, and B. Ding CHASE: learning convex hull adaptive shift for skeleton-based multi-entity action recognition. In Annual Conference on Neural Information Processing Systems (NeurIPS), Cited by: 1st item, 2nd item, 3rd item, §1, §4.3. Wen et al. (2023b) Y. Wen, Z. Tang, Y. Pang, B. Ding, and M. Liu Interactive spatiotemporal token attention network for skeleton-based general interactive action recognition. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. , p. 7886–7892. Cited by: §2.3, §4.2, §4.3, §4.3, §4.5, Table 2. Wu et al. (2019) J. Wu, L. Wang, L. Wang, J. Guo, and G. Wu Learning actor relation graphs for group activity recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.3. Wu and He (2018) Y. Wu and K. He Group normalization. In Proceedings of the European conference on computer vision (ECCV), p. 3–19. Cited by: §2.1. Xiang et al. (2023) W. Xiang, C. Li, Y. Zhou, B. Wang, and L. Zhang Generative action description prompts for skeleton-based action recognition. In IEEE/CVF International Conference on Computer Vision (ICCV), p. 10276–10285. Cited by: §2.2, §3.4. Xie et al. (2025) J. Xie, Y. Zhao, Y. Meng, H. Zhao, A. Nguyen, and Y. Zheng Are spatial-temporal graph convolution networks for human action recognition over-parameterized?. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 24309–24319. Cited by: §2.2. Xu et al. (2023) H. Xu, Y. Gao, Z. Hui, J. Li, and X. Gao Language knowledge-assisted representation learning for skeleton-based action recognition. CoRR abs/2305.12398. Cited by: §2.2. Xu et al. (2024) L. Xu, X. Lv, Y. Yan, X. Jin, S. Wu, C. Xu, Y. Liu, Y. Zhou, F. Rao, X. Sheng, Y. Liu, W. Zeng, and X. Yang Inter-x: towards versatile human-human interaction analysis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.3. Yan et al. (2018) S. Yan, Y. Xiong, and D. Lin Spatial temporal graph convolutional networks for skeleton-based action recognition. In AAAI Conference on Artificial Intelligence, AAAI’18. Cited by: §2.2. Yang et al. (2024a) C. Yang, B. Indurkhya, J. See, B. Gao, Y. Ke, Z. Boukhers, Z. Yang, and M. Grzegorzek Skeleton ground truth extraction: methodology, annotation tool and benchmarks. International Journal of Computer Vision 132 (4), p. 1219–1241. Cited by: §1. Yang et al. (2024b) D. Yang, Y. Wang, A. Dantcheva, L. Garattoni, G. Francesca, and F. Bremond View-invariant skeleton action representation learning via motion retargeting. International Journal of Computer Vision 132 (7), p. 2351–2366. Cited by: §1. Yin et al. (2023) Y. Yin, C. Guo, M. Kaufmann, J. J. Zarate, J. Song, and O. Hilliges Hi4D: 4d instance segmentation of close human interaction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 17016–17027. Cited by: §2.3. Yuan et al. (2021) H. Yuan, D. Ni, and M. Wang Spatio-temporal dynamic inference network for group activity recognition. In IEEE/CVF International Conference on Computer Vision (ICCV), Vol. , p. 7456–7465. Cited by: §2.3. Yuan and Ni (2021) H. Yuan and D. Ni Learning visual context for group activity recognition. AAAI Conference on Artificial Intelligence 35 (4), p. 3261–3269. Cited by: §2.3. Yun et al. (2012) K. Yun, J. Honorio, D. Chattopadhyay, T. L. Berg, and D. Samaras Two-person interaction detection using body-pose features and multiple instance learning. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Vol. , p. 28–35. Cited by: §2.3. Zhang et al. (2024) J. Zhang, L. Lin, and J. Liu Shap-mix: shapley value guided mixing for long-tailed skeleton based action recognition. In International Joint Conference on Artificial Intelligence, IJCAI-24, p. 1688–1696. Cited by: §2.2, §4.5. Zhang et al. (2017) P. Zhang, C. Lan, J. Xing, W. Zeng, J. Xue, and N. Zheng View adaptive recurrent neural networks for high performance human action recognition from skeleton data. In IEEE/CVF International Conference on Computer Vision (ICCV), Vol. , p. 2136–2145. Cited by: §2.2. Zhang et al. (2019) P. Zhang, C. Lan, J. Xing, W. Zeng, J. Xue, and N. Zheng View adaptive neural networks for high performance skeleton-based human action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (8), p. 1963–1978. Cited by: §2.2. Zhou et al. (2022a) H. Zhou, A. Kadav, A. Shamsian, S. Geng, F. Lai, L. Zhao, T. Liu, M. Kapadia, and H. P. Graf COMPOSER: compositional reasoning of group activity in videos with keypoint-only modality. In European Conference on Computer Vision (ECCV), p. 249–266. Cited by: §2.3, §4.1, §4.1. Zhou et al. (2025) Y. Zhou, T. Xu, C. Wu, X. Wu, and J. Kittler Adaptive hyper-graph convolution network for skeleton-based human action recognition with virtual connections. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 12648–12658. Cited by: §4.2, Table 2. Zhou et al. (2022b) Y. Zhou, Z. Cheng, C. Li, Y. Geng, X. Xie, and M. Keuper Hypergraph transformer for skeleton-based action recognition. arXiv:2211.09590. Cited by: §2.2. Zhou et al. (2024) Y. Zhou, X. Yan, Z. Cheng, Y. Yan, Q. Dai, and X. Hua BlockGCN: redefine topology awareness for skeleton-based action recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 2049–2058. Cited by: §2.2. Zhu et al. (2025) F. Zhu, Y. Xie, W. Xie, and H. Jiang Diagnosing human-object interaction detectors. International Journal of Computer Vision, p. 1–18. Cited by: §2.3. Zhu et al. (2016) W. Zhu, C. Lan, J. Xing, W. Zeng, Y. Li, L. Shen, and X. Xie Co-occurrence feature learning for skeleton based action recognition using regularized deep lstm networks. In AAAI Conference on Artificial Intelligence, AAAI’16, p. 3697–3703. Cited by: §2.2.