Paper deep dive
Identity-Aware Human-Object Interaction Motion Captioning
Yiming Wang, Yonghao Dang, Huilai Li, Jiawei Tu, Jianqin Yin
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/24/2026, 5:06:58 AM
Summary
This paper introduces Identity-Aware Human-Object Interaction (HOI) Motion Captioning, a new task that generates captions specifying both the subject's identity and the interaction motion, addressing the limitation of existing methods that use generic terms like 'a person'. The authors propose ID-HOINet, a model comprising a Multi-View Identity-Motion Learning Module (MVIML) and a Two-Stage Caption Rewriting Strategy (TSCR). MVIML extracts features from multi-view videos, while TSCR retrieves subject identity via gallery matching and rewrites identity-agnostic captions with the specific identity. The method is evaluated on datasets derived from BEHAVE and InterCap, achieving state-of-the-art performance.
Entities (8)
Relation Signals (6)
ID-HOINet → containsmodule → MVIML
confidence 98% · ID-HOINet contains two core components: Multi-View Identity-Motion Learning Module (MVIML) and Two-Stage Caption Rewriting Strategy (TSCR).
ID-HOINet → containsmodule → TSCR
confidence 98% · ID-HOINet contains two core components: Multi-View Identity-Motion Learning Module (MVIML) and Two-Stage Caption Rewriting Strategy (TSCR).
TSCR → usesfor → Identity-Aware Human-Object Interaction Motion Captioning
confidence 95% · At inference, the TSCR first retrieves the subject identity and generates identity-agnostic HOI motion captions. TSCR then rewrites these captions with the predicted identity to produce the final identity-aware HOI motion captions.
ID-HOINet → achievesstateofthearton → Identity-Aware Human-Object Interaction Motion Captioning
confidence 92% · Experiments demonstrate that ID-HOINet achieves state-of-the-art performance.
MVIML → processesinputfrom → BEHAVE
confidence 90% · For this task, we design identity-aware HOI motion captions based on the BEHAVE and InterCap datasets. ... MVIML learns from multi-view videos
MVIML → processesinputfrom → InterCap
confidence 90% · For this task, we design identity-aware HOI motion captions based on the BEHAVE and InterCap datasets. ... MVIML learns from multi-view videos
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Existing human-object interaction (HOI) motion captioning methods typically describe what happens while referring to the subject using generic terms such as "a person" or "someone", without grounding the caption in subject identity. To address this limitation, we introduce Identity-Aware Human-Object Interaction Motion Captioning task. This task requires each generated caption to specify both the subject identity and the corresponding HOI motion. For example, the model generates "Sub_ID lifts the chair" rather than "A person lifts the chair". For this task, we design identity-aware HOI motion captions based on the BEHAVE and InterCap datasets. We further propose ID-HOINet, which learns from multi-view videos while supporting single-view identity-aware HOI motion caption generation. ID-HOINet contains two core components: Multi-View Identity-Motion Learning Module (MVIML) and Two-Stage Caption Rewriting Strategy (TSCR). MVIML learns from multi-view videos by modeling dependencies across temporal stages and camera viewpoints, capturing identity and interaction motion features. At inference, the TSCR first retrieves the subject identity and generates identity-agnostic HOI motion captions. TSCR then rewrites these captions with the predicted identity to produce the final identity-aware HOI motion captions. Experiments demonstrate that ID-HOINet achieves state-of-the-art performance. Code will be released upon acceptance.
Tags
Links
- Source: https://arxiv.org/abs/2608.20690v1
- Canonical: https://arxiv.org/abs/2608.20690v1
Trouble viewing inline? Open PDF directly →
Full Text
39,602 characters extracted from source content.
Expand or collapse full text
Identity-Aware Human-Object Interaction Motion Captioning Yiming Wang Yonghao Dang Huilai Li Jiawei Tu Jianqin Yin Abstract Existing human-object interaction (HOI) motion captioning methods typically describe what happens while referring to the subject using generic terms such as “a person” or “someone”, without grounding the caption in subject identity. To address this limitation, we introduce Identity-Aware Human-Object Interaction Motion Captioning task. This task requires each generated caption to specify both the subject identity and the corresponding HOI motion. For example, the model generates “Sub_ID lifts the chair” rather than “A person lifts the chair”. For this task, we design identity-aware HOI motion captions based on the BEHAVE and InterCap datasets. We further propose ID-HOINet, which learns from multi-view videos while supporting single-view identity-aware HOI motion caption generation. ID-HOINet contains two core components: Multi-View Identity-Motion Learning Module (MVIML) and Two-Stage Caption Rewriting Strategy (TSCR). MVIML learns from multi-view videos by modeling dependencies across temporal stages and camera viewpoints, capturing identity and interaction motion features. At inference, the TSCR first retrieves the subject identity and generates identity-agnostic HOI motion captions. TSCR then rewrites these captions with the predicted identity to produce the final identity-aware HOI motion captions. Experiments demonstrate that ID-HOINet achieves state-of-the-art performance. Code will be released upon acceptance. Introduction Human-Object interaction (HOI) motion captioning aims to describe the interaction relation between a human and an object using natural language. Recent captioning approaches have achieved significant progress in formulating coherent natural-language captions from visual motion inputs (20; 10; 27; 22). However, these methods typically generate identity-agnostic HOI motion captions, such as “A person picks up a box” or “Someone sits on a chair”. Such captions describe what interaction occurs but do not specify who performs it. This limitation is critical in identity-dependent scenarios, such as personalized rehabilitation monitoring (8; 7; 5) and multi-subject motion analysis (4; 23; 11), where the observed motion must be associated with the correct individual. Recently, multiple identity-aware video captioning techniques have been introduced to link captions with their respective subject identities within cinematic and sports videos (15; 16; 24). However, these methods mainly describe subjects performing actions that involve a fixed object or no explicit object interaction. In complex human-object interaction scenarios, the captions generated by these methods fail to explicitly associate the subject identity with the corresponding HOI motion. Figure 1: Motivation of identity-aware HOI motion captioning. Existing methods generate identity-agnostic captions that only describe what happens, while our work further identifies who performs the motion, enabling who-and-what motion captioning. Accordingly, as illustrated in Figure 1, we propose Identity-Aware Human-Object Interaction Motion Captioning task. To the best of our knowledge, this is the first study to extend identity-aware video captioning to the human-object interaction motion task. Unlike prior work focused on movies and sports videos, our generated caption explicitly associates the subject identity with the corresponding HOI motion. Given a multi-view HOI video, this task requires the model to generate an identity-aware HOI motion caption such as “Sub_ID lifts the chair”, rather than the identity-agnostic caption “A person lifts the chair”. To construct an identity-aware HOI motion captioning dataset, we reorganize the multi-view video data from BEHAVE (2) and InterCap (6). We divide the synchronized videos into temporally complete HOI segments and provide each segment with an identity-aware HOI motion caption annotation that explicitly associates the observed subject identity with the corresponding interaction motion. The annotated HOI segments are then partitioned by subject identity into disjoint training and test sets, ensuring that no test subject appears in the training identity set. During testing, the model is required to determine the identity of each subject by matching a single-view query video to reference subject samples in a gallery (32) and incorporates the retrieved identity into the generated HOI motion caption. To this end, we propose ID-HOINet for identity-aware human-object interaction motion captioning. ID-HOINet employs a Multi-View Identity-Motion Learning Module (MVIML) to learn identity and interaction motion features from multi-view videos. During inference, ID-HOINet then adopts a Two-Stage Caption Rewriting Strategy (TSCR) that first determines the subject identity and generates an identity-agnostic HOI motion caption, and subsequently combines them through a pretrained Caption Rewriting Decoder to produce the final identity-aware HOI motion caption. Extensive experiments demonstrate that ID-HOINet achieves state-of-the-art performance on the proposed task, showing its effectiveness in associating subject identities with the corresponding HOI motion captions. The main contributions of this work are summarized as follows: • We introduce Identity-Aware Human-Object Interaction Motion Captioning, a new task that requires the generated caption to explicitly associate subject identity with the corresponding HOI motion. • We propose ID-HOINet, which comprises two core components: the Multi-View Identity-Motion Learning Module (MVIML) and the Two-Stage Caption Rewriting Strategy (TSCR). MVIML captures identity and motion cues from multi-view videos. At test time, TSCR first matches the query subject to a reference gallery and generates an identity-agnostic HOI motion caption, then integrates the retrieved identity via a pretrained Caption Rewriting Decoder to produce the final identity-aware HOI motion caption. • Extensive experiments demonstrate that ID-HOINet achieves state-of-the-art performance under the proposed evaluation setting, validating the effectiveness of the proposed framework. Related Work Human-Object Interaction Motion Captioning Human-object interaction (HOI) motion captioning aims to generate a caption that expresses how a subject interacts with an object. To capture the visual information required for HOI motion features, related methods focus on modeling object interactions, human motion, and general video understanding. For interaction modeling, SINet-Caption jointly models HOI relations among multiple objects for fine-grained caption generation (12), while SAVCHOI incorporates HOI features into dense captioning for surveillance videos (13). For motion modeling, existing methods introduce pose-based angular features (31) or human-mesh motion features (19) to capture fine-grained body movements. General video captioners further improve video-to-language generation through CLIP-based visual-language knowledge (20), sparse video attention (10), or unified sequence modeling (27; 22). These methods generally generate identity-agnostic HOI motion captions that specify what HOI motion is performed but not who performs it. In contrast, our task requires the generated caption to explicitly associate the subject identity with the corresponding HOI motion. Identity-Aware Video Captioning Identity-aware video captioning extends conventional video captioning by requiring the generated descriptions to not only express the observed events but also associate their corresponding subject identities. Existing studies mainly investigate this problem in films and sports events. Specifically, Park et al. (15) study identity-aware multi-sentence movie description using a two-stage framework that first generates anonymous descriptions and then resolves character identities across multiple video segments. MICap (16) further unifies identity-aware video caption generation and identity filling within a shared autoregressive framework, enabling both complete caption generation and identity recognition from partially anonymized descriptions. In sports videos, Xi et al. (24) introduce player-centric visual features and multimodal prompts to associate basketball-related event descriptions with specific players. Existing identity-aware video captioning methods are not specifically designed for temporally complete human–object interaction motions involving diverse object categories. In contrast, our task involves a subject interacting with varying objects, requiring the model to jointly reason about identity and HOI motion. Dataset Processing Source Datasets We use BEHAVE (2) and InterCap (6) as the source datasets. Both datasets capture real human-object interactions using synchronized multi-view cameras and provide subject identities and interaction motions. However, they do not provide identity-aware HOI motion captions and therefore cannot be directly used for our task. To facilitate model training for this task, we restructure the source datasets through the following steps. Temporal Video Segmentation A temporally complete human-object interaction process serves as the basic data unit in our task. Following the temporal identity-agnostic annotations of InterAct (26), each synchronized multi-view HOI video is divided according to its annotated start and end timestamps. Every resulting segment is associated with an identity-agnostic motion caption describing the interaction performed within that temporal interval. Identity-Aware HOI Motion Annotation Identity-aware HOI motion captioning requires an explicit correspondence between the physical participant and the performed interaction. Accordingly, the subjects in BEHAVE and InterCap are mapped to a unified set of 18 global identity labels, denoted as Sub01 to Sub18. Each label represents one subject and remains consistent across all motion segments and camera views. DeepSeek-V4-Pro (25) is then employed to rewrite the identity-agnostic caption of each segment into an identity-aware HOI motion caption. Specifically, the generic subject expression is replaced with the corresponding global identity label, while the principal interaction motion and object are preserved. Finally, all captions are manually inspected and corrected to ensure consistency between the subject identity and interaction semantics. Data Splits The data partition is designed to support multi-view feature learning and gallery-based identity-aware HOI motion captioning evaluation. As summarized in Table 1, the processed dataset contains 2,073 multi-view HOI segments from 18 subjects. Among them, 1,700 segments from 10 subjects are used for training, while the remaining 373 segments from 8 subjects are assigned to testing. For each test subject, one segment is selected as the gallery, resulting in 8 gallery segments, and the remaining 365 segments form the query set. During training, the model receives multi-view inputs to learn identity and interaction motion features. During testing, each query is processed from a single view, and its identity is determined by matching its feature against the multi-view gallery samples. Although each query segment contains four synchronized camera views, testing is performed under a single-view setting. Specifically, the four views of each query segment are unfolded and evaluated independently, resulting in 365×4=1,460365× 4=1,460 single-view query instances. Captioning and identity-recognition metrics are computed over all 1,460 view-level predictions. Statistic Training Testing Query Gallery Segments 1,700 365 8 Subjects 10 8 8 Views Per Segment 4 4 4 Table 1: Statistics of the training, query, and gallery sets for identity-aware HOI motion captioning. Method Figure 2: Overview of the proposed ID-HOINet. During training, synchronized multi-view videos are processed by a frozen CLIP visual encoder, and the Multi-View Identity-Motion Learning Module learns identity and interaction motion features for identity recognition and identity-agnostic caption generation. During testing, the Two-Stage Caption Rewriting Strategy first determines the query identity through gallery matching and generates an identity-agnostic motion caption, then rewrites the caption with the determined identity using a pretrained frozen Caption Rewriting Decoder. Overview During training, ID-HOINet takes synchronized multi-view HOI videos as input. The Video Encoder extracts visual features from sampled frames, which are subsequently processed by the Multi-View Identity-Motion Learning Module (MVIML) to learn identity and interaction motion features. The Identity Decoder determines the subject identity, while the Identity-Agnostic Caption Decoder generates the corresponding caption. During testing, the proposed Two-Stage Caption Rewriting Strategy (TSCR) operates in two stages. In the first stage, the gallery samples and each single-view query are processed by the shared Video Encoder and Multi-View Identity-Motion Learning Module (MVIML). The query identity is determined through gallery matching, while an identity-agnostic motion caption is generated independently. In the second stage, the determined identity and generated motion caption are integrated by the pretrained frozen Caption Rewriting Decoder to produce the final identity-aware HOI motion caption. Input Representation Given a multi-view HOI clip, we first sample frames from all camera views. Each frame corresponds to the same moment observed from different viewpoints. The sampled frames are encoded by a frozen CLIP (17) visual encoder to obtain multi-view visual feature FmvF_mv: Fmv=ECLIP(ℐ),F_mv=E_CLIP(I), (1) where B, T, D, and V denote the batch size, temporal length, feature dimension, and number of camera views, respectively. Fmv∈ℝB×T×V×DF_mv ^B× T× V× D provides a multi-view visual feature for subsequent identity-aware motion captioning learning. Multi-View Identity-Motion Learning Multi-view observations capture consistent subject appearance and human-object motion cues across temporal stages and viewpoints. Based on these physical characteristics, the Multi-View Identity-Motion Learning Module (MVIML) transforms FmvF_mv into a shared temporal-view feature and employs two independent branches to learn identity and interaction motion features. Multi-View Representation. To distinguish the temporal and viewpoint positions associated with these observations, MVIML projects FmvF_mv into a shared token space and incorporates temporal pttime∈ℝB×TV×Dp_t^time ^B× TV× D and viewpoint ptview∈ℝB×TV×Dp_t^view ^B× TV× D features: Stv=Flatten(Linear(Fmv)+pttime+pvview).S_tv=Flatten (Linear(F_mv)+p_t^time+p_v^view ). (2) where Linear(⋅)Linear_(·) denotes a linear projection layer, Stv∈ℝB×TV×DS_tv ^B× TV× D is the temporal-view feature, in which each token represents the interaction observed at a specific temporal stage and camera viewpoint. Multi-View Identity Learning. To learn cross-view identity information, a learnable identity Sidcls∈ℝB×1×DS_id^cls ^B× 1× D token is introduced. The identity token represents global subject characteristics shared across different observations. It is concatenated with the temporal-view feature to construct the identity-augmented temporal-view token sequence Sidin∈ℝB×(TV+1)×DS_id^in ^B×(TV+1)× D: Sidin=[Sidcls;Stv].S_id^in=[S_id^cls;S_tv]. (3) where [⋅;⋅][·\,;\,·] denotes concatenation along the temporal-view dimension. Then an identity-specific transformer is introduced. The transformer models the identity-augmented temporal-view token sequence to capture the identity feature Fid∈ℝB×1×DF_id ^B× 1× D: Fid=Linear(Transformerid(Sidin))cls,F_id=Linear (Transformer_id (S_id^in ) )_\cls\, (4) where [⋅]cls[·]_\cls\ denotes the selection of the encoded outputs corresponding to the identity token. Multi-View Motion Learning. To learn cross-view motion information, a set of learnable motion query tokens Smotq∈ℝB×Nmot×DS_mot^q ^B× N_mot× D is also introduced. These tokens are designed to mine key HOI motion features from different temporal stages and viewpoints. Together with StvS_tv, these tokens form the motion-augmented temporal-view token sequence Smotin∈ℝB×(TV+Nmot)×DS_mot^in ^B×(TV+N_mot)× D: Smotin=[Stv;Smotq].S_mot^in=[S_tv;S_mot^q]. (5) where NmotN_mot denotes the number of motion query tokens. Then, the motion-specific transformer models the relation between the temporal-view feature and motion query tokens. The transformer produces the interaction motion feature Fmot∈ℝB×(TV+Nmot)×DF_mot ^B×(TV+N_mot)× D: Fmot=Linear(Transformermot(Smotin)).F_mot=Linear (Transformer_mot (S_mot^in ) ). (6) Two-Stage Caption Rewriting TSCR is applied only during testing. Its first stage uses the trained Identity Decoder and Identity-Agnostic Caption Decoder to retrieve the query identity and generate an identity-agnostic HOI motion caption, respectively. In the second stage, the retrieved identity and generated caption are fed into the Caption Rewriting Decoder to produce the final identity-aware HOI motion caption. The training process of these decoders are detailed in the supplementary material. Training. To obtain the identity and motion outputs required by the first testing stage, we train the Identity Decoder and Identity-Agnostic Caption Decoder in parallel. The Identity Decoder maps the identity feature to a subject entity YidY_id over the identity feature FidF_id: Yid=IDD(Fid).Y_id=IDD (F_id ). (7) In parallel, the Identity-Agnostic Caption Decoder, initialized from FLAN-T5-Base (3), maps the interaction motion feature FmotF_mot to an identity-agnostic HOI motion caption: p(Yac∣Fmot)=∏l=1LpIACD(ul∣u<l,Fmot),p (Y_ac F_mot )= _l=1^Lp_IACD (u_l u_<l,F_mot ), (8) where Yac=ull=1LY_ac=\u_l\_l=1^L denotes the ground-truth identity-agnostic HOI motion caption, ulu_l denotes its l-th token, u<lu_<l denotes the preceding tokens, and L denotes the caption length. Two-Stage Testing. The test subjects are excluded from the training identity set; therefore, the predicted subject identity is not suitable during testing. (1) First stage. Each single-view query video and each reference gallery video are independently processed by the shared identity branch to obtain their identity feature. The query identity feature is compared with each gallery identity feature, and the identity label associated with the most similar gallery sample is retrieved: j∗=argmaxj∈1,…,Gsim(Yidq,Yidgj),j^*= j∈\1,…,G\argmax\;sim (Y_id^q,Y_id^g_j ), (9) Y^id=Yidgj∗, Y_id=Y_id^g_j^*, (10) where G denotes the number of gallery samples, YidqY_id^q denotes the query identity feature, YidgjY_id^g_j denotes the identity feature of the j-th gallery sample, and sim(⋅,⋅)sim(·,·) denotes the similarity function. Yidgj∗Y_id^g_j^* is the identity label of the matched gallery sample, and Y^id Y_id is the retrieved query identity. Meanwhile, the Identity-Agnostic Caption Decoder independently generates an HOI motion caption from the query motion feature: Y^ac=IACD(Fmotq), Y_ac=IACD (F_mot^q ), (11) where FmotqF_mot^q denotes the interaction motion feature of the query video and Y^ac Y_ac denotes the generated identity-agnostic HOI motion caption. Thus, the first stage obtains the query identity and HOI motion semantics independently. (2) Second stage. The pretrained Caption Rewriting Decoder incorporates the retrieved identity into the generated motion caption: Y^iac=CRD(Y^id,Y^ac), Y_iac=CRD ( Y_id, Y_ac ), (12) where Y^iac Y_iac denotes the final identity-aware HOI motion caption. This stage explicitly associates the retrieved subject identity with the corresponding HOI motion within a unified caption. Method BLEU-4 METEOR ROUGE-L CIDEr ID Acc Weighted ID Acc CLIP-Captioner (29) 5.28 29.28 24.21 39.33 71.57 57.50 CARE (28) 8.04 31.80 28.81 55.12 65.99 47.88 NACF (30) 4.29 25.80 25.56 56.45 65.80 57.20 CoCap (18) 4.32 28.05 24.08 48.63 71.62 56.62 SwinBERT (10) 8.81 34.45 29.29 55.26 65.57 46.21 ID-HOINet 10.75 35.99 33.94 97.53 72.94 58.31 Table 2: Quantitative comparison with representative video captioning methods for identity-aware HOI motion captioning. Experiments Evaluation Settings We compare ID-HOINet with representative video captioning methods to evaluate its effectiveness. Ablation experiments further examine the contributions of each component of ID-HOINet, while qualitative visualizations illustrate the model’s ability to associate subject identities with the corresponding HOI motions in generated captions. Metrics We evaluate caption quality using four standard metrics: BLEU-4 (14), METEOR (1), ROUGE-L (9), and CIDEr (21). BLEU-4 evaluates n-gram precision up to four-grams, METEOR measures unigram-level semantic alignment, ROUGE-L captures sequence-level similarity based on the longest common subsequence, and CIDEr assesses n-gram caption consensus. Following NBA-Identity (Xi et al. 2025), we additionally employ ID Accuracy (ID Acc) and Weighted ID Accuracy (Weighted ID Acc) to evaluate identity recognition. ID Acc measures the proportion of query samples whose predicted identity labels match their ground-truth identities by treating all query samples equally. Weighted ID Acc first computes the prediction accuracy for each identity and then averages the identity-level accuracies, thereby reducing the influence of imbalanced query sample distributions across identities. Baselines We compare ID-HOINet with five representative video captioning methods: CLIP-Captioner (29), CARE (28), NACF (30), CoCap (18), and SwinBERT (10). For a fair comparison, all baseline methods are evaluated under the same evaluation setting. Variant BLEU-4 METEOR ROUGE-L CIDEr ID Acc Weighted ID Acc ID-HOINet w/o MVIML 6.24 26.72 28.77 37.40 45.49 12.50 ID-HOINet w/o view randomization 9.46 33.27 32.11 77.65 52.27 49.79 ID-HOINet w/ direct identity-aware generation 12.15 38.19 34.24 95.41 67.39 52.75 ID-HOINet 10.75 35.99 33.94 97.53 72.94 58.31 Table 3: Ablation study of ID-HOINet. Implementation Details We implement ID-HOINet in PyTorch. All video frames are represented by pre-extracted CLIP visual features, and each clip is sampled with 32 temporal steps from 4 camera views. The CLIP feature dimension is 768. In MVIML, we employ 1 learnable identity token and 8 learnable motion query tokens, which are processed by two independent Transformer branches for identity recognition and identity-agnostic captioning, respectively. During training, we apply a view-randomization strategy that randomly keeps 2, 3, or 4 views with probabilities of 0.5, 0.3, and 0.2, respectively, so that the model learns robust multi-view features while remaining compatible with single-view inference. The final identity-aware HOI motion caption is produced by a pretrained FLAN-T5-Base Caption Rewriting Decoder. The model is trained for 50 epochs using two AdamW optimizers with a weight decay of 1×10−41× 10^-4. The main net use a learning rate of 5×10−55× 10^-5, while the trainable FLAN-T5-Base decoder uses a learning rate of 1×10−51× 10^-5. Both learning rates are linearly warmed up over the first five epochs and then kept constant. The batch size is 32. All experiments are conducted on a single NVIDIA GeForce RTX 4090 GPU. Datasets BEHAVE (2) is a multi-view HOI video dataset captured by four synchronized cameras. It contains 321 video sequences of eight subjects interacting with 20 everyday objects across five indoor environments. Diverse full-body interactions are recorded in RGB video, including both hand-object and foot-object motions. InterCap (6) contains 223 multi-view HOI videos captured by six synchronized cameras. It records ten subjects interacting with ten everyday objects of different sizes and affordances. It provides RGB interaction videos depicting diverse full-body interactions, including both hand-object and foot-object motions. Comparison with Captioning Baselines Table 2 shows that the ID-HOINet consistently outperforms representative video captioning methods across both captioning and identity-matching metrics, demonstrating its effectiveness for identity-aware HOI motion captioning. These improvements indicate that ID-HOINet can generate more accurate interaction captions while maintaining reliable subject identification. Taken together, these results establish ID-HOINet as a strong baseline for future research on this task. Figure 3: Qualitative comparison of identity-aware HOI motion captions generated by ID-HOINet and representative video captioning methods. Each case presents sampled video frames, the ground-truth caption, and generated results, with correct and incorrect identity, object, and motion expressions highlighted in green and red, respectively. Ablation Study Table 3 examines how multi-view identity-motion learning, view-randomized training, and two-stage caption generation contribute to identity matching and HOI motion captioning under single-view testing. Removing MVIML causes substantial degradation across all metrics, particularly ID Acc and Weighted ID Acc, demonstrating the importance of multi-view information for jointly understanding subject identity and HOI motion. Removing the view randomization strategy also reduces both captioning and identity-matching performance, indicating that randomized multi-view training improves robustness under single-view testing. We further evaluate a direct identity-aware HOI motion caption generation strategy. Although this variant achieves higher captioning scores, its ID Acc decreases from 72.94 to 67.39, and its Weighted ID Acc decreases from 58.31 to 52.75. This result reveals a trade-off between textual similarity and identity reliability: a caption may achieve high lexical overlap with the reference while still being associated with an incorrect subject. In identity-aware HOI motion captioning, identity misattribution is particularly detrimental, as assigning an otherwise accurate HOI motion caption to the wrong subject fundamentally compromises the semantic validity of the final prediction. Therefore, we adopt the proposed two-stage generation strategy rather than direct identity-aware caption generation. Qualitative Results Figure 3 presents two representative comparisons of identity-aware HOI motion captions. In Case 1, ID-HOINet correctly identifies Sub07 and accurately describes the interacted object, the right-hand operation, and the complete motion sequence of lifting and putting down the square table. In contrast, the baseline methods either confuse the object, incorrectly describe the use of both hands, omit the subsequent putting-down action, or assign the interaction to the wrong subject. In Case 2, ID-HOINet correctly associates the sitting action with Sub18 and the stool, whereas all comparison methods misidentify the subject and most incorrectly interpret the interaction as lifting an object. Although ID-HOINet introduces additional motion details in this case, it still preserves the correct identity and core HOI semantics. Overall, these examples show that ID-HOINet achieves better consistency among subject identity, interacted object, and HOI motion than video captioning baselines. The results also indicate that identity-aware HOI motion captioning requires more than lexical similarity: an effective prediction must correctly associate the observed interaction with the corresponding subject. Limitations As an initial study of identity-aware HOI motion captioning, this work has several limitations. First, the experiments are conducted on reorganized BEHAVE and InterCap data containing only 18 subjects and a relatively limited range of objects, actions, and capture environments. The generalization ability of the proposed method to larger subject populations and more diverse real-world scenes therefore remains to be investigated. Second, the current setting focuses on interactions involving a single subject and a single object, without considering more complex multi-person or multi-object interactions. Future work will address these limitations by expanding the data scale and extending the task to more complex interactions. Conclusion This paper introduces Identity-Aware HOI Motion Captioning, which requires a model to generate captions describing both the subject identity and the corresponding human-object interaction motion. We process BEHAVE and InterCap through temporal video segmentation, identity-aware HOI motion caption annotation, and data partitioning. We further propose ID-HOINet, whose Multi-View Identity-Motion Learning Module learns complementary identity and interaction motion features from synchronized multi-view videos. During testing, the Two-Stage Caption Rewriting Strategy first determines the query identity through gallery matching and generates an identity-agnostic motion caption, then rewrites the caption with the determined identity. Experimental results demonstrate the effectiveness of ID-HOINet in associating subject identities with the corresponding HOI motions in generated captions. Future work extends this task to multi-person and multi-object interactions and improves the description of fine-grained motion details. References Banerjee and Lavie (2005) S. Banerjee and A. Lavie METEOR: an automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, p. 65–72. Cited by: Metrics. Bhatnagar et al. (2022) B. L. Bhatnagar, X. Xie, I. A. Petrov, C. Sminchisescu, C. Theobalt, and G. Pons-Moll Behave: dataset and method for tracking human object interactions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 15935–15946. Cited by: Introduction, Source Datasets, Datasets. Chung et al. (2024) H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research 25 (70), p. 1–53. Cited by: Training.. Doering et al. (2022) A. Doering, D. Chen, S. Zhang, B. Schiele, and J. Gall Posetrack21: a dataset for person search, multi-object tracking and multi-person pose tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 20963–20972. Cited by: Introduction. Hakim and Shimshoni (2019) T. Hakim and I. Shimshoni A-mal: automatic motion assessment learning from properly performed motions in 3d skeleton videos. In Proceedings of the IEEE/CVF international conference on computer vision workshops, p. 0–0. Cited by: Introduction. Huang et al. (2024) Y. Huang, O. Taheri, M. J. Black, and D. Tzionas InterCap: joint markerless 3d tracking of humans and objects in interaction from multi-view rgb-d images. International Journal of Computer Vision 132 (7), p. 2551–2566. Cited by: Introduction, Source Datasets, Datasets. Kryeem et al. (2023) A. Kryeem, S. Raz, D. Eluz, D. Itah, H. Hel-Or, and I. Shimshoni Personalized monitoring in home healthcare: an assistive system for post hip replacement rehabilitation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 1868–1877. Cited by: Introduction. Li et al. (2024) J. Li, J. Xue, R. Cao, X. Du, S. Mo, K. Ran, and Z. Zhang Finerehab: a multi-modality and multi-task dataset for rehabilitation analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 3184–3193. Cited by: Introduction. Lin (2004) C. Lin Rouge: a package for automatic evaluation of summaries. In Text summarization branches out, p. 74–81. Cited by: Metrics. Lin et al. (2022) K. Lin, L. Li, C. Lin, F. Ahmed, Z. Gan, Z. Liu, Y. Lu, and L. Wang Swinbert: end-to-end transformers with sparse attention for video captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 17949–17958. Cited by: Introduction, Human-Object Interaction Motion Captioning, Table 2, Baselines. Liu et al. (2025) K. Liu, Y. Fu, W. Yuan, J. Lin, P. Li, X. Gu, L. Qiu, H. Wang, Z. Dong, and X. Han Motions as queries: one-stage multi-person holistic human motion capture. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 17529–17539. Cited by: Introduction. Ma et al. (2018) C. Ma, A. Kadav, I. Melvin, Z. Kira, G. AlRegib, and H. P. Graf Attend and interact: higher-order object interactions for video understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 6790–6800. Cited by: Human-Object Interaction Motion Captioning. Mittal et al. (2022) A. Mittal, S. Ghosal, R. Bansal, and D. Ngyuyen SAVCHOI: detecting suspicious activities using dense video captioning with human object interactions. arXiv preprint arXiv:2207.11838 2. Cited by: Human-Object Interaction Motion Captioning. Papineni et al. (2002) K. Papineni, S. Roukos, T. Ward, and W. Zhu Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, p. 311–318. Cited by: Metrics. Park et al. (2020) J. S. Park, T. Darrell, and A. Rohrbach Identity-aware multi-sentence video description. In European Conference on Computer Vision, p. 360–378. Cited by: Introduction, Identity-Aware Video Captioning. Raajesh et al. (2024) H. Raajesh, N. R. Desanur, Z. Khan, and M. Tapaswi Micap: a unified model for identity-aware movie descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14011–14021. Cited by: Introduction, Identity-Aware Video Captioning. Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, p. 8748–8763. Cited by: Input Representation. Shen et al. (2023) Y. Shen, X. Gu, K. Xu, H. Fan, L. Wen, and L. Zhang Accurate and fast compressed video captioning. In Proceedings of the IEEE/CVF international conference on computer vision, p. 15558–15567. Cited by: Table 2, Baselines. Song et al. (2025) G. Song, G. Wang, Z. Huang, J. Lin, X. Zhe, J. Li, and H. Wang Towards fine-grained human motion video captioning. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 846–855. Cited by: Human-Object Interaction Motion Captioning. Tang et al. (2021) M. Tang, Z. Wang, Z. Liu, F. Rao, D. Li, and X. Li CLIP4Caption: CLIP for video caption. In Proceedings of the 29th ACM International Conference on Multimedia, p. 4858–4862. External Links: Document Cited by: Introduction, Human-Object Interaction Motion Captioning. Vedantam et al. (2015) R. Vedantam, C. Lawrence Zitnick, and D. Parikh Cider: consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, p. 4566–4575. Cited by: Metrics. Wang et al. (2024) J. Wang, D. Chen, C. Luo, B. He, L. Yuan, Z. Wu, and Y. Jiang Omnivid: a generative framework for universal video understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 18209–18220. Cited by: Introduction, Human-Object Interaction Motion Captioning. Wang et al. (2020) M. Wang, J. Tighe, and D. Modolo Combining detection and tracking for human pose estimation in videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11088–11096. Cited by: Introduction. Xi et al. (2025) Z. Xi, H. Sun, Y. Wu, J. Yan, H. Zhang, L. Wu, L. Wang, and C. Chen Player-centric multimodal prompt generation for large language model based identity-aware basketball video captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 24330–24339. Cited by: Introduction, Identity-Aware Video Captioning. Xu et al. (2026) A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. Deepseek-v4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Cited by: Identity-Aware HOI Motion Annotation. Xu et al. (2025) S. Xu, D. Li, Y. Zhang, X. Xu, Q. Long, Z. Wang, Y. Lu, S. Dong, H. Jiang, A. Gupta, et al. Interact: advancing large-scale versatile 3d human-object interaction generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 7048–7060. Cited by: Temporal Video Segmentation. Yang et al. (2023a) A. Yang, A. Nagrani, P. H. Seo, A. Miech, J. Pont-Tuset, I. Laptev, J. Sivic, and C. Schmid Vid2seq: large-scale pretraining of a visual language model for dense video captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10714–10726. Cited by: Introduction, Human-Object Interaction Motion Captioning. Yang et al. (2023b) B. Yang, M. Cao, and Y. Zou Concept-aware video captioning: describing videos with effective prior information. IEEE Transactions on Image Processing 32, p. 5366–5378. Cited by: Table 2, Baselines. Yang et al. (2022) B. Yang, T. Zhang, and Y. Zou Clip meets video captioning: concept-aware representation learning does matter. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), p. 368–381. Cited by: Table 2, Baselines. Yang et al. (2021) B. Yang, Y. Zou, F. Liu, and C. Zhang Non-autoregressive coarse-to-fine video captioning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35, p. 3119–3127. Cited by: Table 2, Baselines. Zhao et al. (2025) B. Zhao, X. Liu, Z. Yu, T. Yang, Y. Song, M. Jin, S. Li, and Y. Wang Exploring fine-grained human motion video captioning. In Proceedings of the 31st International Conference on Computational Linguistics, p. 5247–5264. Cited by: Human-Object Interaction Motion Captioning. Zheng et al. (2016) L. Zheng, Z. Bie, Y. Sun, J. Wang, C. Su, S. Wang, and Q. Tian Mars: a video benchmark for large-scale person re-identification. In European conference on computer vision, p. 868–884. Cited by: Introduction.