Paper deep dive
Label-Guided Knowledge Distillation for 3D-CNNs in Action Recognition
Yanjiang Shi, Peng Zhao, Nan Qi, Guiqin Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 9/20/2026, 2:45:24 AM
Summary
The paper proposes Label-Guided Knowledge Distillation (LGKD), a method for compressing 3D-CNNs in action recognition by transferring knowledge from a teacher to a student model. LGKD utilizes ground truth labels to guide distillation at two levels: sample-wise, using temporal importance scores derived from teacher probabilities to weight feature loss, and class-wise, using a prototype network to align student features with category-level semantic representations. Experiments on UCF101 and HMDB51 demonstrate improved performance over baseline methods.
Entities (8)
Relation Signals (7)
LGKD → evaluatedon → UCF101
confidence 98% · we conducted comprehensive experiments on two benchmark action recognition datasets, UCF101 and HMDB51
LGKD → evaluatedon → HMDB51
confidence 98% · we conducted comprehensive experiments on two benchmark action recognition datasets, UCF101 and HMDB51
LGKD → uses → Sample-wise Distillation
confidence 95% · Our method entails two components: sample-wise distillation and class-wise distillation
LGKD → uses → Class-wise Distillation
confidence 95% · Our method entails two components: sample-wise distillation and class-wise distillation
Class-wise Distillation → employs → Prototype Network
confidence 93% · class-wise feature distillation employs a prototype network to further capture the relational knowledge among samples
Sample-wise Distillation → uses → Ground Truth Labels
confidence 92% · Sample-wise distillation utilizes label information and the teacher's probability distribution
LGKD → targets → 3D CNN
confidence 90% · Label-Guided Knowledge Distillation for 3D-CNNs in Action Recognition
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As a key model compression technique, knowledge distillation aims to transfer knowledge from a high-capacity teacher model to a lightweight student model for enhancing the latter's performance. In this work, we reviewed the feature knowledge distillation for 3D-CNNs and observed that most feature distillation methods in video analysis are simple adaptations of those used in image analysis, often neglecting the differences of video features in the temporal dimension. To address this issue, we proposed Label-Guided Knowledge Distillation (LGKD) to guide the distillation of student model features using ground truth labels. Our method entails two components: sample-wise distillation and class-wise distillation, enabling the student model to learn feature representation of the teacher model at two levels. Sample-wise distillation utilizes label information and the teacher's probability distribution to guide the learning of features that significantly impact temporal accuracy while mitigating noise. Meanwhile, class-wise feature distillation employs a prototype network to further capture the relational knowledge among samples within the same category, enhancing the student's ability to learn higher-dimensional semantic information and improving model generalization. To demonstrate the effectiveness and superiority of our method, we conducted comprehensive experiments on two benchmark action recognition datasets, UCF101 and HMDB51, achieving competitive results.
Tags
Links
- Source: https://arxiv.org/abs/2609.13024v1
- Canonical: https://arxiv.org/abs/2609.13024v1
Trouble viewing inline? Open PDF directly →
Full Text
53,526 characters extracted from source content.
Expand or collapse full text
Label-Guided Knowledge Distillation for 3D-CNNs in Action Recognition IEEE Publication Technology Yanjiang Shi Peng Zhao Nan Qi Guiqin Wang †thanks: Y. Shi is with the School of Computer Science and Technology, and with the National Engineering Laboratory for Big Data Analytics (NEL-BDA), Xi’an Jiaotong University, Xi’an 710049, China (Email: syj2908@stu.xjtu.edu.cn).†thanks: Nan Qi and G. Wang are with the School of Computer Science and Technology, and with the National Engineering Laboratory for Big Data Analytics (NEL-BDA), Xi’an Jiaotong University, Xi’an 710049, China (Email: qinan@stu.xjtu.edu.cn; gqwang@stu.xjtu.edu.cn).†thanks: P. Zhao is with the School of Computer Science and Technology, and with the National Engineering Laboratory for Big Data Analytics (NEL-BDA), Xi’an Jiaotong University, Xi’an 710049, China (Email: p.zhao@mail.xjtu.edu.cn).†thanks: Manuscript received April 19, 2021; revised August 16, 2021. Abstract As a key model compression technique, knowledge distillation aims to transfer knowledge from a high-capacity teacher model to a lightweight student model for enhancing the latter’s performance. In this work, we reviewed the feature knowledge distillation for 3D-CNNs and observed that most feature distillation methods in video analysis are simple adaptations of those used in image analysis, often neglecting the differences of video features in the temporal dimension. To address this issue, we proposed Label-Guided Knowledge Distillation (LGKD) to guide the distillation of student model features using ground truth labels. Our method entails two components: sample-wise distillation and class-wise distillation, enabling the student model to learn feature representation of the teacher model at two levels. Sample-wise distillation utilizes label information and the teacher’s probability distribution to guide the learning of features that significantly impact temporal accuracy while mitigating noise. Meanwhile, class-wise feature distillation employs a prototype network to further capture the relational knowledge among samples within the same category, enhancing the student’s ability to learn higher-dimensional semantic information and improving model generalization. To demonstrate the effectiveness and superiority of our method, we conducted comprehensive experiments on two benchmark action recognition datasets, UCF101 and HMDB51, achieving competitive results. Index Terms: Knowledge Distillation, Model Compression, Prototype Network I Introduction In recent years, deep learning has been increasingly applied to video understanding. As a fundamental task in this field, action recognition has garnered significant attention from both academia and industry. Deep learning models for video analysis can be broadly categorized into two architectures: 3D convolutional neural networks (3D-CNNs) and video transformers [1]. While 3D-CNNs have evolved considerably, achieving higher accuracy through increased model depth, this also results in a greater number of parameters and higher computational complexity. Consequently, deploying these high-precision models on resource-constrained devices remains challenging. Knowledge distillation (KD), a key model compression technique, addresses this issue by transferring knowledge from a high-accuracy, computationally intensive teacher model to a lightweight, lower-accuracy student model, thereby improving the latter’s performance. As a result, knowledge distillation for 3D-CNNs continues to be a valuable research direction. Feature-based knowledge distillation has gained prominence recently as an effective approach due to the rich learnable information contained in feature maps [6, 3, 4]. This method enables the student model to learn the teacher model’s ability to capture data representations by mimicking its feature map distribution. Current research on feature-based knowledge distillation primarily focuses on the image domain, particularly on model compression for 2D-CNNs. In contrast, knowledge distillation for 3D-CNNs remains relatively underexplored, with recent studies shifting toward cross-modal knowledge distillation. This approach utilizes feature maps as intermediaries to transfer knowledge from the optical flow branch to the RGB branch, making it applicable only to teacher-student networks designed for multi-modal data. In the standard distillation paradigm, where knowledge is transferred from a larger teacher model to a smaller student model, existing methods largely adapt feature-based knowledge distillation techniques developed for 2D-CNNs. However, unlike static images, video data include a temporal dimension, which affects model inference in tasks such as action recognition. The importance of different frames within a video varies; for instance, in the action “cliff diving,” preparatory movements, takeoff, and descent contribute differently to classification. This variation, as observed in [5], suggests that labeled video frames can be categorized based on their significance for classification versus regression. The accuracy of features extracted from the latter has a greater impact on the final classification outcome. Despite this, conventional feature-based knowledge distillation methods treat all features extracted by the teacher model equally, potentially introducing noise during knowledge transfer. Motivated by this phenomenon, we aim to investigate how adjusting distillation loss weights based on the importance of video frame features can enhance knowledge distillation. To address the aforementioned problem, we began with a simple validation experiment by incorporating label information into the distillation process. Unlike conventional methods that use label information solely to calculate the classification loss of the student model, we conduct knowledge distillation only on samples where the teacher model makes correct predictions. Specifically, ground truth labels are used to select the samples for distillation, with results presented in Table I. It can be observed that simply introducing label information enhances the effectiveness of both response-based and feature-based knowledge distillation. Notably, by leveraging ground truth labels and distilling only on correctly classified samples, accuracy improves by approximately 0.4% in both distillation methods. This observation motivates us to further utilize label information to help the student model differentiate the importance of various frame features. TABLE I: The validation results on the UCF101 dataset. Method Top-1(%) Top-5(%) Baseline Student 63.36 82.08 Logits-based KD 67.20 85.44 Logits-based KD w/o Wrong Prediction 67.57 86.18 Feature-based KD 67.72 86.47 Feature-based KD w/o Wrong Prediction 68.20 86.60 Motivated by this observation, we propose Label-Guided Knowledge Distillation(LGKD), a method tailored for action recognition tasks. As illustrated in Fig. 1, LGKD performs feature distillation at both the sample and class levels, leveraging label information to fully extract learnable information from the teacher’s features. Specifically, in sample-wise feature distillation, based on traditional feature distillation, we use label information to extract the probability distribution of the ground truth category inferred by the teacher model along the temporal dimension. This distribution serves as the basis for calculating importance scores of frame features, which are then used to assign corresponding weights to the feature loss. Additionally, to prevent the student model from overfitting the teacher’s features during the sample-wise distillation, we introduce class-wise distillation loss. Considering samples within the same category have similarities, aligning features at the category level can guide the student model to capture high-order semantic information, making the feature representations of each category more compact while improving the generalization of the student model[36]. To achieve this, we propose a prototype network that transfers relational knowledge within each category. By filtering samples correctly classified by the teacher using ground truth labels, we generate action category feature prototypes—or category feature centers—to guide the student model. Additionally, we introduce a weight adjustment module in sample-wise feature distillation to dynamically adjust temporal loss weights for each teacher-student feature pair. In class-wise feature distillation, the prototype network generates prototype features for each action category, further enhancing knowledge transfer. Fig. 1: Label-guided knowledge distillation method. Compared to traditional feature distillation, sample-wise distillation leverages label information and the teacher’s probability distribution to explore the temporal importance differences in teacher features. Meanwhile, class-wise distillation, rather than distilling knowledge independently for each sample, leverages label information to generate class prototypes that represent high-level semantic features. To the best of our knowledge, we are the first to consider temporal weighting in feature distillation and introduce label information to the distillation for video understanding tasks. Our main contributions are summarized as follows: • To extract frame features that are more critical for classification during knowledge distillation, we propose a sample-wise knowledge distillation approach and design a weight adjustment module. This module leverages label information and the teacher’s probability distribution to guide the student model in learning features that are more important for classification accuracy in the temporal dimension. • We propose a class-wise knowledge distillation method based on a prototype network to mitigate feature divergence caused by sample-wise distillation. By leveraging label information to filter the features of samples correctly inferred by the teacher. This approach aligns features at the category level, facilitates the transfer of relational knowledge among samples within the same category, and enables the student model to learn higher-order semantic information from the teacher model. • We conduct extensive experiments on two widely used action recognition datasets, UCF101 and HMDB51. The experiment results demonstrate that our method significantly improves the feature representation capability of the student model, thereby surpassing the performance of other compared methods. Furthermore, we additional explore experiments to validate the effectiveness of our approach and its generalization capability across different teacher-student model pairs. I Related Works I-A Action Recognition As a fundamental task in video understanding, action recognition has recently transitioned from purely 3D-CNN-based models to Transformer-based models[7]. However, the latter still require 3D-CNNs to extract accurate features[8]. The feature extraction networks of existing 3D-CNN architectures can be divided into fully 3D-CNNs[9, 10, 11, 12] and partially 3D-NNs[13, 14, 15]. Fully 3D-CNNs treat video as a sequence of continuous frames and use 3D convolutional kernels to extract representation, with typical models like I3D[9], R-C3D[10]. Partially 3D-CNNs replace 3D convolutional layers with 2D convolutional layers to reduce computation costs and parameter requirements, i.e., Top-I3D, Bottom-I3D, and S3D[15]. However, while reducing resource demands, these partial 3D-CNNs also suffer from a noticeable decline in accuracy, necessitating further exploration to narrow the gap between fully 3D-CNNs and partially 3D-CNNs. I-B Knowledge Distillation Knowledge distillation was first proposed by Hinton et al.. As a branch of model compression technology, its purpose is to transfer the knowledge contained in the heavyweight, high-precision teacher model to the lightweight, low-precision student model to improve the student’s accuracy[16]. Based on the form of the knowledge, knowledge distillation can be mainly categorized into response-based knowledge distillation, feature-based knowledge distillation, and api-based knowledge distillation[17]. Api-based knowledge distillation has emerged with the development of Large Language Models(LLMs), where the LLM acts as a black-box teacher to guide the student model. However, this method is only applicable to the teacher-student pairs which both of them are LLM[19, 18]. Response-based knowledge distillation involves having the student model mimic the probability distribution outputted by the teacher to learn the ”dark knowledge” contained in soft labels[16, 20, 21]. In recent years, feature-based knowledge distillation has become the mainstream research direction due to its inclusion of more learnable information[6]. It uses feature maps as the medium of knowledge, enabling the student model to learn the teacher model’s feature extraction capabilities by mimicking the distribution of the teacher model’s feature maps. In the research of feature distillation for convolutional neural networks, most current work still focuses on image data[10, 23, 24]. In the field of video analysis, the mainstream research method is cross-modal knowledge distillation[25, 26, 27, 28]. These methods treat the model branch that processes optical flow data as the teacher, aiming to transfer its feature extraction capabilities to the model branch that handles RGB data, thereby improving the accuracy of the unimodal model. Moreover, most feature distillation methods aimed at model compression simply adapt 2D-CNN distillation techniques, neglecting the differences in the importance of features across the temporal dimension[29]. I-C Prototype Network Prototype network was first applied to few-shot classification[30], where it computes the distance between new samples and every class prototype in a learnable metric space for classification. Due to its simplicity and strong generalization capability, the prototype network has been extended to various tasks, such as transfer learning[31] and zero-shot learning[32]. ProDA[31] introduces the concept of prototype network into pseudo-label denoising. PSKD[32] utilizes prototype network to refine the teacher’s probability distribution through a correlation matrix in zero-shot learning tasks. In the field of knowledge distillation, ProKD[33] proposes a contrastive learning-based prototype alignment method to enhance the student model’s ability to acquire specific language knowledge in zero-resource named entity recognition task. In our work, we utilize the prototype network to extract relational knowledge among samples within each action category, guiding the student model to learn the teacher model’s feature representation capabilities. I Methodology In this section, we first review the basic concepts of feature knowledge distillation and then introduce our label-guided knowledge distillation. I-A Vanilla Feature Knowledge Distillation As a key method in model compression technology, knowledge distillation has been widely applied to many visual tasks. As a branch of knowledge distillation, feature knowledge distillation has increasingly become the mainstream method due to its excellent accuracy improvement. The primary objective of feature knowledge distillation is to transfer the knowledge contained in the heavyweight, high-precision teacher model to the lightweight, low-precision student model via feature maps[34]. Formally, let T and S represent the teacher and student models, respectively. When video data i is sent to the models, they generate feature maps FTF^T and FSF^S, where F∈RT×C×H×WF∈ R^T× C× H× W. Here, T,C,H and W represent the temporal, channel, and spatial dimensions, respectively. In traditional feature knowledge distillation, the distillation loss is composed of the L2L_2 distance between the feature maps produced by the teacher and student models: ℒvanillafeatureKD=∑n=1N‖f(Fi,nT)−f(Fi,nS)‖22,L_vanilla\ feature\ KD= _n=1^N\|f(F^T_i,n)-f(F^S_i,n)\|_2^2, (1) where f(⋅)f (· ) is a projection function that aligns the feature maps of the teacher and student models to the same dimension, N represents the number of frames contained in the video. This constraint encourages the student model to mimic the feature map distribution of the teacher model, thereby capturing the teacher model’s feature extraction capability. However, vanilla feature distillation method originally designed for image data applies equal weight to all feature maps during distillation. When transferred to video data domain that includes temporal dimension, it overlooks the fact that different video frame features may have different importance to the inference result. Fig. 2: We propose a label-guided feature knowledge distillation framework. Our approach primarily consists of two submodules: a weight adjustment module for sample-wise distillation and a prototype network module for class-wise distillation. I-B Label-guided Knowledge Distillation To address the aforementioned problem, we introduce label information into traditional feature distillation to assist the student model in better learning the teacher model’s feature representation. The proposed distillation framework is illustrated in Fig. 2 and it consists of two main submodules: sample-wise knowledge distillation and class-wise knowledge distillation. In sample-wise knowledge distillation, the loss between teacher and student feature maps is weighted according to importance scores and then used as the final distillation loss. In class-wise knowledge distillation, the student model’s sample features are aligned with the corresponding category feature prototypes computed by the teacher model in the feature space. I-B1 Sample-wise Knowledge Distillation In video understanding tasks, video data is represented as a set of stacked feature maps[35]. In sample-wise knowledge distillation, we introduce a weight adjustment module to calculate the importance score of each feature map using the ground truth labels and the probability distribution produced by the teacher. This importance score is then used to adjust the weights of the loss between each pair of teacher and student feature maps, which serves as the final distillation loss for the student model. Formally, let qiT∈RN×Class×1q^T_i∈ R^N× Class× 1 represents the prediction result of the teacher model before Softmax function. For the video features FiTF^T_i extracted by the teacher, the corresponding feature map importance score (i.e., weight) is calculated as follows: weight_scorei=θ(qi,GT),weight\_score_i=θ(q^T_i,GT), (2) qi,GT=qiT×GT,q^T_i,GT=q^T_i× GT, (3) where, qi,GT∈RN×1q^T_i,GT∈ R^N× 1 represents the probability distribution of the teacher’s inference results qiTq^T_i corresponding to the ground truth label GTGT. The function θ(⋅)θ(·) is a normalization and post-processing function. In our implementation, we explore two different methods for normalization and post-processing: direct normalization and masked normalization. Direct Normalization In direct normalization, the importance score of the feature is calculated based on the distribution of the ground truth class over the temporal dimension of the teacher’s inference results. After normalization, the result is used as the importance score of the feature: θ(qi,GT)=Softmax(qi,GT).θ(q^T_i,GT)=Softmax(q^T_i,GT). (4) Here, the Softmax function normalizes the importance scores to [0,1], allowing for subsequent processing. Masked Normalization During direct normalization, the differences in the inference results across different time are reduced after normalization. Since direct normalization does not specifically account for negative values, we use masking mechanism to suppress low-importance features and highlight high-importance features. Specifically, we mask out negative results and only use positive results as valid importance scores for the subsequent distillation loss computation: θ(qi,GT)=Softmax(mask(qi,GT))∘(qi,GT>0),θ(q^T_i,GT)=Softmax(mask(q^T_i,GT)) (q^T_i,GT>0), (5) where ∘ is element-wise multiplication, (⋅)I(·) is the indicator function, the masking function mask(⋅)mask(·) is defined as: mask(x)=max(xn,0)n=1N.mask(x)=\ (x_n,0)\_n=1^N. (6) The final sample-wise distillation loss introduces the feature importance scores as weights based on vanilla feature distillation loss, which is defined as: ℒSample−wise=∑n=1Nweight_scorei,n×‖f(Fi,nT)−f(Fi,nS)‖22,L_Sample-wise= _n=1^Nweight\_score_i,n×\|f(F^T_i,n)-f(F^S_i,n)\|_2^2, (7) where f(⋅)f(·) is the project function. I-B2 Class-wise Knowledge Distillation In addition to conducting one-to-one distillation of teacher-student features for the same sample through sample-wise knowledge distillation, there is also relational knowledge among features of different instances within the same category that the student model can learn[36]. Features of samples within the same category contain high similarity, while features of samples from different categories show low similarity. For example, consider a collection of videos labeled as ”cliff diving” and ”cycling”. The features extracted by an excellent teacher model should form two compact feature clusters in the feature space, corresponding to two types of video samples. Aligning the features of student and teacher models at the category level can help the student model mimic high-level semantic information of the teacher model, making the feature representation of samples within the same category more compact, ultimately improving the effectiveness of knowledge distillation[37]. In this paper, we propose a class-wise knowledge distillation approach based on a prototype network. We generate corresponding feature prototypes by selecting samples correctly inferred by the teacher model according to the ground truth labels. Formally, for an action category c, its corresponding prototype prototypecprototype_c is calculated as follows: prototypec=1I×∑i=1IFiT,prototype_c= 1I× _i=1^IF^T_i, (8) where I is the number of samples in the training set with ground truth label c, and FiTF^T_i represents the feature of sample i extracted by the teacher model. The final class-wise distillation loss is defined as: ℒClass−wise=∑c=1C∑i=1I‖prototypec−f(Fc,iS)‖22,L_Class-wise= _c=1^C _i=1^I\|prototype_c-f(F^S_c,i)\|_2^2, (9) where C represents the number of categories, I is the number of samples in each category, f(⋅)f(·) is the projection function, and Fc,iSF^S_c,i denotes the features of sample i with ground truth label c calculated by the student model. During training, it is impossible to obtain the features of all samples from the same category simultaneously due to memory constraints. Therefore, when calculating the category prototype, we update the prototype using a moving average method [38] within each batch of samples. Specifically, in each batch, we update the prototype for each category according to the following approach: prototypec=(prototypec×proto_cntc+∑i=1nFc,iT)proto_cntc+n,prototype_c= (prototype_c×proto\_cnt_c+ _i=1^nF^T_c,i)proto\_cnt_c+n, (10) proto_cntc=proto_cntc+n,proto\_cnt_c=proto\_cnt_c+n, (11) where proto_cntcproto\_cnt_c is the number of samples with label c that have already been computed, n is the number of samples with label c in the current batch, and Fc,iTF^T_c,i represents the features of the i-th sample extracted by the teacher model with label c. Additionally, since the teacher model does not compute the prototype features at the beginning of the training, we adopt a warm-up strategy for class-wise knowledge distillation. Specifically, during the first few training epochs, referred to the warm-up period, the teacher model updates the prototype features for each category while computing sample features, but does not perform class-wise knowledge distillation. After the warm-up period, as the prototype features are stable, at which point the teacher model stops updating the prototype features and the framework begins the class-wise knowledge distillation. Class-wise knowledge distillation not only explicitly guides the student model to learn the relationships among sample features within the same category, but also effectively mitigates the impact of a small number of poorly extracted features by the teacher model. Specifically, when calculating the category prototypes, we use only the sample features correctly inferred by the teacher model. This approach avoids the negative impact of the teacher model’s limitations on knowledge distillation. Additionally, the prototype calculation extracts consistent features inherent to samples within the same category, reducing the impact of the noise in individual samples, which has a regularization effect and ultimately improves the generalization ability of the student model[37]. I-C Training and Inference During the training process, we adopt an offline knowledge distillation approach. The teacher model’s parameters are frozen after pretraining and finetuning on the training set, with only the student model’s parameters being updated. Additionally, the two proposed distillation loss functions can be used either independently or in conjunction with the task-related loss function, which for action recognition tasks is the cross-entropy function. The overall training loss function can be expressed as: ℒ=CE(y~,y)+α×ℒSample−wise+β×ℒClass−wise,L=CE( y,y)+α×L_Sample-wise+β×L_Class-wise, (12) where CE(⋅)CE(·) denotes the cross-entropy loss function, y~ y is the predicted label by the student model, y is the corresponding ground truth label, and α and β are hyperparameters that balance the distillation losses. During inference, we only use the student model to predict the category label. IV Experiments In this section, we first analyze the effects of two weight adjustment module implementations. We then present the main experimental results of our proposed method, followed by ablation experiments of each module, generalization experiments, and hyperparameters sensitivity experiments. IV-A Experiment Setup Datasets. To validate the effectiveness of our framework, we conduct experiments on the mainstream action recognition datasets UCF101[39] and HMDB51[40]. • UCF101 consists of 13,320 real action videos across 101 action categories. It has three official data split, each dividing the data into training and testing sets with an approximately 7:3 ratio. • HMDB51 contains 6,849 videos, mostly sourced from movies, and includes 51 action categories, with each category having at least 101 video clips. It also has three official training/testing split with a 7:3 ratio. In our experiments, we selected the first split scheme for each dataset. Evaluation Metrics. We follow standard evaluation protocol and use recognition accuracy as the metric for model evaluation. Specifically, we use Top-1 accuracy and Top-5 accuracy as evaluation metrics for the action recognition task. Implementation Details. We use the TV-L1 algorithm[41] to extract all optical flow frames for each video, with each video set to 64 frames in length. The frame size is scaled to 256 for UCF101 and 240 for HMDB51. During training, we applied data augmentation techniques including multi-scale cropping and random horizontal flipping. During inference, we only use center cropping. We initialize the I3D model parameters with weights pretrained on ImageNet[42] and finetune the model for 20 epochs on the UCF101 and HMDB51 datasets to serve as the teacher model. The Top-I3D model is trained from scratch on the action recognition datasets until convergence, serving as the baseline student model. For the baseline network on the UCF101 dataset, we use the SGD optimizer with a learning rate of 10−310^-3, momentum of 0.9, and weight decay of 5×10−45×10^-4. The batch size is set to 12. For the HMDB51 dataset, the same SGD optimizer settings are used. For both datasets, we train the model on RGB and optical flow data for both 200 epochs. During testing, the batch size is set to 1, and the final result is obtained by averaging the predictions from the RGB and optical flow branches. During knowledge distillation, the teacher model’s parameters are frozen, and only the student model’s parameters are updated. Since category prototypes are not available at the beginning of training, we adopt warm-up strategy to compute teacher prototypes, i.e., during the initial warm-up epochs, only the teacher model computes the category prototypes without performing class-wise distillation based on the prototype network. In our experiments, the warm-up period is set to 1 epoch. Note that the proposed weight adjustment module and prototype network module are used only during training, and the student model does not increase any additional parameters or computation cost during the testing phase. IV-B Sample-wise Distillation weight adjustment module We start by evaluating the effectiveness of the feature weight adjustment module. Specifically, we propose two implementation approaches for the feature weight adjustment module. The propose of this module is to use the teacher’s inference results to represent the temporal distribution of video frame features, exploiting the variation in importance along the temporal dimension. In our implementation, we find that directly using the outputs from the teacher’s logits layer achieves good results. Our two approaches show different ways to use the ground truth labels and handle negative values. Approach 1 uses the ground truth labels to select the distribution of the correct class in the teacher’s output along the temporal dimension. After normalization and scaling, the distribution is used as the importance score for the feature maps: weight_scorei=Softmax(qiT×GTi),weight\_score_i=Softmax(q^T_i×GT_i), (13) where qiTq^T_i represents the probability distribution of the i-th sample generated by the teacher model, GTiGT_i is the ground truth label for the i-th sample. Approach 2 uses a masking mechanism to conceal results from the teacher that are negative, and only the results greater than zero are considered as valid importance scores for subsequent distillation loss calculation: weight_scorei=Softmax(mask(qiT×GTi))∘(qi,GT>0),weight\_score_i=Softmax(mask(q^T_i×GT_i)) (q^T_i,GT>0), (14) where mask(⋅)mask(·) is the masking function, ∘ is element-wise multiplication, (⋅)I(·) is the indicator function. Additionally, for comparison, we aggregate the teacher’s output results along the category dimension and serves as the importance score after normalization and scaling: weight_scorei=Softmax(fsum(qiT)),weight\_score_i=Softmax(f_sum(q^T_i)), (15) where fsum(⋅)f_sum(·) is the aggregate function. The reason for this approach is that the aggregated results represent the likelihood of an action occurring at a given moment, a higher score indicates a higher likelihood that the feature contains information about the corresponding action. Note that this method does not use label information. TABLE I: Comparison of different implementation methods for the weight adjustment module. Method Top-1(%) Top-5(%) Student baseline 63.36 82.08 Student+FeatDistill 69.92 87.81 Student+reweight(Approach 1) 70.45 87.74 Student+mask(Approach 2) 70.24 87.71 Student+sum(comparison) 70.26 87.55 We conduct experiments on the UCF101 dataset for the two proposed approaches and the comparison method. We also include the accuracy of the vanilla feature distillation for comparison. The results are shown in Table I. For fairness, we use the same parameter setting for all implementations. It can be discovered that all three approaches achieve an accuracy improvement of nearly 7%7\% compared to the baseline student model (63.36%63.36\%) and exceeded the vanilla feature distillation(69.92%69.92\%). Approach 1, which provides the highest accuracy improvement, reaches a final accuracy of 70.45%70.45\%. Approach 2 with masking mechanism and the method using probability aggregation followed in performance. Comparing approach 1 with approach 2, we conclude that directly discarding feature segments with negative scores during knowledge distillation is too aggressive. This approach also loses beneficial knowledge contained in background segments while suppressing irrelevant noise, which leads to the student model failing to capture the complete semantic features of the action. Comparing approach 1 with the probability aggregation method, we find that incorporating label information is crucial for approach 1’s superior performance. Label information indicates the ground truth category, and adjusting features based on the correct class probability distribution is more targeted and reliable than simply aggregating probabilities across all action categories. It also highlights the importance of using ground truth information to guide feature distillation. Therefore, in subsequent experiments, we adopt approach 1 to adjust video feature weight. TABLE I: Validation accuracy and computation cost on UCF101 and HMDB51. We set I3D as the teacher model, Top-I3D as the student model. For fair comparison, we keep the same training configuration for all methods. Model Knowledge UCF101 HMDB51 FLOPS(G) Params(M) Top-1(%) Top-5(%) Top-1(%) Top-5(%) Teacher - 90.35 98.76 69.02 88.76 111.3 12.7 Student - 63.36 82.08 52.35 74.97 45.5 10.5 KD[16] Logits 67.20 85.44 53.00 75.16 45.5 10.5 CTKD[20] Logits 67.63 88.37 52.65 75.01 45.5 10.5 LSKD[21] Logits 68.33 86.36 53.20 75.10 45.5 10.5 Feature KD[6] Feature 69.92 87.81 53.14 75.27 45.5 10.5 FN[22] Feature 70.95 88.54 53.25 75.34 45.5 10.5 MGD[23] Feature 71.43 89.39 54.15 76.07 45.5 10.5 SimKD[24] Feature 72.01 89.60 53.67 75.95 45.5 10.5 AT[43] Attention 71.27 88.57 52.93 75.14 47.4 37.8 CTKD[44] Logits+Feature 71.77 89.14 53.91 75.87 45.5 10.5 Ours Feature 73.51 90.30 54.77 76.41 45.5 10.5 IV-C Main Results To validate the effectiveness of our proposed method, we compared it with other knowledge distillation methods on the action recognition task. We used the I3D[9] model as the teacher model and the Top-I3D[15] model as the student model. The Top-I3D model is derived from the I3D model by replacing some shallow 3D convolutional blocks with 2D convolutional blocks to reduce the model’s parameters and computational resource requirements. As shown in Table I, our method achieves superior improvements in student model accuracy compared to existing knowledge distillation methods on both datasets. Specifically, on the UCF101 dataset, our method improves the Top-1 accuracy of the student model by 10.15%10.15\% and the Top-5 accuracy by 8.22%8.22\%, without introducing additional parameters or computational cost. Compared to our baseline algorithm, i.e., the vanilla feature distillation, our approach provides an additional 3.43%3.43\% and 2.20%2.20\% improvement in Top-1 and Top-5 accuracy, respectively. On the more challenging HMDB51 dataset, we also obtain similar results, with our method improving Top-1 and Top-5 accuracy by 2.42%2.42\% and 1.44%1.44\%, respectively. For fairness, we applied other knowledge distillation methods to the same teacher-student model pair. Given that feature-based knowledge contains more learnable information, our method easily outperforms recent state-of-the-art logits-based knowledge distillation methods. Compared to other recent feature-based knowledge distillation methods, such as MGD[23] method using a masking-recovery training paradigm and SimKD[24] using feature alignment and teacher classifier reuse strategy, our method also demonstrates superiority. We believe the improvement of our method comes from two aspects. On the one hand, the sample-wise feature distillation using the feature weight adjustment module leverages the teacher’s inference results and label information as guidance. This approach allows the student model to focus on learning features based on their contribution to the correct inference result, rather than blindly learning all features extracted by the teacher. On the second hand, the class-wise feature distillation based on the prototype network further aligns the teacher and student models’ category feature representations, enhancing the student model’s ability to extract high-level semantic information. In addition to accuracy improvements, the student model shows a significant reduction in computational and parameter costs compared to the teacher model. Specifically, the student model’s computational cost is reduced by 59.1%59.1\%, and the parameters is reduced by 17.3%17.3\%. Combined with our proposed method, this lightweight resource usage makes it feasible to deploy video analysis models on resource-constrained edge devices. IV-D Ablation Study To thoroughly investigate the validity of our proposed method, we systematically analyze the effects of each submodule and the generalization of the method across different backbone networks in this section. IV-D1 Contribution of Each Submodule TABLE IV: The impact of each submodule on model accuracy improvement on the UCF101 dataset. baseline ℒSample−wiseL_Sample-wise ℒClass−wiseL_Class-wise Top-1(%) Top-5(%) ✓ 63.36 82.08 ✓ ✓ 72.64 89.53 ✓ ✓ 70.47 88.11 ✓ ✓ ✓ 73.51 90.30 We first explore the impact of each submodule on the overall performance. We start with the baseline model, which is the student model trained solely with the classification loss. We then introduce sample-wise feature distillation and class-wise feature distillation separately. The former means the student model assigns different loss weights to different feature maps based on their importance, while the latter implies the student model’s loss function includes the cross-entropy between the student’s inference and ground truth label, and the distance between the feature produced by the student and the teacher’s prototype feature. Finally, we incorporate both the sample-wise distillation with feature weight adjustment and the class-wise distillation with prototype network. Table IV summarizes the ablation results for these four settings on the UCF101 dataset. It can be observed that both proposed distillation methods independently improve the accuracy of the student model, and introducing two submodules together further enhances the model’s performance. Specifically, compared to the baseline model, introducing sample-wise distillation improves the student model’s accuracy by 9.28%9.28\%, indicating that our module, by adjusting feature map weights, helps the student model focus more on the teacher’s features that contain richer action information, thereby improving the knowledge distillation effect. Similarly, introducing class-wise feature distillation improves the student model’s accuracy by 7.11%7.11\% compared to the baseline, showing that class-wise distillation also enables the student model to learn the teacher model’s feature extraction capability. Finally, combining both sample-wise and class-wise distillation methods further enhances the student model’s performance, with Top-1 accuracy increasing by 10.15%10.15\% and Top-5 accuracy increasing by 8.22%8.22\% compared to the baseline, achieving a final Top-1 accuracy of 73.51%73.51\% and Top-5 accuracy of 90.30%90.30\%. IV-D2 Effectiveness of Weight Adjustment Module TABLE V: Comparison between the effects of sample-wise knowledge distillation and traditional feature distillation on the UCF101 dataset. baseline Feature Dist. Sample-wise Dist. Top-1(%) Top-5(%) ✓ 63.36 82.08 ✓ ✓ 69.92 87.81 ✓ ✓ 72.64 89.53 Our proposed weight adjustment module for sample-wise distillation is an improvement over vanilla feature distillation methods. To demonstrate that the effectiveness of the weight adjustment module is not solely due to vanilla feature distillation, we compare the results of sample-wise distillation with those of vanilla feature distillation. Experiments are conducted on the UCF101 dataset, and the results are shown in Table V. As observed, our proposed sample-wise distillation method surpasses vanilla feature distillation method by nearly 2.72%2.72\% in Top-1 accuracy. This indicates that the accuracy improvement provided by our sample-wise distillation method is not only due to vanilla feature distillation but also due to its ability to selectively focus on frames that are more critical for correct inference. IV-E Generalization of Our Method TABLE VI: The improvement of our proposed method on different student models on the UCF101 dataset. Model Top-1(%) Top-5(%) Top-I3D 63.36 82.08 Top-I3D+Ours 73.51 90.30 Bottom-I3D 53.64 72.69 Bottom-I3D+Ours 57.73 76.87 I2D 56.07 77.24 I2D+Ours 64.98 83.27 To demonstrate the generalization of our proposed method, we select a series of variant I3D models[15] as student models. Specifically, in addition to the Top-I3D model, we also chose Bottom-I3D and I2D models for experiments. The results are shown in Table VI. As observed, for Top-I3D model, our method improve Top-1 and Top-5 accuracy by 10.15%10.15\% and 8.22%8.22\%, respectively, achieving final accuracies of 73.51%73.51\% and 90.30%90.30\%. For Bottom-I3D model, which is derived by replacing deep 3D convolutional layers with 2D convolutional layers, our method improve Top-1 and Top-5 accuracy by 4.09%4.09\% and 4.18%4.18\%, respectively, reaching final accuracies of 57.73%57.73\% and 76.87%76.87\%. Lastly, for I2D model, which is composed entirely of 2D convolutional layers, our method improve Top-1 and Top-5 accuracy by 8.91%8.91\% and 6.03%6.03\%, respectively, achieving final accuracy of 64.98%64.98\% and 83.27%83.27\%. These results demonstrate that our knowledge distillation method is effective even when transferring knowledge from a pure 3D network teacher to a pure 2D network student, proving the method’s effectiveness across different network architectures. IV-F Hyperparameter Sensitivity Study In our method, the hyperparameters α and β serve as the weights for the loss functions in sample-wise distillation and class-wise distillation, respectively. Table VII and Table VIII show the impact of these hyperparameters on the method’s performance improvement on the UCF101 dataset when set to different values. Notably, since the method showed the most significant improvement when α=4α=4, we retained α=4α=4 in the sensitivity experiments for β. TABLE VII: Sensitivity experiment of the hyperparameter α on the UCF101 dataset Parameter 1 2 4 8 Top-1 Top-5 Top-1 Top-5 Top-1 Top-5 Top-1 Top-5 α 70.92 88.50 71.56 88.98 72.64 89.53 72.32 89.85 TABLE VIII: Sensitivity experiment of the hyperparameter β on the UCF101 dataset Parameter 1 2 4 8 Top-1 Top-5 Top-1 Top-5 Top-1 Top-5 Top-1 Top-5 β 73.38 90.06 73.51 90.30 73.35 90.01 72.91 89.88 As shown in tables, regardless of the values of α and β, the model’s accuracy always improves compared to the baseline accuracy. Specifically, when α=4α=4 and β=2β=2, the method achieves the most significant accuracy improvement, reaching a Top-1 accuracy of 73.51%73.51\% and a Top-5 accuracy of 90.30%90.30\%. Therefore, we keep these hyperparameters throughout the experiments. This sensitivity analysis confirms that our method is robust across different hyperparameter settings, with the chosen values of α=4α=4 and β=2β=2 providing the optimal balance for performance improvement. V Conclusion In this paper, we propose Label-Guided Knowledge Distillation to explicitly exploit the temporal importance of teacher features. Our approach conducts knowledge distillation at both the sample-wise and class-wise levels. By introducing sample-wise knowledge distillation, the student model can selectively learn teacher features, focusing more on those that significantly impact classification results. By introducing class-wise knowledge distillation, the student model can acquire relational knowledge among samples within the same class, mimicking the teacher’s ability to capture high-level semantic information, thereby compacting feature representations and improving model generalization. Experimental results show that our method effectively enhances the capabilities of lightweight 3D-CNN models, further exploring the trade-off between resource efficiency and model accuracy. Additionally, this paper provides insightful research into the full utilization of label information in knowledge distillation. For future work, we believe that exploring finer-grained label information to guide feature distillation is a promising direction. References [1] L. Zhao, N. B. Gundavarapu, L. Yuan, H. Zhou, S. Yan, J. J. Sun, L. Friedman, R. Qian, T. Weyand, Y. Zhao et al., “Videoprism: A foundational visual encoder for video understanding,” arXiv preprint arXiv:2402.13217, 2024. [2] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid, “Vivit: A video vision transformer,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, p. 6836–6846. [3] M. Ji, S. Shin, S. Hwang, G. Park, and I.-C. Moon, “Refine myself by teaching myself: Feature refinement via self-knowledge distillation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, p. 10 664–10 673. [4] I. Chung, S. Park, J. Kim, and N. Kwak, “Feature-map-level online adversarial knowledge distillation,” in International Conference on Machine Learning. PMLR, 2020, p. 2006–2015. [5] J. Shao, X. Wang, R. Quan, J. Zheng, J. Yang, and Y. Yang, “Action sensitivity learning for temporal action localization,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, p. 13 457–13 469. [6] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “Fitnets: Hints for thin deep nets,” arXiv preprint arXiv:1412.6550, 2014. [7] X. Wang, S. Zhang, J. Cen, C. Gao, Y. Zhang, D. Zhao, and N. Sang, “Clip-guided prototype modulating for few-shot action recognition,” International Journal of Computer Vision, vol. 132, no. 6, p. 1899–1912, 2024. [8] Y. Liu, Y. Zhang, Y. Wang, F. Hou, J. Yuan, J. Tian, Y. Zhang, Z. Shi, J. Fan, and Z. He, “A survey of visual transformers,” IEEE Transactions on Neural Networks and Learning Systems, 2023. [9] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in CVPR, 2017. [10] H. Xu, A. Das, and K. Saenko, “R-c3d: Region convolutional 3d network for temporal activity detection,” in ICCV, 2017. [11] S. Ji, W. Xu, M. Yang, and K. Yu, “3d convolutional neural networks for human action recognition,” TPAMI, 2012. [12] K. Hara, H. Kataoka, and Y. Satoh, “Learning spatio-temporal features with 3d residual networks for action recognition,” in ICCV, 2017. [13] D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri, “A closer look at spatiotemporal convolutions for action recognition,” in CVPR, 2018. [14] Z. Qiu, T. Yao, and T. Mei, “Learning spatio-temporal representation with pseudo-3d residual networks,” in ICCV, 2017. [15] S. Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy, “Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification,” in ECCV, 2018. [16] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015. [17] Y. Tang, Y. Wang, J. Guo, Z. Tu, K. Han, H. Hu, and D. Tao, “A survey on transformer compression,” arXiv preprint arXiv:2402.05964, 2024. [18] X. Xu, M. Li, C. Tao, T. Shen, R. Cheng, J. Li, C. Xu, D. Tao, and T. Zhou, “A survey on knowledge distillation of large language models,” arXiv preprint arXiv:2402.13116, 2024. [19] Y. Gu, L. Dong, F. Wei, and M. Huang, “Minillm: Knowledge distillation of large language models,” in The Twelfth International Conference on Learning Representations, 2024. [20] Z. Li, X. Li, L. Yang, B. Zhao, R. Song, L. Luo, J. Li, and J. Yang, “Curriculum temperature for knowledge distillation,” in AAAI, 2023. [21] S. Sun, W. Ren, J. Li, R. Wang, and X. Cao, “Logit standardization in knowledge distillation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, p. 15 731–15 740. [22] K. Xu, L. Rui, Y. Li, and L. Gu, “Feature normalized knowledge distillation for image classification,” in ECCV, 2020. [23] Z. Yang, Z. Li, M. Shao, D. Shi, Z. Yuan, and C. Yuan, “Masked generative distillation,” in ECCV, 2022. [24] D. Chen, J.-P. Mei, H. Zhang, C. Wang, Y. Feng, and C. Chen, “Knowledge distillation with the reused teacher classifier,” in CVPR, 2022. [25] N. Crasto, P. Weinzaepfel, K. Alahari, and C. Schmid, “Mars: Motion-augmented rgb stream for action recognition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, p. 7882–7891. [26] J. Stroud, D. Ross, C. Sun, J. Deng, and R. Sukthankar, “D3d: Distilled 3d networks for video action recognition,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, p. 625–634. [27] P. Lee, T. Kim, M. Shim, D. Wee, and H. Byun, “Decomposed cross-modal distillation for rgb-based temporal action detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, p. 2373–2383. [28] R. Dai, S. Das, and F. Bremond, “Learning an augmented rgb representation with cross-modal knowledge distillation for action detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, p. 13 053–13 064. [29] H. Ullah and A. Munir, “A 3dcnn-based knowledge distillation framework for human activity recognition,” Journal of Imaging, vol. 9, no. 4, p. 82, 2023. [30] J. Snell, K. Swersky, and R. Zemel, “Prototypical networks for few-shot learning,” Advances in neural information processing systems, vol. 30, 2017. [31] P. Zhang, B. Zhang, T. Zhang, D. Chen, Y. Wang, and F. Wen, “Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, p. 12 414–12 424. [32] K. Wang, Y. Wang, X. Xu, X. Liu, W. Ou, and H. Lu, “Prototype-based selective knowledge distillation for zero-shot sketch based image retrieval,” in Proceedings of the 30th ACM international conference on multimedia, 2022, p. 601–609. [33] L. Ge, C. Hu, G. Ma, H. Zhang, and J. Liu, “Prokd: An unsupervised prototypical knowledge distillation network for zero-resource cross-lingual named entity recognition,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 11, 2023, p. 12 818–12 826. [34] B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y. Choi, “A comprehensive overhaul of feature distillation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, p. 1921–1930. [35] D.-A. Huang, V. Ramanathan, D. Mahajan, L. Torresani, M. Paluri, L. Fei-Fei, and J. C. Niebles, “What makes a video a video: Analyzing temporal information in video understanding models and datasets,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, p. 7366–7375. [36] S. Yun, J. Park, K. Lee, and J. Shin, “Regularizing class-wise predictions via self-knowledge distillation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, p. 13 876–13 885. [37] W. Park, D. Kim, Y. Lu, and M. Cho, “Relational knowledge distillation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, p. 3967–3976. [38] R. J. Hyndman, “Moving averages.” 2011. [39] K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,” arXiv preprint arXiv:1212.0402, 2012. [40] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre, “Hmdb: a large video database for human motion recognition,” in ICCV, 2011. [41] T. Brox, C. Bregler, and J. Malik, “Large displacement optical flow,” in CVPR, 2009. [42] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, p. 248–255. [43] S. Zagoruyko and N. Komodakis, “Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,” in ICLR, 2017. [44] H. Zhao, X. Sun, J. Dong, C. Chen, and Z. Dong, “Highlight every step: Knowledge distillation via collaborative teaching,” IEEE Transactions on Cybernetics, 2020.