Paper deep dive
Cross-Sign Language Transfer Learning Using Domain Adaptation with Multi-scale Temporal Alignment
Keren Artiaga, Yang Li, Ercan Engin Kuruoglu, Wai Kin, Chan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/23/2026, 2:53:32 AM
Summary
This paper addresses the challenge of Sign Language Recognition (SLR) for low-resource languages by proposing a cross-sign language transfer learning approach using Domain Adaptation. Specifically, the authors utilize the Temporal Attentive Adversarial Adaptation Network (TA3N) with a Temporal Relation Network (TRN) module to align multi-scale temporal relations between source domains (Argentine Sign Language, Chinese Sign Language) and a target domain (American Sign Language). The study compares input modalities (RGB vs. Optical Flow) and learning settings (full-scale vs. few-shot), finding that RGB outperforms Optical Flow and that aligning shorter-term temporal relations yields better accuracy.
Entities (12)
Relation Signals (9)
TA3N → usesmodule → TRN
confidence 98% · TA3N, which utilizes the Temporal Relational Network (TRN) module for aligning multi-scale temporal relations.
TA3N → appliedto → Sign Language Recognition
confidence 97% · we present our work on sign language recognition using transfer learning and the domain adaptation method TA3N
TRN → aligns → multi-scale temporal relations
confidence 96% · utilizes the Temporal Relational Network (TRN) module for aligning multi-scale temporal relations.
Argentine Sign Language → sourcedomainfor → American Sign Language
confidence 95% · Our study focuses on Argentine Sign Language (LSA) and Chinese Sign Language (CSL) as the source domains, with American Sign Language (ASL) as the target domain.
Chinese Sign Language → sourcedomainfor → American Sign Language
confidence 95% · Our study focuses on Argentine Sign Language (LSA) and Chinese Sign Language (CSL) as the source domains, with American Sign Language (ASL) as the target domain.
RGB → outperforms → Optical Flow
confidence 94% · ultimately determining that RGB outperforms Optical Flow in the majority of cases.
WLASL300 → contains → American Sign Language
confidence 92% · WLASL300 13 datasets for LSA, CSL, and ASL, respectively.
LSA64 → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Sign language serves as a vital means of communication for individuals with hearing impairments, yet recognition resources for the over 100 distinct sign languages are severely lacking. In response, we present our work on sign language recognition using transfer learning and the domain adaptation method TA3N, which utilizes the Temporal Relational Network (TRN) module for aligning multi-scale temporal relations. Our findings highlight the superior performance of Domain Adaptation to neural network-based transfer learning, particularly in improving recognition of American Sign Language (ASL). Our research also identifies the effectiveness of aligning shorter-term temporal features between source and target domains. In addition to using RGB, we conducted experiments using Optical Flow mode for the sign language samples, ultimately determining that RGB outperforms Optical Flow in the majority of cases. Our work aims to improve accessibility and communication for individuals who rely on sign language as their primary mode of communication.
Tags
Links
- Source: https://arxiv.org/abs/2608.16804v1
- Canonical: https://arxiv.org/abs/2608.16804v1
Trouble viewing inline? Open PDF directly →
Full Text
64,282 characters extracted from source content.
Expand or collapse full text
2021 This version of the article has been accepted for publication, after peer review and is subject to Springer Nature’s AM terms of use, but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: https://doi.org/10.1007/s11042-023-16703-0 Sign language serves as a vital means of communication for individuals with hearing impairments, yet recognition resources for the over 100 distinct sign languages are severely lacking. In response, we present our work on sign language recognition using transfer learning and the domain adaptation method TA3N, which utilizes the Temporal Relational Network (TRN) module for aligning multi-scale temporal relations. Our findings highlight the superior performance of Domain Adaptation to neural network-based transfer learning, particularly in improving recognition of American Sign Language (ASL). Our research also identifies the effectiveness of aligning shorter-term temporal features between source and target domains. In addition to using RGB, we conducted experiments using Optical Flow mode for the sign language samples, ultimately determining that RGB outperforms Optical Flow in the majority of cases. Our work aims to improve accessibility and communication for individuals who rely on sign language as their primary mode of communication. Cross-Sign Language Transfer Learning Using Domain Adaptation with Multi-scale Temporal Alignment Keren Artiaga Email: kerenartiaga@outlook.com Affiliation: Tsinghua-Berkeley Shenzhen Insitute, Tsinghua Shenzhen International Graduate School, The University Town, Nanshan District, Shenzhen, 518055, P.R.China Yang Li Email: yangli@sz.tsinghua.edu.cn Affiliation: Tsinghua-Berkeley Shenzhen Insitute, Tsinghua Shenzhen International Graduate School, The University Town, Nanshan District, Shenzhen, 518055, P.R.China Ercan Engin Kuruoglu Email: kuruoglu@sz.tsinghua.edu.cn Affiliation: Tsinghua-Berkeley Shenzhen Insitute, Tsinghua Shenzhen International Graduate School, The University Town, Nanshan District, Shenzhen, 518055, P.R.China Wai Kin (Victor) Chan Email: chanw@sz.tsinghua.edu.cn Affiliation: Tsinghua-Berkeley Shenzhen Insitute, Tsinghua Shenzhen International Graduate School, The University Town, Nanshan District, Shenzhen, 518055, P.R.China keywordssign language recognition, domain adaptation, temporal relations, optical flow 1 Introduction The World Health Organization predicts that by 2050, one out of ten people will have some degree of hearing loss. Translation between sign languages and spoken languages is crucial for facilitating communication between sign language users and those who do not use sign language. However, Sign Language Recognition (SLR) is not as extensively researched as other types of action recognition, and SLR studies tend to focus on only a few of the 135 sign languages used worldwide due to the lack of high-resource datasets for most of these sign languages. It is common knowledge that low-resource datasets are prone to overfitting in classification or recognition tasks. Our solution to the low-resource dataset problem for SLR is cross-sign language transfer learning, where we transfer knowledge from higher-resource sign language datasets. This method is domain-specific and differs from state-of-the-art SLR methods that use non-sign language-specific datasets, such as ImageNet. By applying sign language to sign language transfer learning, we can improve SLR by learning high-level, specific visual features that cannot be provided by general large-scale non-sign language datasets, which are limited to low-level, superficial features. By definition, transfer learning is a machine learning technique that works by applying the knowledge gained in one domain to another. Specifically, this paper aims at achieving effective cross-sign language transfer learning using Domain Adaptation. Our study focuses on Argentine Sign Language (LSA) and Chinese Sign Language (CSL) as the source domains, with American Sign Language (ASL) as the target domain. However, the methodology presented in this paper can be universally applied to any sign language recognition task. We decided on ASL to be the target domain as being a low-resource dataset, it has a less number of samples per class than LSA and CSL, as seen in Table 2. Domain adaptation is a subset of transfer learning that brings the target task’s distribution closer to that of the source. There are studies 18; 14; 15; 2 on neural network-based transfer learning, specifically pre-training applied to SLR, and a study on domain adaptation between ASL copy-sign data and ASL native data 17. However, there are currently no studies on the application of domain adaptation between different sign languages. Moreover, as evidenced by the results of our pre-training experiments 4.3, it is clear that pre-training is not sufficient for domains with different distributions, such as sign languages. Therefore, we deem it advantageous to utilize Domain Adaptation. Our major contribution, specifically, is the use of multi-scale temporal alignment to align the domain of the source sign language to that of the target. We furthered our domain adaptation with multi-scale temporal alignment work by determining the different effects of aligning the source and target domains’ temporal relations at both short-term and long-term timescales. Temporal relations can be defined as a group of select time-ordered frames of a video. From the results of our experiment, we were able to prove our hypothesis that aligning shorter-term relations instead of longer-term ones is the better approach for improving the accuracy of the target SLR model in all adaptations (from LSA to ASL and from CSL to ASL) and in all input modalities we used in this study. We experimented in two learning settings: full-scale transfer learning and few-shot transfer learning. In full-scale transfer learning, we use an 80:20 ratio of target training data to target test data for each k-fold cross-validation. Few-shot transfer learning aims to create an accurate model with a test base using less target training data, mimicking human learning. Compared to full-scale learning, few-shot learning requires less training data and computational resources. We implement few-shot transfer learning with a 20:80 ratio of target training data to target test data. We used RGB and Optical Flow as input modalities, which can easily be generated from video frames, eliminating the need for additional equipment such as a specialized camera used in several SLR works, such as 2. We specifically implemented the Gunner Farneback optical flow algorithm6, which has the advantage of being invariant to appearance21, making generalization easier for low-resource sign language datasets. The datasets we used in this work are LSA64: A Dataset of Argentinian Sign Language 19 , Chinese isolated SLR dataset 32; 16; 9 and WLASL300 13 datasets for LSA, CSL, and ASL, respectively. To summarize, our contributions are as follows: (1) We implemented multi-scale temporal alignment (Sec. 3.2.1) to conduct domain adaptation between different sign languages (2) We determined the different effects of aligning temporal relations of the sign languages at both short-term and long-term timescales (3) We conducted domain adaptation in two learning settings: full-scale transfer learning (Sec. 4.2.1) and few-shot transfer learning (Sec. 4.2.2) (4) We used two input modalities for the samples: RGB and Optical Flow (Sec. 3.1), and provided a comparison of their performance for each experiment (Sec. 4.2). This paper is organized as follows: In Section 2, related works on Sign Language Recognition (SLR) and Domain Adaptation are discussed. Section 3 outlines the technical approach used in the study, including pre-processing of the videos and details of the sign language models. Section 4 presents the dataset, experimentation setup, results, and analysis. Finally, Section 6 provides conclusions and suggestions for future research in this field. 2 Related Works 2.1 Related Works on Sign Language Recognition In SLR, transfer learning is commonly done by pre-training on large-scale datasets like ImageNet of a different domain and using its weights to initialize the target’s weights. One of the earliest applications of transfer learning in SLR is the creation of ASL word models by Farhadi et al. 5 that transfer between different signers as well as different aspects. For these ASL word models, the authors used one subject in the source dataset and one in the target dataset. The subject for the source is an avatar. Nishat et al. 15 conducted unsupervised transfer learning to the Bangla sign language dataset (BsDL) to recognize Bangla character sign language such as digits, vowels, and consonants using VGG16 architecture pre-trained on ImageNet. Another work on BsDL is by Das et al. 4 wherein a hybrid approach using deep transfer learning model with random forest classifier is used. Cayamcela et al. 14 used a convolutional neural network (CNN) pre-trained on the ImageNet dataset to help translate American Sign Language Alphabet in real time. Both 14 and 15 are dealing with static signs where the inputs to the network are still images and not videos. Recently, Halvardsson et al. 7, developed a Swedish Sign Language Alphabet model that uses an InceptionV3 network that is also pre-trained on ImageNet. This study involves 8 subjects and similar to 14, 15 and 4, does not utilize any videos as inputs. Recent research has shown that pre-training neural networks on large-scale datasets like ImageNet can improve the accuracy of sign language recognition (SLR). This method is currently considered the state-of-the-art in transfer learning for SLR, as demonstrated in 18; 7; 14; 15; 31; 22; 27; 4; 11; 23. Meanwhile, a study by Suharjito et al. 25 uses both ImageNet as well as kinetic dataset - an action recognition dataset, as source for training their Indonesian Signal System recognition model. However, domain-specific transfer learning is gaining popularity in improving neural network models for SLR. Bird et al. 2 used transfer learning by pre-training with British Sign Language (BSL) to improve recognition of American Sign Language (ASL) using VGG16 architecture. The study included 18-class BSL and ASL datasets, with ASL consisting of one-handed signs from two subjects. The authors found that the late fusion of RGB and Leap Motion approaches (multimodality) in SLR improved their ASL recognition model compared to using RGB or LM alone. They suggested that similarities in movement between the selected BSL and ASL gestures transferred useful knowledge between the two sign languages. Another domain-specific transfer learning study was conducted by Abdullayeva et al. 1, where they pre-trained their Azerbaijian fingerspelling (alphabet) recognition model on both the ImageNet dataset and an ASL alphabet dataset. To the best of our knowledge, no previous research has explored few-shot transfer learning in SLR. Existing studies are limited by the type of sign language they can recognize (i.e. static signs such as alphabet, characters, and digits), the number of subjects in their datasets, or the type of gestures used. Our work is particularly challenging as we focus on ASL, a low-resource language with two-handed gestures performed by different subjects, with an average of only 1.3 repetitions per sign in the dataset. Additionally, the ASL dataset we used is sourced from various online platforms with different lighting, backgrounds, aspects, and subject distances from the camera. The most relevant work to our study 2 only included one-handed gestures from two signers. See Table 11 for a detailed comparison with other studies. 2.2 Related Works on Domain Adaptation Domain Adaptation is a type of Transfer Learning that aims to generalize a model trained in one domain to perform well in another domain. This is necessary because models trained in one domain may not work well in another domain due to domain shift. Domain shift occurs when the source and target domains have different probability distributions, even though they share the same task. To address this issue, domain adaptation adjusts the target data’s distribution to match that of the source data. While both neural network-based transfer learning and domain adaptation can help networks learn discriminative features for low-resource settings, their execution differs. Neural network-based transfer learning reuses the source task’s features and weights in the target task’s network. Domain Adaptation, on the other hand, attempts to reduce shift between two different domains by re-weighting source features. Similar to video classification tasks, Sign Language classification can suffer from domain discrepancy in both spatial and temporal aspects. Hence, there is a need for domain adaptation approaches, specifically for video recognition and classification. Currently, there are only a few studies that apply Domain Adaptation to video classification 26; 30; 10; 20. Only Temporal Attentive Adversarial Adaptation Network (TA3N) 3, Temporal Attentive Moment Alignment Network (TAMAN) 30, and Contrast and Mix 20 are tested on large-scale datasets. TAMAN is created for multi-source video domain adaptation, while Contrast and Mix are designed for unsupervised domain adaptation. Although TAMAN and Contrast and Mix may be applied to a single-source problem, we decided to use TA3N as it incorporates within its architecture an extensible, plug-and-play module called Temporal Relation Network (TRN) 33 that generates multiscale temporal relations. TRN is based on humans’ ability to connect meaningful transformations in an object or event without observing all changes, known as temporal relational reasoning. TRN aims to discover important temporal relations between frames in videos when applied to neural networks. Technical implementation of TRN is described rigorously in Section 3.2.1 Our aim is to determine the optimal timescale (short-term or long-term) for aligning the temporal relations of the source and target domains. To achieve this goal, we concluded that TA3N, which includes the TRN module, is the most suitable architecture for our cross-sign language domain adaptation task. 3 Technical Approach Our technical approach is a two-step process that consists of a pre-processing step and a domain adaptation step as shown in Figure 1. In the pre-processing step, we convert videos into RGB frames and Optical Flow frames. After generating the frames, we proceeded with the transfer learning step, where we use the Domain Adaptation approach. Domain Adaptation differs from neural network-based transfer learning in that it is utilized when the source and target tasks share a similar feature space but have distinct probability distributions due to having unrelated datasets. This is particularly relevant in cases where we are adapting the domains of CSL and LSA to ASL. We implemented Domain Adaptation in five different multiscale temporal relations: 3, 5, 7, 10, and 15. The shortest term is 3 and the highest term is 15. Section 3.2.1 describes multiscale temporal relations in detail. The experiment results in Section 4.2 show in detail the effects of N different multi-scale TRN on the recognition accuracy of the target domain. Figure 1: The process involved two steps: pre-processing and domain adaptation, to generate an SLR model. In the pre-processing step, we convert videos into both Optical Flow and RGB frames. The features of these frames are then fed into the Domain Adaptation network five times, each time utilizing a different N-multiscale temporal relations network (TRN) to determine the optimal timescale for aligning the temporal relations of the sign language domains. 3.1 Pre-processing of Videos To facilitate cross-sign language transfer learning, we first convert selected individual gesture videos into Optical Flow and RGB frames. To capture the sequential differences between gestures, we experimented with the Optical Flow input mode, which tracks the motion of pixels between frames. We extracted a maximum of 200 frames per sample, as most samples have less than 100 frames. The Figures 2, 3, 4 and 5 show the subsampled frames from a collection of frames for the gesture Copy in CSL and ASL. For Optical Flow, the backgrounds are changed to white from black for the readers of this paper to see the hand movements more clearly. Figure 2: Subsampled RGB frames (starting from left to right) for the gesture Copy in ASL from WLASL dataset Figure 3: Subsampled optical flow frames (starting from left to right) for the gesture Copy in ASL from WLASL dataset Figure 4: Subsampled RGB frames (starting from left to right) for the gesture Copy in LSA from LSA64 dataset Figure 5: Subsampled optical Flow frames (starting from left to right) for the gesture Copy in LSA from LSA64 dataset It can be observed that for the optical flow mode of the ASL gesture Copy in Figure 3, the resulting images are pixelated in comparison to its LSA counterpart in Figure 5. The video quality of the samples in ASL is on average poorer than those of CSL and LSA. This is also a reason for selecting ASL as the target domain to improve using Domain Adaptation, in addition to having fewer samples per class. 3.2 Domain Adaptation for Sign Classification 3.2.1 Temporal Relations Network We are using the Temporal Attentive Adversarial Adaptation Network (TA3N) 3 to effectively align two different sign language domains. TA3N aligns multi-scale temporal relation networks of source and target samples by assigning weights to them. The more domain discriminative the temporal relations are, the higher weights they will receive. Section 3.2.2 explains the implementation of TA3N. The multi-scale temporal relations network used in TA3N is from a separate module called Temporal Relation Network (TRN) 33 by Zhou et al. These temporal relation features are generated by combining the CNN features of N equidistant frames that are sparsely sampled. A single relation feature that represents N number of frames is referred to as an N-frame temporal relation. The function below defines a 2-frame temporal relation: T2(V)=hΦ(∑i<jgΘ(fi,fj))T_2(V)=h_ ( _i<jg (fi,fj)) wherein T2(V)T_2(V) means 2-frame temporal relation of the video V with n chosen time-ordered frames f1,f2,…,fnf_1,f_2,...,f_n. The function gΘg_ and gΦg_ fuse the features of different time-ordered frames. The uniformly sampled i and j frames need not be consecutive so as long as they are sorted chronologically. For a 3-frame temporal relation, below is the corresponding composite function: T3(V)=hΘ(∑i<j<kgΘ(fi,fj,fk))T_3(V)=h_ ( _i<j<kg (fi,fj,fk)) The final output of the TRN module that is implemented in TA3N is a multiscale TRN which is a fusion of different N-frame temporal relations at multiple time scales. The function below shows the accumulation of N-frame temporal relations at N-time scales: MTN(V)=T2(V)+T3(V)…+TN(V)MT_N(V)=T_2(V)+T_3(V)...+T_N(V) Figure 6: TRN showing the process of generating N-frame temporal relations and finally a multiscale temporal relation for an ASL gesture The TRN module determines the representative frames to be used by converting each frame into features and fusing them into different N-frame temporal relation features, which are then passed into the TRNs. For instance, for 2-frame relations, the TRN module will vote for the combination of two frames that best classify the video. TA3N focuses on aligning all N-frame temporal relation features TN(V)T_N(V) of the source domain’s multi-scale TRN MTN(Vs)MT_N(V_s) and the target domain’s multi-scale TRN MTN(Vt)MT_N(V_t), whereas our objective is to learn the best N of multi-scale TRN MTNMT_N for adapting the domain of one sign language to another. Further details of the TA3N architecture are described below in Section 3.2.2. 3.2.2 Detailed TA3N Architecture In Figure 7, TA3N applied to our SLR task is shown. The process begins with raw video frames inputted into a convolutional network (CNN) to convert them into frame-level feature vectors, which are then passed to the Spatial Module. This module contains multilayer perceptrons to extract the spatial information of objects, such as hands, in a video, and convert the feature vectors into task-driven feature vectors for video classification. The task-driven feature vectors are generated and then sent to the Temporal Relation Network (TRN) Module. The TRN module extracts N-frame relations to generate a multiscale TRN MTNMT_N, as shown in Figure 7. The process of the TRN module is described in detail in Section 3.2.1. The next step involves Domain Adversarial Training for class and domain prediction, where two networks are used: the Temporal Classification Network for class prediction and the Temporal Adversarial Discriminator for domain prediction. Each network consists of a multi-layer perceptron (MLP) with a fully connected layers network that attends to the N-frame temporal relations of the multiscale TRN generated in the previous step. Figure 7: The architecture of TA3N applied on Sign Language Recognition task. The network will assign higher weights to N-frame temporal relations that will score low in Temporal Domain Loss which is calculated by the Temporal Adversarial Discriminator. This discriminator will calculate the loss based on the domain prediction. If the domain prediction is correct, the Temporal Adversarial Discriminator will output a low domain loss. Through backpropagation, the Domain Attention block will update the weights of the N-frame temporal relations TN(V)T_N(V) of the target domain’s multi-scale TRN MTN(Vt)MT_TN(V_t) where t means target domain in an attempt to increase the domain loss. A fully connected layer as seen in Figure 7 converts the N-frame relations into class predictions to which Prediction Loss is computed. Prediction Loss must be calculated as well because, as discussed in Section 2.2, Domain Adaptation works by minimizing the class prediction loss and then maximizing the domain prediction loss. The final loss to be calculated is called Attentive Entropy Loss which is a product of domain entropy and class entropy. Domain entropy and class entropy are calculated from domain loss and class loss, respectively. TA3N added this type of loss in the architecture to enhance the certainty of videos that have low domain discrepancy to enable it to focus more on videos, specifically their N-frame relations, that have high domain discrepancy by assigning higher weights to them during backpropagation. When the model experiences low domain discrepancies, indicated by lower domain prediction loss and low attentive entropy loss, it implies confusion between the source and target, whereas high domain discrepancies are observed when the model accurately predicts the domain, indicating high discrepancies between the source and target. The network needed to enhance the certainty of low-domain discrepant videos as their N-frame temporal relations do not need attention weights as much as those of high-domain discrepant videos. In Section 2.2, we elaborate on why we opted for TA3N for our SLR tasks, as well as its distinguishing features from other related domain adaptation architectures. 3.2.3 Fine-Tuning TA3N for Sign Language Recognition N-multiscale TRN MTN(Vs)MT_N(V_s) is based on Temporal Relation Network (TRN) 33 which is an essential parameter for the TA3N architecture as it is used to determine the number of multiple temporal relations to align between the source and the target domain. Section 3.2.1 describes TRN in detail. In the TA3N architecture, the multi-scale TRN parameter is set to 5 by default; therefore, all of TA3N’s experiments with large-scale action recognition datasets are conducted in 5-multiscale TRN. Therefore, we argue that we can fine-tune this value for SLR by using a smaller N-multiscale TRN. For our study, we compared the performance of 3, 5, 7, 10, and 15 multi-scale TRNs to prove our hypothesis that short-term multi-scale TRNs would produce better results for SLR domain adaptation. We refer to 3, 5, and 7 multi-scale TRNs as shorter-term multiscale TRNs while 10 and 15 as longer-term multiscale TRNs. The reason shorter-term multiscale TRNs would perform better is that they would prevent overfitting, especially in the case where sign languages have significant differences from each other. It is worth noting that LSA, CSL, and ASL are from different sign language families. In our experiment results and analysis section ( see Section 4.2 ), we would select the best multi-scale TRN for RGB and Optical Flow in each LSA to ASL and CSL to ASL adaptations. Lastly, we implemented TA3N in a supervised manner by setting the use_target parameter to sV (i.e. supervised). The default learning implementation of TA3N is unsupervised, that is, the classification loss is calculated solely from the labeled source data. By modifying the use_target parameter to sV, the classification loss in our experiments would now be based on the concatenation of the labeled source and target data. 3.3 Few-Shot Transfer Learning We conducted experiments using both full-scale transfer learning and few-shot transfer learning settings. Few-shot transfer learning is a type of Few-Shot Learning (FSL) that aims to achieve good learning performance with a limited training set in a supervised manner 29. We used few-shot transfer learning to train our SLR model because we wanted it to classify sign language gestures with just a few examples, similar to how humans learn. Our target domain, ASL, has an average of about 14 samples per class. For full-scale transfer learning, we split the target training and test data in an 80/20 ratio. For few-shot transfer learning, we used a 20/80 split, resulting in a limited target training data of approximately 2 samples per class. 4 Experiments TA3N was originally tested on large-scale action recognition datasets, HMDB51 12 and UCF101 24, using a ResNet-101 architecture. However, its default implementation showed poor performance on sign language recognition. We hypothesize that this is mainly due to the lower number of samples in comparison to HMDB51 and UCF101 datasets, which is a common characteristic among sign language datasets. Hence, we needed to modify the default implementation to fit a more domain-specific, low-resource task of sign language recognition. We experimented to determine the effects of using shorter-term and longer-term multiscale TRNs on the accuracy of our target sign language recognition model. We also wanted our model to learn from few-shot transfer learning settings. Additionally, we aim to compare RGB inputs against Optical Flow inputs. Section 4.2 will show how our SLR models performed in the following scenarios: (1) short-term versus long-term multiscale TRNs, (2) full-scale versus few-shot transfer learning, (3) RGB vs Optical Flow input mode. 4.1 Experiment Setup This study utilizes the CSL Isolated Chinese Sign Language Dataset 32; 16; 9 for CSL, the Word-Level American Sign Language (WLASL300) 13 dataset for ASL, and LSA64:A Dataset of Argentinian Sign Language 19 for LSA. The table presented below displays the classes that we used for the domain adaptation from LSA to ASL and CSL to ASL. Table 1: List of classes used in all transfer learning experiments. and WLASL300 labels LSA and WLASL300 labels deaf red red green green yellow yellow colors colors son son call call milk milk none none name name chair water thin man white women work learn blanket country coffee where friend birthday future music husband candy light catch new help student dance ugly buy copy run give of 23 classes Total of 26 classes To ensure sufficient training and test sets, we used a subset of the larger WLASL dataset, namely WLASL300, which comprises the top 300 classes with the most samples. We selected classes for the experiments based on the labels present in both the source datasets (LSA64 and Isolated Chinese Sign Language) and the target dataset (WLASL300). We identified 26 mutual labels for LSA and ASL datasets, and 23 mutual labels for CSL and ASL datasets. For the subset of the WLASL300 dataset used in this study, a subject only repeats a gesture on average 1.3 times compared to LSA64 and Chinese isolated SLR, where each subject repeats a gesture 5 times. This demonstrates the generalization ability of the WLASL dataset over other sign language datasets. However, we believe that this ability can cause underfitting, as the average number of samples per class in this dataset is less than 20. In comparison, LSA64 has 50 samples per class, and the Chinese isolated SLR has 250. Table 2: Training sample sizes and testing sample sizes video-level00footnotetext: DA stands for Domain Adaptation LSA CSL ASL for LSA to ASL DA ASL for CSL to ASL DA Sample Size 1040 4600 284 257 Testing Sample Size 260 1,150 71 64 Tables 3 and 4 show the number of learnable parameters in our Domain Adaptation models per N-multiscale TRN. Table 3: Complexity of the Domain Adaptation Models for LSA to ASL. The number of parameters for each N-multiscale TRN is indicated in the first row, while the number of multiscale TRN for each model is listed in the second row. of Parameters 2582844 3895616 5732676 9471306 18323796 -multiscale TRN 3 5 7 10 15 Table 4: Complexity of the Domain Adaptation Models for CSL to ASL. The number of parameters for each N-multiscale TRN is indicated in the first row, while the number of multiscale TRN for each model is listed in the second row. of Parameters 2580534 3893306 5730366 9468996 18321486 -multiscale TRN 3 5 7 10 15 All of our experiments are benchmarked with randomized 5-fold cross-validation, and training time is up to 100 epochs. Our training-to-testing sample size ratio is 80:20 for full-scale domain adaptation, and 20:80 for few-shot domain adaptation. We also have selected a batch size of 20. For both full-scale and few-shot domain adaptation, we have decided to compare the results from utilizing 3, 5, 7, 10, and 15 multiscale TRNs. Aside from calculating the mean accuracy from each 5-fold cross-validation, we also computed their standard deviations. 4.2 Experiment Results and Analysis For our experiment, we aim to compare the performance between (1) domain adaptation and neural network-based transfer learning, (2)full-scale and few-shot domain adaptation, and (3) N-different multi-scale TRNs. Our analysis will first highlight the best N-multiscale TRN for each group of the experiment: (1) full-scale domain adaptation between LSA to ASL, (2) full-scale domain adaptation between CSL to ASL, (3) few-shot domain adaptation between LSA to ASL, and (4) few-shot domain adaptation between CSL to ASL. Then, in Section 4.3, we will discuss how domain adaptation outperforms neural network-based transfer learning. We are not only considering improvements from the baseline or non-transfer learning base model in determining the best N-multiscale TRN. We are also identifying the N-multiscale TRN that produced the maximum accuracy for each group of experiments. This approach ensures that improvements over weak baselines or baselines with the lowest classification accuracy are not misleading. The best N-multiscale TRN should produce a domain-adapted model with the highest classification accuracy and a significant improvement over its baseline. Although the resulting accuracy from our experiments may appear low compared to other studies on Sign Language Recognition (SLR), our transfer learning task is particularly challenging. We experiment with both one-handed and two-handed dynamic word-level signs from the most diverse American Sign Language (ASL) dataset. This diversity includes not only signers or subjects but also video recording conditions such as aspect, lighting, and distance from the camera. These conditions persist because we collect WLASL samples from various sources, including YouTube and educational websites. To our knowledge, no cross-sign language transfer learning study has focused on two-handed dynamic signs. Table 11 summarizes the main differences between our target domain ASL and the sign language datasets used in other relevant SLR studies. One critical factor to note is that signers in the WLASL subset we used appeared almost only once per class, with each class containing around 14 video samples. This is in contrast to Bird et al.’s study, the most relevant literature to our work, where their target ASL dataset features only two signers for all classes. 4.2.1 Full-scale Domain Adaptation Results The bar charts in Figures 8 and 9 shows how the classification accuracies of the baseline ASL models and the domain-adapted ASL models change across different N-multiscale TRNs. Conversely, Table 5 shows all the results of conducting full-scale domain adaptation. Full-scale LSA to ASL domain adaptation The bar chart in Figure 8 shows that using the 5-multiscale TRN resulted in the highest classification accuracy of 12.93% for the domain adapted model in RGB mode, in the LSA to ASL Domain Adaptation group of experiments. This improvement represents a 59.62% increase over the baseline. Among all RGB domain adaptations for this group, this improvement is the highest, making it easy to conclude that the 5-multiscale TRN is the best N-multiscale TRN for RGB mode in this group. Figure 8: Bar chart showing the Full-scale Domain Adaptation experiment between LSA to ASL across different N-multiscale TRNs When analyzing the results of our LSA to ASL full-scale domain adaptation in Optical Flow mode, we found that the 3-multiscale TRN domain adaptation achieved a maximum classification accuracy of 10.08%, representing a 15.15% improvement from its baseline. On the other hand, the 7-multiscale TRN domain adaptation showed the highest improvement of approximately 92.017% from its corresponding baseline. In addition, the 7-multiscale TRN obtained the second-highest classification accuracy of 9.958%. Given these results, we conclude that the 7-multiscale TRN is the best N-multiscale TRN for LSA to ASL full-scale domain adaptation in Optical Flow mode. Full-scale CSL to ASL domain adaptation In the bar chart shown in Figure 9, it can be observed that the 7-multiscale TRN yielded the highest classification accuracies for CSL to ASL domain adaptation in both RGB (11.86%) and optical flow mode (10.08%). Compared to the corresponding RGB baseline, this represents an improvement of 24.19%, while for optical flow, the improvement is 10.31%. Figure 9: Bar chart showing the Full-scale Domain Adaptation experiment between CSL to ASL across different N-multiscale TRNs The 15-multiscale TRN helped the RGB baseline improve the most at 57.38% compared to other N-multiscale TRNs. However, the resulting domain-adapted model has the second-lowest accuracy among other domain-adapted models in RGB mode at 9.906%. Therefore, for RGB mode, 7-multiscale TRN is the best for this group of experiments. In contrast, for optical flow mode, the 7-multiscale TRN was not the most effective as the 3-multiscale TRN produced an improvement of nearly 20% from the baseline. Although the 7-multiscale TRN achieved the highest classification accuracy of 10.08%, the 3-multiscale TRN came in a close second with an accuracy of 10.01%. Therefore, for this set of experiments, the 3-multiscale TRN is the most effective in domain adaptation for optical flow mode. It is worth noting that a negative transfer is observed in the optical flow input mode under 15-multiscale TRN, where the domain-adapted model’s accuracy of 8.618% is lower than its baseline accuracy of 10.112%. Overall, shorter-term multiscale TRNs such as 3, 5, and 7, on average, outperform longer-term multiscale TRNs in terms of improvement over strong baselines in Full-scale Domain Adaptation. Table 5: Overall results of the Full-scale Domain Adaptation experiment. The second column of the table corresponds to the relevant subsets of ASL, which were selected based on their shared classes with LSA and CSL, as shown in Table 1. The third column specifies the type of domain adaptation that was performed where Non-transfer learning refers to the baseline.00footnotetext: Standard deviations correspond to the mean of the 5-fold cross-validation results of each N-multiscale TRN domain adaptation -multiscale TRN 3 5 7 10 15 ASL with LSA classes Non-transfer learning mean 8.32% 8.1% 9.364% 9.174% 8.308% std. 1.399 2.502 1.26 1.716 1.696 LSA to ASL mean 11.75% 12.93% 9.564% 9.318% 9.024% std. 1.501 1.821 1.91 2.583 2.571 ASL with CSL classes Non-transfer learning mean 9.524% 9.418% 9.55% 7.59% 6.294% std. 1.64 1.261 4.922 0.396 0.5 CSL to ASL mean 10.46% 10.55% 11.86% 8.56% 9.906% std. 0.602 3.321 1.772 3.00 2.688 Optical Flow ASL with LSA classes Non-transfer learning mean 8.754% 7.79% 5.186% 7.618% 7.16% std. 3.242 2.919 0.451 3.173 2.452 LSA to ASL mean 10.08% 9.412% 9.958% 9.756% 8.45% std. 1.33 2.45 1.99 2.129 1.123 ASL with CSL classes Non-transfer learning mean 8.342% 7.538% 9.138% 6.672% 10.112% std. 1.341 1.546 2.33 0.585 1.3 CSL to ASL mean 10.01% 9.076% 10.08% 6.6832% 8.618% std. 1.309 2.039 1.047 1.047 1.047 (l)1-9 4.2.2 Few-shot Domain Adaptation The bar charts in Figures 10 and 11 show how the classification accuracy changes between different N-multiscale TRNs in a few-shot learning setting. Conversely, Table 6 shows all the results of our few-scale domain adaptations. Few-shot LSA to ASL domain adaptation The bar chart from Figure 10 shows that for RGB input mode, 3-multiscale TRN achieved the maximum classification accuracy of 6.85%. Its corresponding baseline is the strongest among all the other comparable baselines, with a classification accuracy of 6.4%. The percentage difference between this baseline and the domain-adapted model is 7.03%. From this figure, it is not difficult to conclude that 3-multiscale TRN is the most effective N-multiscale TRN for LSA to ASL few-shot domain adaptation in RGB mode. Meanwhile, for the Optical Flow mode, 10-multiscale TRN produced the maximum classification accuracy of 7.34% for the domain-adapted model. This is a 26.48% increase from the corresponding baseline, which shows the highest improvement in the group. We consider 10-multiscale TRN as the best N-multiscale TRN for this domain adaptation, as it generated the highest classification accuracy as well as the highest improvement over its baseline in this group of experiments. Figure 10: Bar chart showing the Few-shot Domain Adaptation experiment between LSA to ASL across different N-multiscale TRNs Few-shot CSL to ASL domain adaptation Figure 11 shows that for CSL to ASL few-shot domain adaptation, the 7-multiscale TRN produced the maximum classification accuracies for all input modes (RGB and Optical Flow). For RGB, the maximum classification accuracy is 9.266% which is an improvement of 3.41% from its corresponding baseline, whose accuracy is 8.96%. For Optical Flow, the maximum classification accuracy achieved is 7.948%. This accuracy is a 4.55% increase from the baseline of 7.602%. Figure 11: Bar chart showing the Few-shot Domain Adaptation experiment between CSL to ASL across different N-multiscale TRNs The 5-multiscale TRN yielded the most improvement of approximately 19.32% from the RGB baseline. However, its corresponding domain-adapted model has the second-lowest classification accuracy among the group at 8.572%. Therefore, for RGB mode, the 7-multiscale TRN is considered best. For optical flow mode, we consider the 3-multiscale TRN to be the most effective. The reason is that its domain-adapted model has the second-highest classification accuracy in the group at 7.788% which translated to an improvement of 12.54%. Having the second-highest classification accuracy and the highest baseline improvement made 3-multiscale TRN the best for CSL to ASL few-shot domain adaptation. Similar to full-scale CSL to ASL Domain Adaptation (see Figure 9), the 15-multiscale TRN produced domain-adapted models with lower classification accuracy than its baseline (8.796% vs 8.95%). Also akin to full-scale domain adaptation, shorter-term multiscale TRNs such as 3, 5, and 7, on average, outperformed longer-term multiscale TRNs in terms of improvement over strong baselines in Few-scale Domain Adaptation as seen in Figures 9 and 8. Table 6: Overall results of the Few-shot Domain Adaptation experiment. The second column of the table corresponds to the relevant subsets of ASL, which were selected based on their shared classes with LSA and CSL, as shown in Table 1. The third column specifies the type of domain adaptation that was performed where Non-transfer learning refers to the baseline.00footnotetext: Standard deviations correspond to the mean of the 5-fold cross-validation results of each N-multiscale TRN domain adaptation -multiscale TRN 3 5 7 10 15 ASL with LSA classes Non-transfer learning mean 6.4% 6.386% 6.078% 5.88% 6.32% std. 0.998 0.929 0.77 0.543 0.48 LSA to ASL mean 6.85% 6.764% 6.24% 6.022% 6.702% std. 0.821 0.686 0.537 0.588 0.267 ASL with CSL classes Non-transfer learning mean 7.816% 7.184% 8.96% 8.792% 8.958% std. 1.54 1.3 2.234 2.099 1.998 CSL to ASL mean 8.056% 8.572% 9.266% 8.842% 8.796% std. 0.57 1.245 0.881 0.792 1.584 Optical Flow ASL with LSA classes Non-transfer learning mean 6.268% 6.58% 7.108% 5.806% 5.714% std. 1.132 1.166 0.884 0.906 1.122 LSA to ASL mean 7.184% 6.654% 7.262% 7.344% 6.738% std. 0.288 1.357 0.236 1.1 0.788 ASL with CSL classes Non-transfer learning mean 6.92% 7.364% 7.602% 7.248% 7.004% std. 2.688 0.018 1.437 1.999 0.872 CSL to ASL mean 7.788% 7.684% 7.948% 7.536% 7.34% std. 0.267 0.548 0.601 0.619 0.257 (l)1-9 Conducting full-scale learning has resulted in domain-adapted models with higher accuracy. Moreover, based on the results of these experiments, it can be concluded that RGB is still superior to optical flow in terms of producing better domain-adapted models. Although LSA and CSL are not related sign languages to ASL, it can be observed that LSA and ASL in general share more similarities in gestures than CSL and ASL. For instance, we discovered that signs such as “Deaf”, “Call”, “Candy”, “Catch”, and “Copy” in our LSA and ASL subsets share notable similarities. Therefore, it is not surprising that LSA as a source helped ASL to reach the highest accuracy in all domain adaptation experiments (see Table 5). However, LSA underperformed in our few-shot domain adaptation experiments. The reason is that, since we are only using 20% of the ASL samples for target training in few-shot learning settings, LSA, being a smaller source domain than CSL, was unable to provide sufficient representations for ASL to learn. 4.3 Pre-training As part of our study, we also experimented with the pre-training approach wherein the weights learned by a network from the source domain are used as initial weights for training the target domain. This transfer learning method is used by Bird et al. 2, the first paper to study transfer learning between two sign languages, and other studies that use transfer learning from large-scale datasets such as ImageNet to improve their SLR models 14; 15; 18; 7. To apply this transfer learning technique in our task, we needed to add a Long short-term memory layer on top of the fully-connected layer of the convolutional neural network (CNN). For this method, we utilized a Resnet-50 architecture 8. We have decided to use this architecture as it is the most popular deep residual network. Similar to our domain adaptation experiment detailed in Section 4, all of our pre-training experiments are benchmarked with randomized 5-fold cross-validation, and training and testing are continued until there is no more improvement in accuracy after 100 epochs. We also have selected a batch size of 2. For full-scale learning settings, the training-to-test sample size ratio is 80:20 while for few-shot learning settings, the sample size ratio is 20:80. Table 7: 5-fold classification accuracies of RGB to RGB and optical flow to optical flow of ASL pre-trained with LSA and CSL as well as the ASL baseline in optical flow (OF) and RGB modes in the full-scale learning setting. ASL Baseline for LSA to ASL LSA to ASL ASL Baseline for CSL to ASL CSL to ASL mean 6.926% 10.676% 9.832% 9.518% std. 2.129 7.209 4.645 4.807 OF mean 10.344% 6.094% 15.998% 11.748% std. 7.409 1.601 8.425 7.489 (l)1-6 Figure 12: Bar chart showing the accuracies of ASL when pre-trained with LSA and CSL in optical flow (OF) and RGB modes in the full-scale learning setting. Baseline results are also indicated From Table 7 and Figure 12, we can observe that only ASL pre-trained with LSA in RGB mode produced a positive transfer outcome in full-scale learning settings. On the other hand, for few-shot learning settings, only ASL pre-trained with CSL in optical flow mode resulted in positive transfer as seen in Table 8 and Figure 13. Table 8: 5-fold classification accuracies of RGB to RGB and optical flow to optical flow of ASL pre-trained with LSA and CSL as well as the ASL baseline in optical flow (OF) and RGB modes in the few-shot learning setting. ASL Baseline for LSA to ASL LSA to ASL ASL Baseline for CSL to ASL CSL to ASL mean 6.686% 6.602% 7.818% 7.59% std. 1.422 1.257 2.201 2.322 OF mean 6.84% 6.086% 7.198% 7.504% std. 1.826 1.609 1.828 2.615 (l)1-6 Figure 13: Bar chart showing the accuracies of ASL when pre-trained with LSA and CSL in optical flow (OF) and RGB modes in the few-shot learning setting. Baseline results are also indicated Table 9: Performance matrix of Full-scale Neural network-based Transfer Learning and Domain Adaptation00footnotetext: N: Neural Network Accuracies (LSA to ASL) RGB 6.926 8.32 8.1 9.364 9.174 8.308 LSA to ASL RGB 10.676 11.75 12.93 9.564 9.318 9.024 Baseline(CSL to ASL) RGB 9.832 9.524 9.418 9.55 7.59 6.294 CSL to ASL RGB 9.518 10.46 10.55 11.86 8.56 9.906 Baseline(LSA to ASL) OF 10.344 8.754 7.79 5.186 7.618 7.16 LSA to ASL OF 6.094 10.08 9.412 9.958 9.756 8.45 Baseline(CSL to ASL) OF 15.998 8.342 7.538 9.138 6.672 10.112 CSL to ASL OF 11.748 10.01 9.076 10.08 6.6832 8.618 N DA 3-TRN DA 5-TRN DA 7-TRN DA 10-TRN DA 15-TRN Table 10: Performance matrix of Few-shot Neural network-based Transfer Learning and Domain Adaptation00footnotetext: N: Neural Network Accuracies (LSA to ASL) RGB 6.686 6.4 6.386 6.078 5.88 6.32 LSA to ASL RGB 6.602 6.85 6.764 6.24 6.022 6.702 Baseline(CSL to ASL) RGB 7.818 7.816 7.184 8.96 8.792 8.958 CSL to ASL RGB 7.59 8.056 8.572 9.266 8.842 8.796 Baseline(LSA to ASL) OF 6.84 6.268 6.58 7.108 5.806 5.714 LSA to ASL OF 6.086 7.184 6.654 7.262 7.344 6.738 Baseline(CSL to ASL) OF 7.198 6.92 7.364 7.602 7.248 7.004 CSL to ASL OF 7.504 7.788 7.684 7.948 7.536 7.34 N DA 3-TRN DA 5-TRN DA 7-TRN DA 10-TRN DA 15-TRN Pre-training in both full-scale and few-shot learning settings has resulted in more negative transfers or lower accuracy than positive transfers compared to their respective baselines. In contrast, for domain adaptation, only two instances of 15-multiscale TRN resulted in negative adaptations, while 18 out of 20 N-multiscale TRNs showed improvements in their baselines. Thus, for pre-training, improvements occurred only 25% of the time, compared to 90% for domain adaptation. The standard deviations in this experiment are consistent with those observed in the domain adaptation experiments presented in Section 4.2, where the test results in the few-shot learning setting were closer to the mean than those in the full-scale learning setting. Table 11: Comparison of our model to other related Sign Language Recognition models. 00footnotetext: The number of signers for our study refers to the average number of signers for each class of our subsets. The number of times a signer repeats a sign refers to the number of times a participant recorded a particular sign. In the classes column, we wrote 23 and 26 as we have two sets of classes, one for LSA to ASL transfer and one for CSL to ASL transfer. The average number of samples for each class of our subsets is 14. WLASL300, the dataset we used for our ASL domain, consists of 109 signers and a mean of 17 samples. However, by the time of this study, several of WLASL300’s source links from YouTube and other educational sign language websites are no longer available. Source Domain Target Domain Level Dynamic Two-handed Signers Number of times a subject repeat a sign Classes Accuracy (Highest Score Achieved) et al.2 BSL ASL Word Yes No 2 2 18 94.44% Halvardsson et al.7 ImageNet SSL Alphabet No No 5 8 26 85% Cayamcela et al.14 ImageNet ASL Alphabet No No Implied as many. Exact number not indicated Not indicated 26 99.39% Nishat et al.15 ImageNet BSDL digits, vowel, consonants No Yes Implied as many. Exact number not indicated Not indicated 46 96.57% Farhadi et al.5 ASL Avatar ASL Word Yes Yes 1 3 50 64.17% Vazquez-Enriquez et al.28 ASL Turkish and Spanish Sign Language Word Yes Yes 43 Not indicated 226 95.24% (Turkish), 93.91% (Spanish) Zakariah et al. 31 Imagenet Arabic Sign Language Alphabet No No 40 Not indicated 32 95% Shania et al. 22 Imagenet Indonesian Sign Language Word Yes Yes ¿2 Not Indicated 11 98.5% Abdullayeva et al. 1 Imagenet, ASL letters Azerbaijan Sign Language Alphabet Yes No Not indicated Not indicated 32 88% Thakar et al. 27 Imagenet ASL Alphabet No No Not indicated Not indicated 29 98.7% Das et al. 4 Imagenet BsDL Digit and characters No Yes Not indicated Not indicated 36 91.67% (character), 97.33% (digit) Jiang et al. 11 Imagenet CSL Finger Signs (A to Z, ZH, CH, SH, and NG) No No Not indicated Not indicated 30 91.48% Sharma et al. 23 Imagenet Indian Sign Language Alphabet and digits No Not indicated Not indicated Not indicated 35 100% Suharjito et al. 25 Imagenet and Kinetic Indonesian Signal System Not indicated Yes Not indicated 2 10 10 97.50% study LSA, CSL ASL Word Yes Yes 109 for WLASL300. For our subsets, on average 11 signers for 14 samples Average 1.3 23 12.93% (LSA), 11.86% (CSL) 5 Conclusion Our research on transfer learning approaches between different sign languages provided insights on how to improve classification and recognition in low-resource conditions by aligning the source sign language’s spatiotemporal features with the target sign language’s in a multiple-timescale fashion. From our best knowledge, we are the first to study the knowledge adaptability between LSA (Argentine Sign Language) to ASL (American Sign Language) and CSL (Chinese Sign Language) to ASL in both RGB and optical flow modes. We are also the first to conduct a few-shot domain adaptation between sign languages. Applying few-shot learning in SLR is important to advance the field and place it at the levels of other fields such as computational linguistics, natural language processing, and other action recognition areas where many studies are being conducted on the application of few-shot learning. The empirical evidence resulting from our study showed that multi-scale temporal Domain Adaptation between LSA and ASL, and CSL and ASL performed significantly better than pre-training ASL with CSL and LSA. The pre-training approach yielded more negative transfers than positive ones. We were also able to prove our hypothesis that aligning shorter-term multiscale TRNs such as 3, 5, or 7 would be more beneficial for the domain adaptation between these sign languages than aligning longer-term multi-scale TRNs such as 10 or 15. This hypothesis holds for all of our experiment settings, such as full-scale and few-shot domain adaptation, and RGB and optical flow input modes (see Section 4.2). This is a piece of great news for the fields of SLR and action recognition in general as training longer-term multi-scale TRNs consume more GPU resources and time to complete. Moreover, we believe that longer-term multi-scale TRNs will only outperform shorter-term ones in cases where the source and target domains bear more similarities than differences. Our study also proved our main hypothesis that adapting the domain of one sign language to another - even if they are not from the same sign language family would make a better SLR model than not doing any domain adaptation at all or only pre-training the target domain with another sign language or with a generic large-scale dataset such as ImageNet. Although our experiment showed that RGB performed better than optical flow (see Section 4.2), we believe that our study opened more possibilities for the application of optical flow in the field of SLR. In the future, we plan to conduct studies that aim to discover more potential in this type of motion representation. Lastly, we envision that our study will be a valuable resource in the real-time translation of sign gestures into words or phrases, or in transcribing videos that feature sign languages. However, to effectively model sign languages, a language modeling technique must be utilized due to differences in grammar structure between spoken languages and sign languages. Nonetheless, these grammar differences can be addressed through domain adaptation. One example is adapting from a sentence-level sign language dataset with spoken language transcription and gloss-level annotation to a dataset with only gloss annotation, to automatically convert gloss-level annotations into transcriptions. Finding similar gloss annotations across datasets is a significant challenge. Alternatively, adapting a non-sign language video with transcriptions to sentence-level sign language videos with gloss-level annotations may be explored in future work. Declarations • Conflict of interest/Competing interests The authors have no relevant financial or non-financial interests to disclose. • Funding This research was funded by the Shenzhen Science and Technology Innovation Commission (JCYJ20210324135011030), Science and Technology Innovation Committee of Shenzhen-Platform and Carrier (International Science and Technology Information Center), High-end Foreign Expert Talent Introduction Plan (G2021032022L), Guangdong Pearl River Plan (2019QN01X890), and National Natural Science Foundation of China (Grant No. 71971127). • Ethics approval Not applicable • Consent to participate Not applicable • Consent for publication Not applicable • Availability of data and materials All data generated or analyzed during this study are included in these published articles 19; 32; 16; 9; 13 (and its supplementary information files). The subsets we used are detailed in Section 4.1. For additional guidance on extracting the subsets from their originating datasets, please contact the authors. • Code availability The codes used for domain adaptation are based on TA3N 3. Our modification includes setting the batch size to 20, the mode of learning to supervised learning, and the value of num_segments to the N-multiscale TRN. The codes for converting videos into RGB and Optical Flow frames are available from this repository, https://doi.org/10.6084/m9.figshare.20223444 . For additional guidance, please contact the authors. All authors contributed to the study’s conception and design. Material preparation, data collection, and analysis were performed by Keren Artiaga, Yang Li, Ercan Engin Kuruoglu, and Wai Kin (Victor) Chan. The first draft of the manuscript was written by Keren Artiaga and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript. References Abdullayeva and Alishzade (2022) G.G. Abdullayeva and N.O. Alishzade Transfer learning for azerbaijani sign language recognition. Informatics and Control Problems. Cited by: §2.1, Table 11. Bird et al. (2020) J. J. Bird, A. Ekárt, and D. R. Faria British sign language recognition via late fusion of computer vision and leap motion with transfer learning to american sign language. Sensors 20. Cited by: §1, §1, §2.1, §2.1, §4.3, Table 11. Chen et al. (2019) M.H. Chen, Z. Kira, G. Al-Regib, J. Yoo, R. Chen, and J. Zheng Temporal attentive alignment for large-scale video domain adaptation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), p. 6320–6329. Cited by: §2.2, §3.2.1, 7th item. Das et al. (2022) S. Das, Md. S. Imtiaz, N. Neom, N. Siddique, and H. Wang A hybrid approach for bangla sign language recognition using deep transfer learning model with random forest classifier. Expert Syst. Appl. 213, p. 118914. Cited by: §2.1, §2.1, §2.1, Table 11. Farhadi et al. (2007) A. Farhadi, D. Forsyth, and R. White Transfer learning in sign language. 2007 IEEE Conference on Computer Vision and Pattern Recognition, p. 1–8. External Links: Document Cited by: §2.1, Table 11. Farnebäck (2003) G. Farnebäck Two-frame motion estimation based on polynomial expansion. In SCIA, p. 363–370. Cited by: §1. Halvardsson et al. (2021) G. Halvardsson, J. Peterson, C. Soto-Valero, and B. Baudry Interpretation of swedish sign language using convolutional neural networks and transfer learning. p. 207. External Links: Link Cited by: §2.1, §2.1, §4.3, Table 11. He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 770–778. Cited by: §4.3. Huang et al. (2018) J. Huang, W. Zhou, Q. Zhang, H. Li, and W. Li Video-based sign language recognition without temporal segmentation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18, p. 2257–2264. External Links: ISBN 978-1-57735-800-8 Cited by: §1, §4.1, 6th item. Jamal et al. (2018) A. Jamal, V.P. Namboodiri, D. Deodhare, and K. Venkatesh Deep domain adaptation in action space. In BMVC, Cited by: §2.2. Jiang et al. (2020) X. Jiang, B. Hu, S. C. Satapathy, S. Wang, and Y. Zhang Fingerspelling identification for chinese sign language via alexnet-based transfer learning and adam optimizer. Sci. Program. 2020, p. 3291426:1–3291426:13. Cited by: §2.1, Table 11. Kuehne et al. (2011) H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre HMDB: a large video database for human motion recognition. In 2011 International Conference on Computer Vision, Vol. , p. 2556–2563. External Links: Document Cited by: §4. Li et al. (2020) D. Li, C. Rodriguez, X. Yu, and H. Li Word-level deep sign language recognition from video: a new large-scale dataset and methods comparison. In The IEEE Winter Conference on Applications of Computer Vision, p. 1459–1469. Cited by: §1, §4.1, 6th item. Morocho-Cayamcela and Lim (2019) M. E. Morocho-Cayamcela and W. Lim Fine-tuning a pre-trained convolutional neural network model to translate american sign language in real-time. 2019 International Conference on Computing, Networking and Communications (ICNC), p. 100–104. Cited by: §1, §2.1, §2.1, §4.3, Table 11. Nishat and Shopon (2020) Z. K. Nishat and Md. Shopon Unsupervised pretraining and transfer learning-based bangla sign language recognition. In Proceedings of International Joint Conference on Computational Intelligence Algorithms for Intelligent Systems, p. 529–540. External Links: Document Cited by: §1, §2.1, §2.1, §2.1, §4.3, Table 11. Pu et al. (2016) J. Pu, W. Zhou, and H. Li Sign language recognition with multi-modal features. In PCM, p. 252–261. Cited by: §1, §4.1, 6th item. Rahman et al. (2021) M. M. Rahman, R. Mdrafi, A. C. Gurbuz, E. Malaia, C. Crawford, D. Griffin, and S. Z. Gurbuz Word-level sign language recognition using linguistic adaptation of 77 ghz fmcw radar data. In 2021 IEEE Radar Conference (RadarConf21), Vol. , p. 1–6. External Links: Document Cited by: §1. Rathi (2018) D. Rathi Optimization of transfer learning for sign language recognition targeting mobile platform. In International Journal on Recent and Innovation Trends in Computing and Communication, Vol. 6, p. 198–203. Cited by: §1, §2.1, §4.3. Ronchetti et al. (2016) F. Ronchetti, F. Quiroga, C. Estrebou, L. Lanzarini, and A. Rosete LSA64: a dataset of argentinian sign language. X I Congreso Argentino de Ciencias de la Computación (CACIC), p. 794–803. Cited by: §1, §4.1, 6th item. Sahoo et al. (2021) A. Sahoo, R. Shah, R. Panda, K. Saenko, and A. Das Contrast and mix: temporal contrastive video domain adaptation with background mixing. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. W. Vaughan (Eds.), Vol. 34, p. 23386–23400. Cited by: §2.2. Sevilla-Lara et al. (2018) L. Sevilla-Lara, Y. Liao, F. Güney, V. Jampani, A. Geiger, and M. J. Black On the integration of optical flow and action recognition. In GCPR, p. 281–297. Cited by: §1. Shania et al. (2022) S. Shania, M. F. Naufal, V. R. Prasetyo, and M. S. B. Azmi Translator of indonesian sign language video using convolutional neural network with transfer learning. Indonesian Journal of Information Systems. Cited by: §2.1, Table 11. Sharma et al. (2021) C. M. Sharma, K. Tomar, R. K. Mishra, and V. M. Chariar Indian sign language recognition using fine-tuned deep transfer learning model. SSRN Electronic Journal. Cited by: §2.1, Table 11. Soomro et al. (2012) K. Soomro, A.R. Zamir, and M. Shah UCF101: a dataset of 101 human actions classes from videos in the wild. External Links: 1212.0402 Cited by: §4. Suharjito et al. (2021) Suharjito, N. Thiracitta, and H. Gunawan SIBI sign language recognition using convolutional neural network combined with transfer learning and non-trainable parameters. Procedia Computer Science 179, p. 72–80. Cited by: §2.1, Table 11. Sultani and Saleemi (2014) W. Sultani and I. Saleemi Human action recognition across datasets by foreground-weighted histogram decomposition. 2014 IEEE Conference on Computer Vision and Pattern Recognition, p. 764–771. Cited by: §2.2. Thakar et al. (2022) S. Thakar, S. Shah, B. Shah, and A. V. Nimkar Sign language to text conversion in real time using transfer learning. 2022 IEEE 3rd Global Conference for Advancement in Technology (GCAT), p. 1–5. Cited by: §2.1, Table 11. Vázquez-Enríquez et al. (2021) M. Vázquez-Enríquez, J. L. Alba-Castro, L. Docío-Fernández, and E. Rodríguez-Banga Isolated sign language recognition with multi-scale spatial-temporal graph convolutional networks. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Vol. , p. 3457–3466. External Links: Document Cited by: Table 11. Wang et al. (2020) Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni Generalizing from a few examples. ACM Computing Surveys (CSUR) 53, p. 1 – 34. Cited by: §3.3. Xu et al. (2016) T. Xu, F. Zhu, E. Wong, and Y. Fang Dual many-to-one-encoder-based transfer learning for cross-dataset human action recognition. Image Vis. Comput. 55, p. 127–137. Cited by: §2.2. Zakariah et al. (2022) M. Zakariah, Y. A. Alotaibi, D. Koundal, Y. Guo, and M. M. Elahi Sign language recognition for arabic alphabets using transfer learning technique. Computational Intelligence and Neuroscience 2022. Cited by: §2.1, Table 11. Zhang et al. (2016) J. Zhang, W. Zhou, C. Xie, J. Pu, and H. Li Chinese sign language recognition with adaptive hmm. In 2016 IEEE International Conference on Multimedia and Expo (ICME), Vol. , p. 1–6. External Links: Document Cited by: §1, §4.1, 6th item. Zhou et al. (2018) B. Zhou, A. Andonian, A. Oliva, and A. Torralba Temporal relational reasoning in videos. European Conference on Computer Vision, p. 831–846. Cited by: §2.2, §3.2.1, §3.2.3.