Paper deep dive
OneSign: Unifying Sign Language Understanding Tasks with One Model
Shiwei Gan, Yafeng Yin, Xiao Liu, Desibieer Tuerdaken, Lei Xie, Sanglu Lu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:SLU encompasses a diverse set of tasks, including ISLR, CSLR, and SLT. Although these tasks share basic semantic and linguistic foundations, they are typically addressed with task-specific architectures and training pipelines, which hinders knowledge sharing and requires costly pretraining and finetuning for each task. In this paper, we focus on two aspects of SLU tasks: (1) training and inference pipelines are highly fragmented: most methods rely on pretraining on large-scale SL datasets followed by task- or dataset-specific finetuning, which leads to multiple specialized models rather than a single checkpoint. (2) current LLM-based methods may overlook the inherent modality discrepancy between sign and text tokens, simply concatenating them and processing both modalities with the same decoder layers. In this paper, we present OneSign, a unified framework that addresses multiple SLU tasks within a single model and a single checkpoint. OneSign reformulates ISLR, CSLR, and SLT under a single training paradigm. To accommodate the heterogeneous characteristics of sign and text representations, we introduce a Modality-Adaptive Mixture-of-Experts (MA-MoE) architecture, consisting of a shared expert and modality-specific experts for sign and text tokens. A modality router dynamically activates the corresponding experts, and their outputs are aggregated to form the final token representations. By enabling modality-dependent expert specialization while preserving a shared expert path, MA-MoE can effectively model the modality differences between continuous sign representations and discrete text tokens. Extensive experiments on multiple benchmarks demonstrate that OneSign achieves competitive or state-of-the-art performance on several benchmarks, highlighting its effectiveness as a unified SLU model. Datasets are available at : this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.33090v1
- Canonical: https://arxiv.org/abs/2609.33090v1
Trouble viewing inline? Open PDF directly →
Full Text
87,139 characters extracted from source content.
Expand or collapse full text
OneSign: Unifying Sign Language Understanding Tasks with One Model Shiwei Gan Affiliation: State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China Yafeng Yin Affiliation: State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China Correspondence to: yafeng@nju.edu.cn Xiao Liu Affiliation: State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China Desibieer Tuerdaken Affiliation: State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China Lei Xie Affiliation: State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China Sanglu Lu Affiliation: State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China Abstract Sign Language Understanding (SLU) encompasses a diverse set of tasks, including Isolated Sign Language Recognition (ISLR), Continuous Sign Language Recognition (CSLR), and Sign Language Translation (SLT). Although these tasks share basic semantic and linguistic foundations, they are typically addressed with task-specific architectures and training pipelines, which hinders knowledge sharing and requires costly pretraining and finetuning for each task. In this paper, we focus on two aspects of SLU tasks: (1) training and inference pipelines are highly fragmented: most methods rely on pretraining on large-scale SL datasets followed by task- or dataset-specific finetuning, which leads to multiple specialized models rather than a single checkpoint. (2) current LLM-based methods may overlook the inherent modality discrepancy between sign and text tokens, simply concatenating them and processing both modalities with the same decoder layers. In this paper, we present OneSign, a unified framework that addresses multiple SLU tasks within a single model and a single checkpoint. OneSign reformulates ISLR, CSLR, and SLT under a single training paradigm, allowing all tasks to be trained and inferred using the same model checkpoint without dataset or task-specific pretraining or retraining. To accommodate the heterogeneous characteristics of sign and text representations, we introduce a Modality-Adaptive Mixture-of-Experts(MA-MoE) architecture, consisting of a shared expert and modality-specific experts for sign and text tokens. A modality router dynamically activates the corresponding experts, and their outputs are aggregated to form the final token representations. By enabling modality-dependent expert specialization while preserving a shared expert path, MA-MoE can effectively model the modality differences between continuous sign representations and discrete text tokens. Extensive experiments on multiple benchmarks demonstrate that OneSign achieves competitive or state-of-the-art performance on several benchmarks, highlighting its effectiveness as a unified SLU model. Datasets are available at : https://github.com/gswycf/OneSign. Figure 1: Overall performance comparison across multiple tasks. Keywords: Sign Language 1 Introduction Sign Language Understanding (SLU) encompasses a family of closely related tasks (Gan et al., 2024; Gan et al., 2023b; Gan et al., 2023a; Gan et al., 2021; Liu et al., 2026a; Gan et al., 2026d; Gan et al., 2026a; Liu et al., 2026b; Gan et al., 2026b; Gan et al., 2026c), including Isolated Sign Language Recognition (ISLR) (Albanie et al., 2021), Continuous Sign Language Recognition (CSLR) (Koller et al., 2019), and Sign Language Translation (SLT) (Gan et al., 2021; liu202). ISLR focuses on recognizing a single sign instance from a short, trimmed sign clip, typically formulated as a classification problem. CSLR aims to recognize unsegmented sign videos and output a sequence of gloss labels, addressing the challenge of temporal alignment between sign sequence and gloss sequence. In contrast, SLT (Sincan et al., 2023) goes beyond recognition by translating sign language (SL) sequences directly into spoken language sentences. Despite strong semantic connections of these tasks, existing SLU methods are predominantly developed in a task-specific manner, where each task is addressed using an independent model architecture and training pipeline. For ISLR, models usually adopt a visual backbone to capture spatiotemporal features, followed by a classification head for sign prediction. For CSLR, a typical pipeline adopts a visual backbone for feature extraction (Zhou et al., 2021b; Gan et al., 2026a; Gan et al., 2023a; Gan et al., 2026d), together with temporal modeling modules, and leverages CTC-based objectives to handle the latent alignment between the sign videos and the gloss labels, yielding the target gloss sequence. A similar task-specific design paradigm also dominates current SLT models. Most SLT approaches first employ a sign language backbone to extract visual representations (Zhou et al., 2021b; Gan et al., 2026b; Gan et al., 2023a; Gan et al., 2026d), followed by temporal encoding modules. The resulting sign features are then concatenated with textual embeddings and fed into a translation model and processed with the same transformer layer (e.g., GPT-2 (Wong et al., 2024), Gemma (Guo et al., 2025)), to generate the corresponding spoken language sentence. Figure 2: Comparison between previous SLU pipelines and the proposed OneSign framework. Recently, several unified SL models (e.g., MSLU (Zhou et al., 2025), UniSign (Li et al., 2025) and ShuBERT (Gueuwou et al., 2025a)) have shown promising progress toward addressing multiple SLU tasks within a pretraining framework. These approaches typically rely on large-scale SL datasets for pretraining, followed by task- or dataset-specific finetuning to obtain dataset-dependent model checkpoints, achieving notable performance improvements. Despite their success, existing unified SLU models still exhibit two key limitations. First, they depend on large annotated SL datasets for pretraining, which are expensive to construct. However, models still need to be finetuned separately for each downstream dataset, limiting training efficiency and model reusability. Second, many recent methods adopt decoder-only architectures, where large language models (LLMs) (e.g., GPT-2 or Gemma) are used as translation modules. In these frameworks, sign tokens and text tokens are typically concatenated into a single sequence and processed by the same Transformer decoder layers, implicitly treating sign and text representations as belonging to the same modality. However, due to the inherent distributional differences between continuous sign representations and discrete textual tokens, this uniform modeling strategy overlooks the modality gap, which can hinder effective cross-modal reasoning and representation learning. Table 1: Comparison with existing SLT datasets. Vocab: vocabulary size; Hours: total video duration in hours; Topic: main domain of videos (e.g., daily life, TV news). Name Language Vocab. Hours Source Topic KETI (Ko et al., 2019) KVK 419 28 Lab Emergency Situations VRT-RAW (Camgöz et al., 2021) VGT - 100 TV Broadcast SWISSTXT (Camgöz et al., 2021) DGS - 88 TV News PHOENIX-2014T (Camgoz et al., 2018) DGS 3K 11 TV Weather DGS Corpus (Hanke et al., 2020) DGS - 50 Lab Dialogue BOBSL (Albanie et al., 2021) BSL 77K 1,447 TV Broadcast How2Sign (Duarte et al., 2021) ASL 16K 79 Lab Daily OpenASL (Shi et al., 2022) ASL 33K 288 Web Multi-Topic YouTube-ASL (Uthus et al., 2023) ASL 60K 984 Web Multi-Topic SP-10 (Yin et al., 2022) Various 17K 14 Web Multilingual AfriSign (Gueuwou et al., 2023) Various 20K 152 Web Multilingual CSL-Daily (Zhou et al., 2021a) CSL 2K 23 Lab Daily CSL-News (Li et al., 2025) CSL 5K 1,985 TV News CSL-OpenWorld (Ours) CSL 12K 216 Web Multi-Topic In this paper, we focus on core Sign Language Understanding (SLU) tasks, including ISLR, CSLR, and SLT, and propose OneSign, a unified framework that aims to solve heterogeneous SLU tasks within a single model and a single checkpoint. Unlike existing frameworks that follow a rigid pretraining–finetuning paradigm, where models are first pretrained on large-scale annotated datasets or carefully designed pretext tasks, and then separately finetuned on each downstream task. OneSign reformulates SLU training under a unified paradigm (See Figure 2). This enables heterogeneous SLU datasets, including ISLR, CSLR, and SLT datasets, to be jointly trained within the same framework, without task-specific pretraining or retraining. Furthermore, we observe that directly modeling sign tokens and text tokens within a standard LLM introduces significant challenges due to the inherent distributional gap between continuous sign representations and discrete textual tokens. To address this issue, we propose a Modality-Adaptive Mixture-of-Experts (MA-MoE) architecture, which explicitly accounts for modality heterogeneity while preserving shared semantic modeling. MA-MoE consists of an always-activated shared expert that captures modality-invariant sign–text semantics, together with modality-specific experts specialized for sign and text representations, respectively. Through unified training and modality-adaptive modeling, OneSign effectively models the difference between heterogeneous sign and text modalities, enabling efficient multi-task learning and inference across diverse SLU tasks using a single model checkpoint (Figure. 1). The main contributions are as follows: • We construct CSL-OpenWorld, a large-scale Chinese SL (CSL) dataset consisting of over 100k video samples with manually curated annotations collected from diverse web sources across multiple domains (e.g., news, law, and culture), substantially expanding existing CSL datasets. The dataset will be publicly released via HuggingFace. • We introduce a Modality-Adaptive Mixture-of-Experts (MA-MoE) architecture for LLM-based SLU modeling, which combines an always-activated shared expert with modality-specific experts to explicitly model the distributional difference between continuous sign representations and discrete text tokens. • We propose OneSign, the first unified framework that reformulates ISLR, CSLR, and SLT under one learning paradigm, enabling unified training and inference across heterogeneous SLU tasks with a single model checkpoint, without task-specific retraining. Extensive experiments on multiple benchmarks demonstrate the effectiveness of OneSign across multiple SLU tasks. 2 Related Work ISLR. SL Recognition (SLR) aims to recognize linguistic units from sign inputs and contains two tasks: ISLR and CSLR. Although both tasks focus on understanding sign semantics from videos, they differ primarily in supervision granularity and output structure, and are therefore typically treated as distinct problems. ISLR (Hu et al., 2021; Li et al., 2020b; Zuo et al., 2023) considers short, trimmed video clips containing a single sign and formulates recognition as a classification task. The standard ISLR paradigm follows a task-specific pipeline, where a visual backbone extracts spatiotemporal features that are directly mapped to predefined sign categories via a classification layer. CSLR. In contrast, CSLR (Zhang et al., 2023; Gan et al., 2024; Wei and Chen, 2023; Hu et al., 2023c; Jiao et al., 2023) addresses the recognition of unsegmented sign sequences from continuous video streams, introducing the challenge of implicitly aligning visual inputs with gloss sequences. As a result, CSLR is commonly formulated as a sequence recognition problem with weak temporal supervision, where the input sign sequences and gloss sequences share the same order but differ in length. To handle this alignment, the majority of CSLR methods adopt Connectionist Temporal Classification (CTC)-based frameworks. These approaches typically employ a vision backbone to extract sign features, followed by temporal modules to capture long-range dependencies, and a CTC-based decoder to align visual sequences with gloss sequences. SLT. Early SLT approaches are closely tied to advances in CSLR (Wei and Chen, 2023; Zuo and Mak, 2022; Gan et al., 2024; Niu and Mak, 2020; Zhou et al., 2020) . The prevailing paradigm in gloss-based SLT (GBSLT) (Gan et al., 2023b; Yin et al., 2021; Kan et al., 2022; Jin and Zhao, 2021; Orbay and Akarun, 2020; Camgoz et al., 2020) relies on a pretrained CSLR backbone to provide intermediate gloss-aligned representations, which are subsequently fed into translation modules. In this setting, CSLR effectively serves as a pretext task that functions as a “sign tokenizer,” aligning visual features with gloss sequences and enabling downstream translation. Further, increasing attention has been directed toward Gloss-Free SLT (GFSLT), which aims to directly translate sign videos into spoken language without intermediate gloss supervision. A fundamental challenge in GFSLT lies in how to tokenize sign sequence into discriminative sign features in the absence of gloss guidance. To address this issue, existing GFSLT methods typically rely on text annotations during pretraining, adopting strategies such as contrastive learning (Zhou et al., 2023; Jiao et al., 2024; Ye et al., 2024; Liang et al., 2024), CTC-based pseudo-gloss objectives (Gan et al., 2026d), text-aligned vector quantization (Gong et al., 2024), and multi-stage or curriculum-based pretraining schemes (Guo et al., 2025; Gueuwou et al., 2025a; Gueuwou et al., 2025b). These approaches aim to compensate for the lack of explicit gloss supervision by leveraging auxiliary alignment signals from textual modalities. Existing SLR and SLT methods, though effective, are largely task-specific, with models and training objectives differing across ISLR, CSLR, and SLT. This fragmentation limits representation sharing and cross-task knowledge transfer, motivating the need for a unified, task-agnostic framework. Recent works, such as UniSign (Li et al., 2025), MSLU (Zhou et al., 2025), and ShuBERT(Gueuwou et al., 2025a), attempt multi-task learning by jointly pretraining their visual backbones or translation modules on large SL datasets and fine-tuning on each specific dataset. However, these approaches still rely on task- or dataset-specific fine-tuning. Mixture of Experts (MOE). Compared with standard LLMs that process all tokens using shared parameters, MoE (Fedus et al., 2022; Jiang et al., 2024; Dai et al., 2024) introduces conditional computation by routing each input token to a subset of experts. This design enables a better trade-off between model capacity and computational efficiency, and has been shown to be particularly effective in handling heterogeneous data distributions and mitigating task interference (Lepikhin et al., 2020). Existing MoE architectures can be broadly categorized into dense MoE and sparse MoE. In dense MoE (Dou et al., 2024; Pan et al., 2024), all experts are activated during both training and inference, with different experts contributing via weighted combinations. In contrast, sparse MoE selects only Top-k experts for each token, leading to significantly improved computational efficiency while maintaining strong model capacity. In addition, prior works such as VLMo (Bao et al., 2022) and BEiT-3 (Wang et al., 2023) introduce mixture-of-modality-experts, which include a vision expert, a language expert, and a vision-language expert to handle different data pairs during vision-language pretraining. During fine-tuning, these models typically encode image and text separately. Inspired by VLMo and BEiT-3, we propose MA-MoE, a sparse MoE architecture consisting of a shared expert and modality-specific experts for sign and text tokens. Compared with existing multimodal MoE models, MA-MoE differs in two key aspects. (1) MA-MoE is specifically designed to address the modality heterogeneity in sign language understanding, which has been largely underexplored in prior SLU works. (2) MA-MoE maintains a consistent architecture during both training and inference, whereas models such as VLMo and BEiT-3 require task-specific adaptations to the inference structure for different downstream tasks. 3 Method Figure 3: Distribution of topics and data sources. 3.1 Datasets The lack of large-scale, high-diversity corpora has long been a major bottleneck in SL research. While substantial progress has been made for American SL (ASL) (Tanzer, 2025; Uthus et al., 2023)s and British SL (BSL) (Albanie et al., 2021) , Chinese SL (CSL) remains under-resourced, limiting the generality and scalability of existing methods. Moreover, existing CSL benchmarks are largely “domain-locked,” consisting of either low-vocabulary laboratory recordings (e.g., CSL-Daily) or highly formalized and linguistically constrained television broadcasts (e.g., CSL-News). As a result, these CSL datasets fail to capture the morphological richness and environmental variability of natural signing. To address these limitations, we introduce CSL-OpenWorld, a large-scale web-sourced dataset for open-domain SLT. As summarized in Table 1, CSL-OpenWorld expands the CSL vocabulary to over 12K tokens by leveraging diverse online platforms, representing a sixfold increase over CSL-Daily and more than that of CSL-News. Furthermore, as illustrated in Figures 3 and 4, CSL-OpenWorld covers a wide range of topics, camera viewpoints, and non-studio environments, providing a rigorous testbed for evaluating model robustness and real-world generalization. CSL-OpenWorld contains 100,458 video samples in total, with 86,387 used for training, making it substantially larger than CSL-Daily and Phoenix14t. The dataset features a high-density token vocabulary, with 12K unique tokens, enabling a challenging benchmark for evaluating the generalization ability of SLT models. The dataset is constructed through a rigorous pipeline involving web crawling, manual annotation, and quality filtering to ensure accurate video–text pairs. By incorporating diverse, open-domain web data, CSL-OpenWorld serves as a critical component in evaluating OneSign, enabling assessment of model robustness across domains and linguistic variations, and providing a useful resource for advancing future SL research. Figure 4: Distribution of video durations and text lengths. Figure 5: Left: the OneSign Architecture. Right: The MA-MoE Layer with Modality Adaptive MoE structure. 3.2 Preliminaries. ISLR task formulation. ISLR focuses on a short sign clip containing a single sign, formulated as a classification problem. Given an input sign sequence X, the objective is to find the optimal sign label g∗∈g^* that maximizes the expected probability under the model parameters θ: g∗=argmaxg∈g′∼pθ(⋅∣)[(g=g′)]g^*= _g E_g p_θ(· )[I(g=g )] (1) where (⋅)I(·) denotes the indicator function, effectively seeking the Maximum A Posteriori (MAP) estimate. CSLR task formulation. CSLR aims to recognize continuous sign sequences by predicting a gloss sequence =g1,…,gLG=\g_1,…,g_L\. Since the mapping involves a latent alignment between signs and glosses, the objective is formulated as the expectation over all valid align paths π that map to G through a many-to-one function ℬB: ∗=argmax∈Lπ∼pθ(⋅∣)[(ℬ(π)=)]G^*= _G ^LE_π p_θ(· )[I(B(π)=G)] (2) SLT task formulation. SLT aims to translate sign sequences into a text sequence ∗=y1,…,yUY^*=\y_1,…,y_U\. This task is commonly formulated as a conditional language generation problem, where the objective is to find ∗Y^* that maximizes the expected log-likelihood (or minimizes the Bayes risk) over the target space UY^U, expressed as: ∗=argmax∈U′∼pθ(⋅∣)[(=′)]Y^*= _Y ^UE_Y p_θ(· )[I(Y=Y )] (3) In practice, this is often decomposed using the chain rule of probability as pθ(∣)=∏u=1Upθ(yu∣<u,)p_θ(Y )= _u=1^Up_θ(y_u _<u,X), where the model learns to predict the next token conditioned on X and the previously generated words <uY_<u. Unified Perspective. Despite differences, all tasks can be unified under a single probabilistic framework. In OneSign, each sample is represented as ,,\X,G,Y\. The model is trained by maximizing the joint conditional likelihood: ℒ(θ)=(,,)∼[log(pθ(,∣))].L(θ)=E_(X,G,Y) [ (p_θ(G,Y )) ]. (4) This unified formulation allows OneSign to be trained under a single maximum-likelihood objective from heterogeneous datasets. Moreover, it naturally subsumes multiple SLU tasks as special cases: • ISLR: ==gG=Y=g, where both denote a single sign label. • CSLR: ==g1,…,gLG=Y=\g_1,…,g_L\, corresponding to a gloss sequence. • GBSLT: G denotes the gloss sequence, while Y represents the spoken language sentence. • GFSLT: G corresponds to a pseudo-gloss sequence, and Y denotes the spoken language sentence. 3.3 Model Architecture As illustrated in Figure 5, our model consists of three main components: (i) a visual encoder that converts an input sign sequence into a sequence of visual features, (i) a text encoder that maps text tokens into continuous text representations, and (i) a Modality-Adaptive Mixture-of-Experts (MA-MoE) decoder that performs unified sign-text modeling. To enable efficient training on large-scale datasets under limited GPU resources, we adopt pose-based representations instead of raw RGB videos for all tasks. Specifically, for a sign sequence =x1,…,xTX=\x_1,…,x_T\, each frame xtx_t is represented using human pose information and decomposed into four semantic parts: xt=xtb,xtf,xtlh,xtrhx_t=\x_t^b,x_t^f,x_t^lh,x_t^rh\, corresponding to torso, face, left hand, and right hand, respectively. We first project each part into a shared high-dimensional feature space using independent linear transformations: tp=Wpxtp,p∈b,f,lh,rh,h_t^p=W_px_t^p, p∈\b,f,lh,rh\, where WpW_p denotes a learnable linear projection. This yields four part-level feature embeddings per frame. Dynamic Graph Fusion for Body Part. To effectively model the spatial correlations and structural dependencies between these body parts, inspired by previous pose modeling (Shi et al., 2019; Ye et al., 2020; Chen et al., 2021), we propose a simple yet effective pose fusing method: the Dynamic Part Graph Fusion (DPGF). Unlike static graph models, DPGF treats each body part as a node in a dynamic graph and learns the underlying topology on-the-fly. Specifically, sign features are concatenated to form a node matrix ∈ℝ4×DH ^4× D, where each node represents a body part. We employ a self-attention mechanism to compute the dynamic adjacency matrix ∈ℝ4×4A ^4× 4, which defines the edge strengths between parts for each frame: =Softmax(Q()K()⊤D)A=Softmax ( Q(H)K(H) D ) (5) where Q and K are linear transformations for Query and Key. This allows the model to adaptively focus on different part correlations (e.g., hand-face coordination) depending on the sign being performed. The node features are updated through a weighted aggregation of their neighbors followed by a GCN-style update with a residual connection: fused=W∗(MLP(V())+)H_fused=W*(MLP(AV(H))+H) (6) where V is the linear projection. The updated nodes are then flattened and projected with W to the final feature space fused∈ℝB×T×CH_fused ^B× T× C, providing a comprehensive representation of the sign gesture for each frame. Multi-Head Classifier Layers. The visual backbone is trained with a CTC objective using G as supervision. When mixing multiple heterogeneous SL datasets, directly using a single shared classifier would require merging their gloss vocabularies, resulting in an excessively large output space and severe label sparsity. Inspired by previous multi-task learning (Liu et al., 2015; Ruder, 2017; Long et al., 2017), we therefore adopt Multi-Head Classifier Layers (MHCL), in which each dataset is assigned a dedicated classification head with its own gloss vocabulary, while all heads share the same visual backbone. This strategy allows the backbone to learn unified visual representations across datasets, while each classifier head specializes in its dataset-specific label space. By decoupling feature learning from dataset-specific vocabularies, this design improves optimization stability and enables scalable multi-dataset pretraining. MHCL is only used during training to stabilize optimization. It is removed at inference, where all outputs are generated by the unified translation module without dataset-specific heads. 3.4 Modality-Adaptive LLM MoE After obtaining sign features, many approaches concatenate sign and text tokens into a single sequence and process them using standard LLM layers. Such a design implicitly treats sign and text tokens as the same modality, overlooking their inherent modality-specific characteristics and relying entirely on shared parameters across all decoder layers. However, this design relies on an implicit assumption: sign tokens and textual tokens are well aligned in the latent space, such that a unified set of transformation layers can model both modalities equally well without explicit distinction. In practice, sign representations derived from pose sequences exhibit markedly different statistical properties, temporal dynamics, and levels of semantic abstraction compared to discrete text tokens. As a result, forcing both modalities to share identical transformation paths introduces a non-negligible modality gap, which can hinder effective cross-modal modeling and degrade overall performance. To address this issue, we propose a Modality-Adaptive Mixture-of-Experts (MA-MoE) architecture that explicitly models modality-specific distributions while preserving shared semantic representations. As illustrated in Fig. 5, MA-MoE augments a standard Transformer block by introducing modality-adaptive expert routing within the feed-forward network (FFN). Specifically, the expert pool is decomposed into three disjoint groups: a shared expert, N sign-specific experts, and M text-specific experts. The shared expert is always activated to capture modality-invariant semantics, while modality-specific experts focus on modeling characteristics unique to sign or text tokens. Let the input to the MA-MoE layer be a sequence of token representations: =1,…,N,i∈ℝDH=\h_1,…,h_N\, _i ^D, where each token is associated with a modality label mi∈sign,textm_i∈\sign,text\. The MA-MoE module consists of a shared expert ℰshared=EsE_shared=\E_s\, a set of sign-specific experts ℰsignE_sign, and a set of text-specific experts ℰtextE_text. For each token ih_i, the shared expert output is first computed as is=Es(i).h_i^s=E_s(h_i). (7) Modality-specific routing is then performed within each expert group. A lightweight router produces routing logits: i=Routersign(i),if mi=sign,Routertext(i),if mi=text,r_i= casesRouter_sign(h_i),&if m_i=sign,\\ Router_text(h_i),&if m_i=text, cases (8) which are normalized over the modality-specific expert set: αi,e=exp(ri,e)∑e′∈ℰmiexp(ri,e′),e∈ℰmi. _i,e= (r_i,e)Σ _e _m_i (r_i,e ), e _m_i. (9) We denote the set of selected top-k expert indices as τ. The final output representation is obtained by aggregating the shared and modality-specific expert outputs: ~i=is+∑e∈ταi,eℰmi,e(i) h_i=h_i^s+ _e∈τ _i,e\,E_m_i,e(h_i) (10) By construction, MA-MoE disentangles modality-invariant alignment from modality-specific modeling, enabling effective joint reasoning over heterogeneous sign and text representations within a unified LLM architecture. Load Balancing Loss. To prevent expert collapse and encourage uniform utilization of modality-specific experts, we introduce a load balancing loss for the MA-MoE. Since the shared expert is always activated, the balancing constraint is applied only to modality-specific experts. For a given modality m∈sign,textm∈\sign,text\, let ℰmE_m denote its corresponding set of modality-specific experts, and let NmN_m be the number of tokens of modality m in a mini-batch. Given the routing probabilities αi,e _i,e for token ih_i and expert e∈ℰme _m, we define the importance of expert e as: Imp(e)=1Nm∑i:mi=mαi,eImp(e)= 1N_m _i:m_i=m _i,e (11) The load balancing loss for modality m is formulated as: ℒbal(m)=|ℰm|∑e∈ℰm(Imp(e)−1|ℰm|)2 _bal^(m)=|E_m| _e _m (Imp(e)- 1|E_m| )^2 (12) The final load balancing loss is computed as the weighted sum over modalities: ℒbal=λsignℒbal(sign)+λtextℒbal(text)L_bal= _sign\,L_bal^(sign)+ _text\,L_bal^(text) (13) where λsign _sign and λtext _text are hyperparameters controlling the strength of the balancing constraint for each modality. This loss encourages the router to distribute tokens evenly across modality-specific experts while still allowing experts to specialize in capturing modality-dependent patterns. As a result, the load balancing loss improves training stability and ensures effective capacity utilization within each modality. 3.5 Training The OneSign model is optimized with two objectives that combine recognition and translation losses: (Θ)=α(X,G)∼[ℓCTC(R(X,ΘR),G)]+β(~i,Y)∼[ℓCE(Y,T(~i))+λsℒbal(s)+λtℒbal(t)]. aligned J( )&=\;α\,E_(X,G) [ _CTC\! (R(X; _R),G ) ]\\ &+β\,E_( h_i,Y) [ _CE\! (Y,T( h_i) )+ _sL_bal^(s)+ _tL_bal^(t) ]. aligned (14) Here, R and T denote the recognition and translation modules, respectively. Following (Gan et al., 2026d), the CTC loss supervises the visual backbone with MHCL, while the translation module T is optimized with a translation loss with MOE balancing losses. During training, we first warm up the visual backbone for 20 epochs by setting α=1α=1,β=0β=0, and activate the translation objective by further setting β=1β=1 in the subsequent 30 epochs. Model DEV TEST WER del/ins WER del/ins OneSign♡ 21.1 7.2/3.0 22.1 8.1/2.3 OneSign♠ W/O MHCL 23.5 11.8/2.1 24.9 11.7/2.5 OneSign♠ 20.0 5.2/3.1 20.1 5.7/2.9 Table 2: Effect of Multi-Head Classification Layer. Dataset TEST WER Rouge BLEU1 BLEU4 Phoenxi14T 22.1 51.18 52.68 26.22 +CSLR 21.5 51.65 52.45 26.25 +ISLR+CSLR 21.0 52.18 52.65 26.24 +ISLR+CSLR+SLT 20.1 52.01 53.36 26.45 Table 3: Effect of heterogeneous training datasets. 4 Ablation Study Following previous work (Camgoz et al., 2018; Chen et al., 2022; Gan et al., 2024), we perform ablation studies on the Phoenix14t dataset to verify the effectiveness of the proposed OneSign model. Please refer to Appendix C for detailed implementation settings. A1: Effect of the Multi-Head Classification Layer. Using a single CTC-based classification head for heterogeneous SLU datasets leads to a large vocabulary and unstable training. Table 2 compares three settings: (1) OneSign♡ , trained on a single dataset; (2) OneSign♠ W/O MHCL, trained on multiple datasets with a shared head; and (3) OneSign♠ with an independent head per dataset. As can be seen, directly sharing a head across datasets dilutes the original distributions and yields limited gains, whereas MHCL improves both performance and training stability, highlighting its effectiveness for multi-dataset learning. Experts DEV TEST Shared Sign Text WER Rouge BLEU1 BLEU4 WER Rouge BLEU1 BLEU4 ✓ 22.3 49.12 49.89 23.45 22.5 48.99 49.72 23.01 ✓ ✓ 21.0 50.12 49.65 23.65 21.4 50.01 49.41 23.47 ✓ ✓ 20.8 50.64 50.12 24.36 21.0 50.03 49.95 24.13 ✓ ✓ ✓ 20.0 52.34 53.41 26.98 20.1 52.01 53.36 26.45 Table 4: Effect of MA-MOE. Ns=NtN_s=N_t Top-k Params (M) Activated (M) Dev-Rouge Dev-BLEU1 Dev-BLEU4 0 0 1235.8 1235.8 49.45 49.36 22.67 2 1 4457.0 2041.1 52.34 53.41 26.98 4 1 7678.2 2041.1 52.74 53.65 26.78 4 2 7678.2 3651.7 52.99 53.94 26.97 Table 5: Impact of activated expert count. Table 6: Effects of Dynamic Part Graph Fusion. Dev Test WER ROUGE BLEU1 BLEU4 WER ROUGE BLEU1 BLEU4 W/O DPGF 21.6 51.01 52.87 25.83 21.1 51.07 52.17 25.63 W/ Fix 20.6 51.87 52.36 26.36 20.7 51.46 52.48 25.87 W/ DPGF 20.0 52.35 53.41 26.98 20.1 52.01 53.36 26.45 Table 7: Training speed and Inference speed. Model Training Time Inference Time (250 frames) ∼ 10.4 min/epoch ∼ 12.5 samples/s +MA-MOE ∼ 11.2 min/epoch ∼ 11.8 samples/s A2: Effect of Heterogeneous Training Datasets. Table 3 shows the impact of jointly training on heterogeneous datasets: ISLR (WLASL, ASLCitizen, MultiVSL), CSLR (CSL-Daily, Phoenix14t), and SLT (CSL-Daily, Phoenix14t, How2Sign, CSL-OpenWorld), evaluated on Phoenix14t. Adding datasets without task-specific finetuning yields only modest gains. The results show that naively adding heterogeneous datasets brings only modest improvements (e.g., +CSLR to +ISLR+CSLR), as the lack of any downstream finetuning can dilute the original dataset distributions. Still, OneSign effectively leverages complementary supervision, demonstrating the benefit of cross-task knowledge transfer in unified multi-dataset training. A3: Effect of the Proposed MA-MoE. Table 4 presents an ablation study on different expert configurations in the proposed MA-MoE. Using only the shared expert (i.e., Llama3.2 1B model) results in suboptimal performance, indicating that modality-agnostic modeling alone is insufficient. Adding either a sign or text specific expert on top of the shared expert consistently improves performance, highlighting the value of modality-aware MOE design. The full configuration, which includes both sign and text specific experts, achieves the best results across all metrics, confirming that MA-MoE effectively balances shared semantic modeling with modality-specific representation learning. A4: Effect of Sparse MA-MoE Configurations. Table 5 examines expert sparsity by varying the total number of experts (Ns=NtN_s=N_t) and activated experts (Top-k). Activating a single expert with more total experts gives minor gains, while activating multiple experts allows aggregation of complementary information, but requires more computational resources. To balance performance and efficiency, we choose Ns=Nt=2N_s=N_t=2 with Top-k=1k=1 as our final setting. A5: Effect of Dynamic Part Graph Fusion. Table 6 reports an ablation study on the proposed Dynamic Part Graph Fusion (DPGF) module. When DPGF is removed (i.e., features from different parts are fused using a simple linear operation), the model performs worse on all metrics, indicating that naive part fusion is insufficient to capture fine-grained part-level interactions. Applying a fixed fusion strategy (W/ Fix), where fusion graphs are predefined for each part, brings modest and consistent improvements, suggesting that part-aware aggregation is beneficial but limited by its static design. In contrast, our model with DPGF module achieves the best performance , reducing WER from 21.1 to 20.1 and improving BLEU4 from 25.63 to 26.45. These gains indicate that dynamically adjusting fusion weights according to input content enables more effective modeling of inter-part dependencies, which in turn leads to more accurate recognition and translation. Method Extra ASL Citizen WLASL2000 MultiVSL1000 Pose RGB FT Top@1 Top@5 P-I P-C Top@1 Top@5 I3D ✓ ✓ 63.10 86.09 37.91 35.90 40.60 73.03 HMA (Hu et al., 2021) ✓ ✓ - - 37.91 35.90 - - NLA-SLR (Zuo et al., 2023) ✓ ✓ - - 61.05 58.05 - - Uni-Sign (Li et al., 2025) ✓ ✓ ✓∗ - - 63.52 61.32 - - SHuBERT (Gueuwou et al., 2025a) ✓ ✓∗ 65.00 87.00 60.90 58.01 - - Pose-based ST-GCN (Yu et al., 2017) ✓ ✓ 59.52 82.68 34.40 32.53 40.61* 73.04* BEST (Zhao et al., 2023) ✓ ✓ - - 46.25 43.52 - - SignBERT+ (Hu et al., 2023a) ✓ ✓ ✓ - - 48.85 46.37 - - MSLU (Zhou et al., 2025) ✓ ✓∗ - - 56.29 53.29 - - Uni-Sign (Li et al., 2025) ✓ ✓∗ 60.92* 83.41* 63.13 60.90 70.92* 89.90* OneSign♡ ✓ ✓ 65.41 84.19 60.45 61.02 80.71 95.43 OneSign♠ ✓ 68.41 89.24 64.15 65.34 82.34 96.43 Table 8: ISLR results on various benchmarks. ∗ denotes results reproduced by us using publicly available code, and ✓∗ ^* indicates pretraining on large-scale SL datasets. A6: Training and Inference Speed We evaluate the training and testing time on the Phoenix14t using a single NVIDIA RTX 4090 GPU. Specifically, training time is measured on the Phoenix14t training set, while inference speed is evaluated on test samples with an average length of 250 frames, averaged over 100 runs. As observed, the model requires about 10–13 minutes per epoch during training. During inference, it achieves a throughput of 10–12 samples per second on a single GPU. These results show that the proposed MA-MOE achieves good efficiency in both training and inference, making it practical for large-scale SL tasks. 5 Comparisons We compare OneSign with existing task-specific models on ISLR, CSLR, and SLT benchmarks. To systematically evaluate the effect of unified training and inference, we report results under two training and evaluation settings: (1) OneSign♡ is trained and evaluated on a single dataset(i.e., each corresponding dataset). (2) OneSign♠ is trained jointly on the training splits of all available datasets and directly evaluated on each benchmark without any further fine-tuning, measuring the generalization capability of a single unified checkpoint.Note that a direct comparison with RGB-based models is not entirely fair, as RGB inputs contain richer visual information. Evaluation on ISLR Tasks. Table 8 compares OneSign with existing ISLR methods. Prior approaches are mostly RGB- or pose-based and often require dataset-specific pretraining or fine-tuning. Under the dataset-specific setting, OneSign♡ already show promising performance across all benchmarks (e.g., 65.41 Top@1 on ASL Citizen vs. 59.52 for ST-GCN). On WLASL2000 and MultiVSL1000, it similarly achieves clear gains. OneSign♠ further improves performance, establishing a new baseline under a unified protocol. These results show that OneSign effectively leverages heterogeneous supervision and achieves strong multi-dataset performance with a single checkpoint. Method Extra CSL-Daily Phoenix14T Pose RGB FT Dev Test Dev Test SignBT (Zhou et al., 2021a) ✓ ✓ 33.2 33.2 - - CorrNet (Hu et al., 2023b) ✓ ✓ 30.6 30.1 18.9 20.5 SEN (Hu et al., 2023c) ✓ ✓ 31.1 30.7 19.3 20.7 C2ST (Zhang et al., 2023) ✓ ✓ 25.9 25.8 17.5 17.7 SignGraph (Gan et al., 2024) ✓ ✓ 27.3 26.4 17.8 19.1 Uni-Sign (Li et al., 2025) ✓ ✓ ✓∗ 26.7 26.0 - - TwoStream (Chen et al., 2022) ✓ ✓ ✓ 25.4 25.3 17.7 19.3 Pose-based CoSign1s (Jiao et al., 2023) ✓ ✓ 29.5 29.1 20.4 20.6 MSLU (Zhou et al., 2025) ✓ ✓ 28.6 27.9 20.1 21.3 Uni-Sign (Li et al., 2025) ✓ ✓∗ 28.2 27.4 21.4* 22.0* OneSign♡ ✓ ✓ 29.0 29.1 21.1 22.1 OneSign♠ ✓ 25.2 25.7 20.0 20.1 Table 9: CSLR results on CSL-Daily and Phoenix14T. “FT” denotes fine-tuning on each dataset. Table 10: Comparison of SLT performance on Phoenix14T and CSL-Daily. ∗ denotes results reproduced by us using publicly available code. We underline pose-based results that outperform our baseline model for clarity. SLT Extra Phoenix14T CSL-Daily DEV TEST DEV TEST Pose RGB FT ROUGE BLEU1 BLEU4 ROUGE BLEU1 BLEU4 ROUGE BLEU1 BLEU4 ROUGE BLEU1 BLEU4 PGG-SLT (Guo et al., 2025) ✓ ✓ 52.01 53.16 27.09 51.85 53.45 26.85 - - - - - - GFSLT-VLP (Yin et al., 2023) ✓ ✓ 43.72 44.08 22.12 42.49 43.71 21.44 36.44 39.20 11.07 36.70 39.37 11.00 SignCL (Ye et al., 2024) ✓ ✓ 49.04 49.76 22.74 - - - 48.92 47.47 16.16 Sign2GPT (Wong et al., 2024) ✓ ✓ - - 48.90 49.54 22.52 - - - 42.36 41.75 15.40 GFSLT-VLP-SignCL (Ye et al., 2024) ✓ ✓ - - - 49.04 49.76 22.74 - - 48.92 47.47 16.16 SignLLM (Gong et al., 2024) ✓ ✓ 44.49 46.88 25.25 47.23 45.21 23.40 39.18 42.45 12.23 39.91 39.55 15.75 LLaVA-SLT (Liang et al., 2024) ✓ ✓ - - - 50.44 51.20 23.43 - - - 51.26 52.15 20.42 FLa-LLM (Chen et al., 2024) ✓ ✓ - - - 45.27 46.29 23.09 - - - 37.25 37.13 14.20 C2RL (Chen et al., 2025) ✓ ✓ - - - 50.96 52.81 26.75 - - - 48.21 49.32 21.61 MixSignGraph (Gan et al., 2026d) ✓ ✓ 51.71 51.07 24.87 51.14 50.01 24.02 49.16 49.98 20.43 49.93 50.24 20.78 TwoStream (Chen et al., 2022) ✓ ✓ ✓ 54.08 54.32 28.66 53.48 54.90 28.95 55.10 55.21 25.76 55.72 55.44 25.79 Uni-Sign (Li et al., 2025) ✓ ✓ ✓∗ - - - - - - 56.03 55.30 26.25 56.51 55.08 26.36 Pose-based VAP (Jiao et al., 2024) ✓ ✓ 51.47 52.78 26.62 51.28 53.07 26.02 51.19 53.31 23.84 51.09 52.98 23.65 Uni-Sign (Li et al., 2025) ✓ ✓∗ 48.14* 49.14* 22.47* 48.10* 49.26* 22.78* 54.34 53.24 25.27 53.86 54.92 25.61 OneSign♡ ✓ ✓ 51.28 52.68 26.38 51.18 52.68 26.22 50.05 52.43 24.85 51.39 51.02 24.42 OneSign♠ ✓ 52.34 53.41 26.98 52.01 53.36 26.45 54.31 53.45 25.28 53.39 54.45 26.15 Table 11: Comparison of gloss-free SLT performance on How2Sign and CSL-OpenWorld. Gloss-free SLT Extra How2Sign CSL-OpenWorld DEV TEST DEV TEST Pose RGB FT ROUGE BLEU1 BLEU4 ROUGE BLEU1 BLEU4 ROUGE BLEU1 BLEU4 ROUGE BLEU1 BLEU4 PGG-SLT (mBART) (Guo et al., 2025) ✓ ✓ ✓ 32.6 41.8 15.9 31.50 38.90 13.10 - - - - - - YouTube-SLT∗ (Uthus et al., 2023) ✓ ✓ - - - - 14.96 1.22 - - - - - - C2C^2RL (Chen et al., 2025) ✓ ✓ - - - 27.02 29.07 9.37 - - - - - - FLa-LLM (Chen et al., 2024) ✓ ✓ - - - 27.81 29.81 9.66 - - - - - - GloFE-VN (Lin et al., 2023) ✓ ✓ 12.98 15.21 2.37 12.61 14.94 2.24 15.92∗ 16.33∗ 4.36∗ 15.86∗ 16.23∗ 4.30∗ SSVP-SLT (Rust et al., 2024) ✓ ✓ - - - 25.70 30.20 7.00 - - - - - - SignMusketeers (Gueuwou et al., 2025b) ✓ ✓ - - - - 18.80 4.20 - - - - - - SLT-IV (Tarrés et al., 2023) ✓ ✓ - 34.01 8.03 - - - - - - MixSignGraph (Gan et al., 2026d) ✓ ✓ 29.24 34.82 11.28 28.01 34.74 10.41 22.30∗ 23.75∗ 8.75∗ 22.32∗ 24.00∗ 8.91∗ SHuBERT (Gueuwou et al., 2025a) ✓ ✓∗ - - - - - 16.20 - - - - - - Pose-based VAP (Jiao et al., 2024) ✓ ✓ ✓ 30.27 42.34 14.91 27.77 39.22 12.87 - - - - - - Uni-Sign (Li et al., 2025) ✓ ✓∗ - - - 34.30 40.40 14.50 30.45∗ 31.44∗ 12.47∗ 31.48∗ 32.17∗ 12.32∗ OneSign♡ ✓ 35.19 39.46 14.92 36.46 39.47 14.51 31.65 34.19 13.88 31.57 33.60 13.48 OneSign♠ ✓ 37.16 41.56 15.41 39.15 42.78 15.07 32.90 35.18 14.71 32.99 34.83 14.44 Evaluation on CSLR Tasks Table 9 shows results on CSL-Daily and Phoenix14t. Under dataset-specific settings, OneSign♡ achieves strong performance using only pose inputs (29.1 WER on CSL-Daily, 22.1 on Phoenix14t), matching or surpassing several RGB-based methods. OneSign♠, trained jointly on heterogeneous datasets without fine-tuning, further reduces WER to 25.7 and 20.1, respectively, demonstrating good performance. Compared with prior pose-based methods that require fine-tuning, OneSign♠ achieves comparable or better results under a unified evaluation setting. Evaluation on SLT. Tables 10 and 11 report results on two gloss-based SLT benchmarks: Phoenix14t, CSL-Daily, and two gloss-free SLT benchmarks: How2Sign, and CSL-OpenWorld. Unlike most existing SLT methods that rely on dataset-specific fine-tuning, OneSign operates purely on pose and supports unified training without fine-tuning. On Phoenix14t and CSL-Daily (Table 10), OneSign♡ already achieves competitive performance using only pose features. Joint training further brings consistent gains: OneSign♠ improves BLEU4 from 26.22 to 26.45 on Phoenix14t and from 24.42 to 26.15 on CSL-Daily. On the more challenging How2Sign and CSL-OpenWorld benchmarks (Table 11), OneSign shows strong generalization. Without fine-tuning, OneSign♠ achieves the best or highly competitive results among pose-based methods, outperforming prior approaches such as Uni-Sign and VAP on many metrics. Overall, OneSign achieves strong cross-dataset performance with a single unified checkpoint, without relying on RGB inputs or dataset-specific fine-tuning. 6 Conclusion In this paper, we present a unified framework that integrates ISLR, CSLR and SLT within a single model. By reformulating all tasks under a unified paradigm, our approach enables joint training and inference across different SL tasks using a single model checkpoint, without task-specific fine-tuning. To support effective learning from heterogeneous datasets, we introduce the MHCL that reduces vocabulary conflicts while allowing the shared backbone to be jointly optimized. Furthermore, to accommodate the diverse characteristics of sign and text modalities within a shared decoder, we propose a MA-MoE architecture with both shared and modality-specific experts, dynamically activated by a modality-aware router. Extensive experiments on seven public benchmarks across ISLR, CSLR, and SLT tasks demonstrate that our unified framework consistently achieves strong performance, validating its effectiveness across tasks and datasets. References Albanie et al. (2021) S. Albanie, G. Varol, L. Momeni, H. Bull, T. Afouras, H. Chowdhury, N. Fox, B. Woll, R. Cooper, A. McParland, et al. Bbc-oxford british sign language dataset. arXiv preprint arXiv:2111.03635. Cited by: Appendix F, Table 1, §1, §3.1. Bao et al. (2022) H. Bao, W. Wang, L. Dong, Q. Liu, O. K. Mohammed, K. Aggarwal, S. Som, S. Piao, and F. Wei VLMo: unified vision-language pre-training with mixture-of-modality-experts. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 32897–32912. External Links: Link Cited by: §2. Camgoz et al. (2018) N. C. Camgoz, S. Hadfield, O. Koller, H. Ney, and R. Bowden Neural sign language translation. In CVPR, p. 7784–7793. Cited by: Table 12, 4th item, Table 1, §4. Camgoz et al. (2020) N. C. Camgoz, O. Koller, S. Hadfield, and R. Bowden Sign language transformers: joint end-to-end sign language recognition and translation. In CVPR, p. 10023–10033. Cited by: §2. Camgöz et al. (2021) N. C. Camgöz, B. Saunders, G. Rochette, M. Giovanelli, G. Inches, R. Nachtrab-Ribback, and R. Bowden Content4all open research sign language translation datasets. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), p. 1–5. Cited by: Appendix F, Table 1, Table 1. Chen et al. (2022) Y. Chen, R. Zuo, F. Wei, Y. Wu, S. Liu, and B. Mak Two-stream network for sign language recognition and translation. NeurIPS 35, p. 17043–17056. Cited by: §4, Table 10, Table 9. Chen et al. (2021) Y. Chen, Z. Zhang, C. Yuan, B. Li, Y. Deng, and W. Hu Channel-wise topology refinement graph convolution for skeleton-based action recognition. In ICCV, p. 13359–13368. Cited by: §3.3. Chen et al. (2025) Z. Chen, B. Zhou, Y. Huang, J. Wan, Y. Hu, H. Shi, Y. Liang, Z. Lei, and D. Zhang C 2 rl: content and context representation learning for gloss-free sign language translation and retrieval. IEEE Transactions on Circuits and Systems for Video Technology. Cited by: Table 10, Table 11. Chen et al. (2024) Z. Chen, B. Zhou, J. Li, J. Wan, Z. Lei, N. Jiang, Q. Lu, and G. Zhao Factorized learning assisted with large language model for gloss-free sign language translation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), p. 7071–7081. Cited by: Table 10, Table 11. Cheng et al. (2020) K. L. Cheng, Z. Yang, Q. Chen, and Y. Tai Fully convolutional networks for continuous sign language recognition. In ECCV, Cited by: Appendix F. Dai et al. (2024) D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, et al. Deepseekmoe: towards ultimate expert specialization in mixture-of-experts language models. In ACL, p. 1280–1297. Cited by: §2. Desai et al. (2023) A. Desai, L. Berger, F. Minakov, N. Milano, C. Singh, K. Pumphrey, R. Ladner, H. Daumé I, A. X. Lu, N. Caselli, et al. ASL citizen: a community-sourced dataset for advancing isolated sign language recognition. NeurIPS 36, p. 76893–76907. Cited by: Table 12, 2nd item, Appendix F. Dinh et al. (2025) N. S. Dinh, T. D. Nguyen, D. T. Tran, N. D. H. Pham, T. H. Tran, N. A. Tong, Q. H. Hoang, and P. Le Nguyen Sign language recognition: a large-scale multi-view dataset and comprehensive evaluation. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), p. 7887–7897. Cited by: Table 12, 3rd item, Appendix F. Dou et al. (2024) S. Dou, E. Zhou, Y. Liu, S. Gao, W. Shen, L. Xiong, Y. Zhou, X. Wang, Z. Xi, X. Fan, et al. LoRAMoE: alleviating world knowledge forgetting in large language models via moe-style plugin. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1932–1945. Cited by: §2. Duarte et al. (2021) A. Duarte, S. Palaskar, L. Ventura, D. Ghadiyaram, K. DeHaan, F. Metze, J. Torres, and X. Giro-i-Nieto How2Sign: a large-scale multimodal dataset for continuous american sign language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 2735–2744. Cited by: Table 12, 6th item, Table 1. Fedus et al. (2022) W. Fedus, B. Zoph, and N. Shazeer Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), p. 1–39. Cited by: §2. Gan et al. (2026a) S. Gan, X. Liu, Y. Yin, Z. Jiang, B. Guo, L. Xie, S. Lu, and H. Wen SignLlama: enhancing gloss-free sign language translation by prioritizing visual features for llms. Proceedings of the ACM international conference on multimedia. Cited by: §1. Gan et al. (2026b) S. Gan, X. Liu, Y. Yin, N. Liu, K. Liu, D. Tuerdaken, Z. Jiang, L. Xie, S. Lu, and H. Wen Learning effective sign features without text for gloss-free sign language translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9827–9836. Cited by: §1. Gan et al. (2026c) S. Gan, L. Wang, X. Liu, Y. Yin, K. Liu, S. Lu, and L. Xie Sign language question answering: a new task, benchmark, and baseline for sign language understanding. arXiv preprint arXiv:2607.27826. Cited by: §1. Gan et al. (2024) S. Gan, Y. Yin, Z. Jiang, H. Wen, L. Xie, and S. Lu SignGraph: a sign sequence is worth graphs of nodes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13470–13479. Cited by: §1, §2, §2, §4, Table 9. Gan et al. (2023a) S. Gan, Y. Yin, Z. Jiang, K. Xia, L. Xie, and S. Lu Contrastive learning for sign language recognition and translation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, p. 763–772. Cited by: §1. Gan et al. (2026d) S. Gan, Y. Yin, Z. Jiang, L. Xie, S. Lu, and H. Wen Mixsigngraph: a sign sequence is worth mixed graphs of nodes. Advances in Neural Information Processing Systems 38, p. 139371–139395. Cited by: Appendix C, §1, §2, §3.5, Table 10, Table 11. Gan et al. (2021) S. Gan, Y. Yin, Z. Jiang, L. Xie, and S. Lu Skeleton-aware neural sign language translation. In M, p. 4353–4361. Cited by: §1. Gan et al. (2023b) S. Gan, Y. Yin, Z. Jiang, L. Xie, and S. Lu Towards real-time sign language recognition and translation on edge devices. In Proceedings of the 31st ACM international conference on multimedia, p. 4502–4512. Cited by: §1, §2. Gong et al. (2024) J. Gong, L. G. Foo, Y. He, H. Rahmani, and J. Liu Llms are good sign language translators. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 18362–18372. Cited by: §2, Table 10. Gueuwou et al. (2025a) S. Gueuwou, X. Du, G. Shakhnarovich, K. Livescu, and A. H. Liu SHuBERT: self-supervised sign language representation learning via multi-stream cluster prediction. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 28792–28810. External Links: Link, ISBN 979-8-89176-251-0 Cited by: §1, §2, §2, Table 8, Table 11. Gueuwou et al. (2025b) S. Gueuwou, X. Du, G. Shakhnarovich, and K. Livescu Signmusketeers: an efficient multi-stream approach for sign language translation at scale. Findings of the Association for Computational Linguistics: ACL 2025. Cited by: §2, Table 11. Gueuwou et al. (2023) S. Gueuwou, K. Takyi, M. Müller, M. S. Nyarko, R. Adade, and R. O. M. Gyening Afrisign: machine translation for african sign languages. In 4th Workshop on African Natural Language Processing, Cited by: Appendix F, Table 1. Guo et al. (2016) D. Guo, W. Zhou, M. Wang, and H. Li Sign language recognition based on adaptive hmms with data augmentation. In 2016 IEEE International Conference on Image Processing (ICIP), p. 2876–2880. Cited by: Appendix F. Guo et al. (2025) J. Guo, P. Li, and T. Cohn Bridging sign and spoken languages: pseudo gloss generation for sign language translation. arXiv preprint arXiv:2505.15438. Cited by: §1, §2, Table 10, Table 11. Hanke et al. (2020) T. Hanke, M. Schulder, R. Konrad, and E. Jahn Extending the public dgs corpus in size and depth. In Proceedings of the LREC2020 9th workshop on the representation and processing of sign languages: Sign language resources in the service of the language community, technological challenges and application perspectives, p. 75–82. Cited by: Appendix F, Table 1. Hao et al. (2021) A. Hao, Y. Min, and X. Chen Self-mutual distillation learning for continuous sign language recognition. In ICCV, p. 11303–11312. Cited by: Appendix F. Hu et al. (2023a) H. Hu, W. Zhao, W. Zhou, and H. Li SignBERT+: hand-model-aware self-supervised pre-training for sign language understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: Table 8. Hu et al. (2021) H. Hu, W. Zhou, and H. Li Hand-model-aware sign language recognition. In AAAI, Vol. 35, p. 1558–1566. Cited by: Appendix F, §2, Table 8. Hu et al. (2020) H. Hu, W. Zhou, J. Pu, and H. Li Global-local enhancement network for nmfs-aware sign language recognition. arXiv preprint arXiv:2008.10428. Cited by: Appendix F. Hu et al. (2022) L. Hu, L. Gao, Z. Liu, and W. Feng Temporal lift pooling for continuous sign language recognition. In European conference on computer vision, p. 511–527. Cited by: Appendix F. Hu et al. (2023b) L. Hu, L. Gao, Z. Liu, and W. Feng Continuous sign language recognition with correlation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 2529–2539. Cited by: Appendix F, Table 9. Hu et al. (2023c) L. Hu, L. Gao, Z. Liu, and W. Feng Self-emphasizing network for continuous sign language recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, p. 854–862. Cited by: Appendix F, §2, Table 9. Jiang et al. (2024) A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088. Cited by: §2. Jiao et al. (2024) P. Jiao, Y. Min, and X. Chen Visual alignment pre-training for sign language translation. In European Conference on Computer Vision, p. 349–367. Cited by: §2, Table 10, Table 11. Jiao et al. (2023) P. Jiao, Y. Min, Y. Li, X. Wang, L. Lei, and X. Chen CoSign: exploring co-occurrence signals in skeleton-based continuous sign language recognition. In ICCV, p. 20676–20686. Cited by: Appendix F, §2, Table 9. Jin and Zhao (2021) T. Jin and Z. Zhao Contrastive disentangled meta-learning for signer-independent sign language translation. In M, p. 5065–5073. Cited by: §2. Kan et al. (2022) J. Kan, K. Hu, M. Hagenbuchner, A. C. Tsoi, M. Bennamoun, and Z. Wang Sign language translation with hierarchical spatio-temporal graph neural network. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 3367–3376. Cited by: §2. Ko et al. (2019) S. Ko, C. J. Kim, H. Jung, and C. Cho Neural sign language translation based on human keypoint estimation. Applied sciences 9 (13), p. 2683. Cited by: Appendix F, Table 1. Koller et al. (2019) O. Koller, C. Camgoz, H. Ney, and R. Bowden Weakly supervised learning with multi-stream cnn-lstm-hmms to discover sequential parallelism in sign language videos. TPAMI. Cited by: §1. Lepikhin et al. (2020) D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen Gshard: scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668. Cited by: §2. Li et al. (2020a) D. Li, C. Rodriguez, X. Yu, and H. Li Word-level deep sign language recognition from video: a new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, p. 1459–1469. Cited by: Table 12, 1st item, Appendix F. Li et al. (2020b) D. Li, X. Yu, C. Xu, L. Petersson, and H. Li Transferring cross-domain knowledge for video sign language recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 6205–6214. Cited by: Appendix F, §2. Li et al. (2025) Z. Li, W. Zhou, W. Zhao, K. Wu, H. Hu, and H. Li Uni-sign: toward unified sign language understanding at scale. arXiv preprint arXiv:2501.15187. Cited by: Table 1, §1, §2, Table 8, Table 8, Table 10, Table 10, Table 11, Table 9, Table 9. Liang et al. (2024) H. Liang, C. Huang, Y. Xu, C. Tang, W. Ye, J. Zhang, X. Chen, J. Yu, and L. Xu LLaVA-slt: visual language tuning for sign language translation. arXiv preprint arXiv:2412.16524. Cited by: §2, Table 10. Lin et al. (2023) K. Lin, X. Wang, L. Zhu, K. Sun, Y. Yang, et al. Gloss-free end-to-end sign language translation. In ACL, Cited by: Table 11. Liu et al. (2016) T. Liu, W. Zhou, and H. Li Sign language recognition with long short-term memory. In ICIP, p. 2871–2875. Cited by: Appendix F. Liu et al. (2026a) X. Liu, S. Gan, Y. Yin, B. Guo, Z. Jiang, S. Meng, L. Xie, and S. Lu SignPR: a progressive vector-quantized diffusion framework for sign language production. p. 2198–2208. Cited by: §1. Liu et al. (2026b) X. Liu, S. Gan, Y. Yin, J. Yin, B. Guo, Y. Sun, Z. Jiang, and L. Xie Bridging the gap between semantics and reconstruction: unifying sign language translation and production. Proceedings of the 31st ACM international conference on multimedia. Cited by: §1. Liu et al. (2015) X. Liu, J. Gao, X. He, L. Deng, K. Duh, and Y. Wang Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In Proceedings of the 2015 conference of the North American chapter of the association for computational linguistics: Human language technologies, p. 912–921. Cited by: §3.3. Long et al. (2017) M. Long, Z. Cao, J. Wang, and P. S. Yu Learning multiple tasks with multilinear relationship networks. Advances in neural information processing systems 30. Cited by: §3.3. Min et al. (2021) Y. Min, A. Hao, X. Chai, and X. Chen Visual alignment constraint for continuous sign language recognition. In ICCV, p. 11542–11551. Cited by: Appendix F. Niu and Mak (2020) Z. Niu and B. Mak Stochastic fine-grained labeling of multi-state sign glosses for continuous sign language recognition. In ECCV, p. 172–186. Cited by: Appendix F, §2. Orbay and Akarun (2020) A. Orbay and L. Akarun Neural sign language translation by learning tokenization. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), p. 222–228. Cited by: §2. Pan et al. (2024) B. Pan, Y. Shen, H. Liu, M. Mishra, G. Zhang, A. Oliva, C. Raffel, and R. Panda Dense training, sparse inference: rethinking training of mixture-of-experts language models. arXiv preprint arXiv:2404.05567. Cited by: §2. Parelli et al. (2022) M. Parelli, K. Papadimitriou, G. Potamianos, G. Pavlakos, and P. Maragos Spatio-temporal graph convolutional networks for continuous sign language recognition. In ICASSP, Cited by: Appendix F. Pu et al. (2020) J. Pu, W. Zhou, H. Hu, and H. Li Boosting continuous sign language recognition via cross modality augmentation. In M, p. 1497–1505. Cited by: Appendix F. Ruder (2017) S. Ruder An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098. Cited by: §3.3. Rust et al. (2024) P. Rust, B. Shi, S. Wang, N. C. Camgoz, and J. Maillard Towards privacy-aware sign language translation at scale. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 8624–8641. External Links: Link Cited by: Table 11. Shi et al. (2022) B. Shi, D. Brentari, G. Shakhnarovich, and K. Livescu Open-domain sign language translation learned from online video. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 6365–6379. Cited by: Appendix F, Table 1. Shi et al. (2019) L. Shi, Y. Zhang, J. Cheng, and H. Lu Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 12026–12035. Cited by: §3.3. Sincan et al. (2023) O. M. Sincan, N. C. Camgoz, and R. Bowden Is context all you need? scaling neural sign language translation to large domains of discourse. In Proceedings of the IEEE/CVF international conference on computer vision, p. 1955–1965. Cited by: §1. Tang et al. (2021) S. Tang, D. Guo, R. Hong, and M. Wang Graph-based multimodal sequential embedding for sign language translation. TMM. Cited by: Appendix F. Tanzer (2025) G. Tanzer Fleurs-asl: including american sign language in massively multilingual multitask evaluation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 6167–6191. Cited by: Appendix F, §3.1. Tarrés et al. (2023) L. Tarrés, G. I. Gállego, A. Duarte, J. Torres, and X. Giró-i-Nieto Sign language translation from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 5624–5634. Cited by: Table 11. Uebersax et al. (2011) D. Uebersax, J. Gall, M. Van den Bergh, and L. Van Gool Real-time sign language letter and word recognition from depth data. In 2011 IEEE international conference on computer vision workshops (ICCV Workshops), p. 383–390. Cited by: Appendix F. Uthus et al. (2023) D. Uthus, G. Tanzer, and M. Georg Youtube-asl: a large-scale, open-domain american sign language-english parallel corpus. Advances in Neural Information Processing Systems 36, p. 29029–29047. Cited by: Appendix F, Table 1, §3.1, Table 11. Wang et al. (2023) W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, et al. Image as a foreign language: beit pretraining for vision and vision-language tasks. In CVPR, p. 19175–19186. Cited by: §2. Wei et al. (2020) C. Wei, J. Zhao, W. Zhou, and H. Li Semantic boundary detection with reinforcement learning for continuous sign language recognition. TCSVT 31 (3), p. 1138–1149. Cited by: Appendix F. Wei and Chen (2023) F. Wei and Y. Chen Improving continuous sign language recognition with cross-lingual signs. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 23612–23621. Cited by: Appendix F, §2, §2. Wong et al. (2024) R. Wong, N. C. Camgoz, and R. Bowden SIGN2GPT: leveraging large language models for gloss-free sign language translation. In ICLR 2024: The Twelfth International Conference on Learning Representations, Cited by: §1, Table 10. Wong et al. (2025) R. Wong, N. C. Camgoz, and R. Bowden Signrep: enhancing self-supervised sign representations. arXiv preprint arXiv:2503.08529. Cited by: Appendix F. Yang et al. (2019) Z. Yang, Z. Shi, X. Shen, and Y. Tai SF-net: structured feature network for continuous sign language recognition. arXiv preprint arXiv:1908.01341. Cited by: Appendix F. Ye et al. (2020) F. Ye, S. Pu, Q. Zhong, C. Li, D. Xie, and H. Tang Dynamic gcn: context-enriched topology learning for skeleton-based action recognition. In Proceedings of the 28th ACM international conference on multimedia, p. 55–63. Cited by: §3.3. Ye et al. (2024) J. Ye, X. Wang, W. Jiao, J. Liang, and H. Xiong Improving gloss-free sign language translation by reducing representation density. In NeurIPS, Cited by: §2, Table 10, Table 10. Yin et al. (2022) A. Yin, Z. Zhao, W. Jin, M. Zhang, X. Zeng, and X. He Mlslt: towards multilingual sign language translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 5109–5119. Cited by: Appendix F, Table 1. Yin et al. (2021) A. Yin, Z. Zhao, J. Liu, W. Jin, M. Zhang, X. Zeng, and X. He SimulSLT: end-to-end simultaneous sign language translation. In M, p. 4118–4127. Cited by: §2. Yin et al. (2023) A. Yin, T. Zhong, L. Tang, W. Jin, T. Jin, and Z. Zhao Gloss attention for gloss-free sign language translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 2551–2562. Cited by: Table 10. Yu et al. (2017) B. Yu, H. Yin, and Z. Zhu Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting. arXiv preprint arXiv:1709.04875. Cited by: Table 8. Zhang et al. (2023) H. Zhang, Z. Guo, Y. Yang, X. Liu, and D. Hu C2ST: cross-modal contextualized sequence transduction for continuous sign language recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 21053–21062. Cited by: Appendix F, §2, Table 9. Zhang et al. (2019) Z. Zhang, J. Pu, L. Zhuang, W. Zhou, and H. Li Continuous sign language recognition via reinforcement learning. In ICIP, p. 285–289. Cited by: Appendix F. Zhao et al. (2023) W. Zhao, H. Hu, W. Zhou, J. Shi, and H. Li BEST: bert pre-training for sign language recognition with coupling tokenization. In Proceedings of the AAAI conference on artificial intelligence, Vol. 37, p. 3597–3605. Cited by: Appendix F, Table 8. Zhou et al. (2023) B. Zhou, Z. Chen, A. Clapés, J. Wan, Y. Liang, S. Escalera, Z. Lei, and D. Zhang Gloss-free sign language translation: improving from visual-language pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 20871–20881. Cited by: §2. Zhou et al. (2021a) H. Zhou, W. Zhou, W. Qi, J. Pu, and H. Li Improving sign language translation with monolingual data by sign back-translation. In CVPR, p. 1316–1325. Cited by: Table 12, 5th item, Table 1, Table 9. Zhou et al. (2020) H. Zhou, W. Zhou, Y. Zhou, and H. Li Spatial-temporal multi-cue network for continuous sign language recognition.. In AAAI, Cited by: §2. Zhou et al. (2021b) H. Zhou, W. Zhou, Y. Zhou, and H. Li Spatial-temporal multi-cue network for sign language recognition and translation. TMM. Cited by: §1. Zhou et al. (2025) W. Zhou, W. Zhao, H. Hu, Z. Li, and H. Li Scaling up multimodal pre-training for sign language understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: Appendix F, §1, §2, Table 8, Table 9. Zhu et al. (2024) Q. Zhu, J. Li, F. Yuan, and Q. Gan Multiscale temporal network for continuous sign language recognition. Journal of Electronic Imaging 33 (2), p. 023059–023059. Cited by: Appendix F. Zuo and Mak (2022) R. Zuo and B. Mak C2SLR: consistency-enhanced continuous sign language recognition. In CVPR, p. 5131–5140. Cited by: Appendix F, §2. Zuo et al. (2023) R. Zuo, F. Wei, and B. Mak Natural language-assisted sign language recognition. In CVPR, p. 14890–14900. Cited by: Appendix F, §2, Table 8. Table 12: Details of datasets used in our paper. Dataset Video Samples Gloss Vocabulary Token Vocabulary Train Test Validation Train Test Validation Train Test Validation ASL Citizen (Desai et al., 2023) 40154 32941 10304 2731 2731 2731 - - - WLASL2000 (Li et al., 2020a) 14289 2878 6149 2000 1999 2000 - - - MultiVSL1000 (Dinh et al., 2025) 20161 4538 3712 999 999 999 - - - PHOENIX14T (Camgoz et al., 2018) 7096 642 519 1085 411 393 2143 976 927 CSL-Daily (Zhou et al., 2021a) 18401 1176 1077 2000 1345 1358 2342 1358 1344 How2Sign (Duarte et al., 2021) 30904 2322 1713 - - - 8816 3312 3046 CSL-OpenWorld (Ours) 86387 7038 7033 - - - 12382 9134 9153 Appendix A Impact Statement This paper presents a unified modeling framework for multiple sign language understanding tasks, with the goal of advancing the field of machine learning and sign language processing. In addition to methodological contributions, the work includes a dataset constructed from publicly available online resources, following standard research practices. Potential positive societal impacts include improved accessibility of information and communication for hard-of-hearing communities, as well as facilitating future research in SL understanding. The dataset is derived from publicly accessible content and is used solely for research purposes. No personally identifiable information is intentionally collected, and the work does not involve interaction with human subjects or user-facing deployment. Overall, we do not foresee any significant negative ethical or societal consequences arising directly from this work beyond those commonly associated with machine learning research using publicly available data. Appendix B Limitations and Discussion We outline potential directions for future improvement. Heterogeneous datasets training. While OneSign explicitly models modality-level distribution differences, it does not yet account for distributional discrepancies across heterogeneous datasets. When multiple datasets are jointly used for training, the original data distribution of a specific dataset may be diluted, which can adversely affect performance on its corresponding test set. Consequently, how to effectively balance multi-task and multi-dataset aggregation while preserving dataset-specific characteristics remains an important direction for future research. Pose estimation. Moreover, pose-based representations enable significantly more efficient training and inference, making them particularly suitable for large-scale scenarios. Considering limited GPU resources and the trade-off between training efficiency and model performance, we adopt pose data as the primary input modality instead of raw RGB video. Prior SL models, such as UniSign, CoSign, TwoStream, and MSLU, have demonstrated that pose-based inputs remain effective and robust across multiple benchmarks, suggesting that pose estimation errors do not substantially hinder overall performance in SL tasks. Nevertheless, pose estimation methods (DWPose) may still introduce noise, which can potentially affect model performance. Appendix C Experiment Dataset. We adopt seven publicly available datasets, including three ISLR datasets (ASL Citizen, WLASL2000, and MultiVSL1000), two CSLR datasets, which also serve as gloss-based SLT datasets (CSL-Daily and PHOENIX14T), and two gloss-free SLT datasets (How2Sign and our newly proposed CSL-OpenWorld). Table 12 summarizes detailed statistics of all datasets used in this paper, including training set size and vocabulary size. • WLASL2000 (Li et al., 2020a) is the largest public Word-Level ASL dataset. It contains 21,083 video clips performing 2,000 common ASL glosses, featuring 119 signers to ensure variety in signing styles and backgrounds. • ASL Citizen (Desai et al., 2023) is a large-scale, crowdsourced dataset designed for isolated sign language recognition in real-world settings. It comprises approximately 84,000 video clips of 2,731 signs performed by 52 signers, capturing significant diversity in lighting, camera angles, and backgrounds. • Multi-VSL1000 (Dinh et al., 2025) is a multi-view dataset for isolated Vietnamese Sign Language recognition, covering 1,000 sign categories. It provides synchronized observations from multiple viewpoints, enabling research on robust sign recognition under varying viewing conditions. • PHOENIX14T (Camgoz et al., 2018) is a German Sign Language (DGS) dataset annotated with both glosses and German translations. It consists of 7,096 training, 519 validation, and 642 test samples from 9 signers, with a vocabulary of 1,066 glosses and 2,877 German words. • CSL-Daily (Zhou et al., 2021a) is a Chinese Sign Language dataset focusing on daily life topics. It contains 18,401 labeled videos for training from 10 signers, providing 2,000 gloss tokens and 2,343 Chinese words for translation tasks. • How2Sign (Duarte et al., 2021) is a large-scale American Sign Language (ASL) dataset comprising over 80 hours of multi-view video and multimodal data. We utilize the RGB frontal-view videos, which include 31,128 training, 1,741 validation, and 2,322 test samples. • CSL-OpenWorld is our proposed large-scale CSL dataset, collected from diverse online platforms. It includes 86,386 training, 7,033 validation, and 7,038 test samples, featuring over 10K video instances that reflect complex real-world signing scenarios. About dataset, although MSASL is a widely used benchmark for ISLR, we were unable to include it in our experiments. During data acquisition, we found that a substantial portion of the YouTube links provided by MSASL were no longer valid or accessible, making it infeasible to reconstruct the dataset reliably. OpenASL contains samples with multiple signers in a single video, which makes it unsuitable for our pose-based modeling. Further processing is required; we plan to incorporate OpenASL results in future work. Therefore, MSASL and OpenASL is excluded from our experimental evaluation. Although our experiments involve multiple sign language datasets, they are independently collected and constructed from distinct data sources. We confirm that there is no sample- or video-level overlap across the datasets used in our experiments. Consequently, joint training on these datasets does not introduce cross-dataset leakage through duplicated samples or videos. Implementation Details. Our architecture consists of following key components. (1) Pose Extraction: We employ the open-source DWPose toolkit to extract pose features from videos, including 21 keypoints for each hand, 68 facial keypoints, and 14 body joint keypoints. (2) MA-LLM MoE: We adopt LLaMA 3.2 (1B) as the backbone of our modality-adaptive LLM with a Mixture-of-Experts (MoE) design. As the default setting, we use two pose experts with one activated expert and two text experts with one activated expert. The dataset descriptor indicates the sign language video dataset, while the task descriptor specifies the corresponding task (i.e., Isolated Sign Language Recognition, Continuous Sign Language Recognition, and Sign Language Translation). (3) Unified Paradigm: Following prior work (Gan et al., 2026d), pseudo-gloss annotations are constructed by removing punctuation, performing lemmatization, and applying word-level tokenization to the text transcripts. In all experiments, the loss weights λsign _sign and λtext _text are both set to 0.5. (4) Optimization: Our model is implemented in PyTorch 2.6 and trained on four NVIDIA RTX 4090 GPUs (24GB memory each) using half-precision training. The initial learning rate is set to 1×10−41× 10^-4 for the recognition module and 1×10−51× 10^-5 for the translation module, with a decay factor of 0.5 applied at epochs 20, 30, and 40. The batch size is set to 8. Notably, the model is trained without incorporating any additional external sign language datasets beyond those described above. Evaluation Metrics. For ISLR, to ensure fair comparison with prior work and their reported baselines, we follow the evaluation protocols used in each dataset. Specifically, we report Top-1 and Top-5 accuracy (Top@1 and Top@5) on ASL Citizen and MultiVSL1000, and per-instance (P-I) and per-class (P-C) Top-1 accuracy on WLASL2000. For CSLR, we use Word Error Rate (WER) as the evaluation metric. For SLT, we adopt ROUGE-L F1 and BLEU-1/2/3/4 to evaluate translation quality. These metrics are widely used in existing CSLR and SLT literature, enabling direct and fair comparison with prior methods. Appendix D More Ablation Study Effect of the Dataset and Task descriptor. Dataset Task Dev TEST WER Rouge BLEU1 BLEU4 WER Rouge BLEU1 BLEU4 20.1 51.58 52.12 26.01 20.2 51.59 52.01 25.64 ✓ 20.0 52.24 53.14 26.76 20.2 51.87 53.16 26.31 ✓ 20.0 52.13 53.35 26.65 20.1 51.64 53.07 26.14 ✓ ✓ 20.0 52.34 53.41 26.98 20.1 52.01 53.36 26.45 Table 13: Effect of Dataset and Task descriptor. Table 14: Phoenix14T results with λ in ℒbalL_bal . λsign=λtext _sign= _text 0.3 0.5 0.7 0.9 1 WER↓ 22.1 20.1 22.3 23.4 25.1 BLUE4 ↑ 25.75 26.45 25.74 25.15 24.78 As illustrated in Fig. 5, we introduce dataset and task descriptors in the MA-MoE module to provide auxiliary conditioning during both training and inference. These descriptors are not used to create separate branches, but rather to guide the model toward the corresponding data distributions in a unified manner. To evaluate their impact, we conduct an ablation study by selectively removing the dataset and/or task descriptors. The results are summarized in Table 13. We observe that removing either descriptor individually leads to only marginal performance degradation, indicating that the model remains robust and effective even without explicit conditioning signals. Nevertheless, incorporating both descriptors yields the best overall performance, suggesting that they provide complementary information that helps the model better capture dataset-specific and task-specific variations. Importantly, these results demonstrate that our framework does not rely on explicit descriptors to maintain unified modeling capability, while still benefiting from them when available. Hyperparameter Analysis of the Load Balancing Coefficient. We further investigate the sensitivity of the load-balancing coefficient λ in Eq. 13. As shown in Table 14, the best performance is achieved when λ=0.5λ=0.5, yielding the lowest WER (21.8) and the highest BLEU-4 (26.45). When λ is relatively small (e.g., 0.3), the load-balancing constraint is insufficient to effectively prevent expert imbalance, leading to slightly inferior performance. As λ increases beyond 0.5, both WER and BLEU-4 gradually deteriorate. This suggests that an excessively strong balancing constraint forces the router to distribute tokens more uniformly across experts, limiting expert specialization and reducing the routing flexibility required for modeling modality-specific patterns. Overall, these results indicate that a moderate load-balancing coefficient provides the best trade-off between balanced expert utilization and expert specialization. Comparison of different expert designs Model Parameter(M) Dev TEST Total Activated WER Rouge BLEU1 BLEU4 WER Rouge BLEU1 BLEU4 Dense FFN 2041.1 2041.1 22.6 49.63 50.23 23.68 22.8 49.56 49.32 23.67 Sparse MOE 4457.0 2041.1 22.4 49.23 49.36 23.15 22.6 48.65 49.36 22.64 shared+modality-specific FFN 2041.1 2041.1 21.4 51.69 51.60 25.68 21.5 51.52 51.98 25.79 MA-MOE 4457.0 2041.1 20.0 52.34 53.41 26.98 20.1 52.01 53.36 26.45 Table 15: Comparison of different experts. To validate the effectiveness of the proposed sparse MA-MoE, we investigate whether the performance gains mainly stem from the MoE routing mechanism or simply from increased parameter capacity. Specifically, we compare MA-MoE with two alternative settings: (1) Shared + modality-specific FFN, which consists of a shared expert along with one sign expert and one text expert, but without sparse routing; (2) Dense FFN, where three experts are always activated without modality-specific specialization. (3) Sparse MoE, which consists of five experts with Top-k=3k=3 activated for each token, but without modality-specific specialization. As shown in Table 15, under comparable parameter scales, the Dense FFN achieves the worst performance. This indicates that merely increasing the number of parameters does not lead to performance improvement. In contrast, the proposed sparse modality-specific experts effectively capture modality heterogeneity, demonstrating that the performance gains primarily come from the adaptive routing mechanism rather than parameter scaling. Model Test +MA-MOE TEST WER Rouge BLEU1 BLEU4 WER Rouge BLEU1 BLEU4 Llama 3.2 1B 22.5 48.99 49.72 23.01 → 20.1 52.01 53.36 26.45 Gemma3 1b 22.9 49.96 48.04 22.85 → 20.2 51.32 52.66 26.35 Qwen2.5 1.5B 23.7 49.06 48.81 22.93 → 20.3 51.17 52.69 26.34 Table 16: Effect of MA-MOE on different LLMs. Effect of MA-MoE on different LLM models. We further evaluate the generalization capability of MA-MoE across different backbone LLMs. Specifically, we conduct experiments on several representative open-source models, including LLaMA3-1B (our baseline), Gemma-3-1B, and Qwen-2.5-1.5B. As shown in Table 16, MA-MoE consistently improves performance across all evaluated models. These results demonstrate that our approach is model-agnostic and can be seamlessly integrated into a variety of LLM architectures. Moreover, the consistent gains across different backbones indicate that MA-MoE effectively addresses modality heterogeneity in a generalizable manner, rather than relying on model-specific inductive biases, leading to stable and significant improvements. Ns=NtN_s=N_t Top-k Params (M) Activated (M) Rouge BLEU1 BLEU4 Rouge BLEU1 BLEU4 0 0 1235.8 1235.8 49.45 49.36 22.67 48.99 49.72 23.01 2 1 4457.0 2041.1 52.34 53.41 26.98 52.01 53.36 26.45 4 1 7678.2 2041.1 52.74 53.65 26.78 52.14 53.24 26.53 4 2 7678.2 2846.4 52.99 53.94 26.97 52.45 53.47 26.56 Table 17: Impact of activated expert count. Full ablation on the Impact of Activated Expert Count. Due to space limitations in the main paper, we only reported the ablation results on the DEV set in Table 5 of main paper. Here, we provide the complete results on both DEV and TEST sets in Table 17. Appendix E Qualitative Analysis example(a) Phoenix14T dataset Groundtruth HEUTE NACHT REGEN DOCH MITTE NORD (TODAY NIGHT RAIN HOWEVER MIDDLE NORTH) OneSign HEUTE NACHT REGEN DOCH MITTE NORD TODAY NIGHT RAIN HOWEVER MIDDLE NORTH W/O MA-MOE HEUTE NACHT NEBEL DOCH MITTE NORD ( TODAY NIGHT FOG HOWEVER MIDDLE NORTH ) example(b) Phoenix14T dataset Groundtruth TAG OST VIER FLUSS SIEBEN GRAD (DAY EAST FOUR RIVER SEVEN DEGREE) OneSign TAG DANN VIER FLUSS SIEBEN GRAD NORD (DAY THEN FOUR RIVER SEVEN DEGREE NORTH) W/O MA-MOE TAG DANN VIER FLUSS SIEBEN ( DAY THEN FOUR RIVER SEVEN) example(c) CSL-daily dataset Groundtruth 去 学校 我 踢足球 (Go School I Play soccer) OneSign 去 学校 我 踢足球 Go School I Play soccer W/O MA-MOE 来 学校 我 踢足球 Come School I Play soccer example(d) CSL-daily dataset Groundtruth 我 房子 在 看 电视 你 看 电视 喜欢 (I House At Watch TV You Watch TV Like) OneSign 我 房子 看 电视 你 看 电视 喜欢 I House Watch TV You Watch TV Like W/O MA-MOE 我 房子 在 看 电视 我 看 电视 I House At Watch TV I Watch TV Table 18: CSLR Qualitative results on Phoenix-2014T and CSL-Daily. example(a) Phoenix14T dataset Groundtruth und nun die wettervorhersage für morgen sonntag den zwölften juli (And now the weather forecast for tomorrow, Sunday, July 11th) OneSign und nun die wettervorhersage für morgen sonntag den zwölften juli (And now the weather forecast for tomorrow, Sunday, July 11th) W/O MA-MOE und nun die wettervorhersage für morgen sonntag den elften juli (And now the weather forecast for tomorrow, Sunday, July 12th) example(b) How2Sign dataset Groundtruth So, we’ve got to find a way to get to the exit. OneSign we’ve got to find a way to get to the exit. W/O MA-MOE we have found a way to get to the exit. example(b) CSL-daily dataset Groundtruth 人生旅途, 多一个人相伴, 是一种幸福。 (On the journey of life, having someone by your side is a form of happiness.) OneSign 人生旅途, 多个人相伴, 是一种幸福。 On the journey of life, having others by your side is a form of happiness. W/O MA-MOE 人生旅途, 相伴是一种幸福。 On the journey of life, companionship is a blessing. example(d) CSL-OpenWorld dataset Groundtruth 文化遗产是全人类的共同财富。 (Cultural heritage is the common wealth of all humanity.) OneSign 文化遗产是人类的共同财富。 Cultural heritage is the common wealth of humanity. W/O MA-MOE 文化是全人类的共同财富。 Culture is the common wealth of all humanity. Table 19: SLT Qualitative results on Phoenix-2014T, How2Sign , CSL-Daily and CSL-OpenWorld. As shown in Table E and Table E, We provide qualitative analyses for both recognition and translation tasks. Specifically, we present recognition results on PHOENIX-2014T and CSL-Daily, and translation results on PHOENIX-2014T, How2Sign, CSL-Daily, and CSL-OpenWorld. As observed, the model without MA-MoE already achieves reasonable recognition and translation performance. With MA-MoE, OneSign consistently produces more accurate and semantically coherent outputs across datasets, further demonstrating the effectiveness of our approach. Appendix F Dataset Visualization We provide visualizations of the formatted pose data used in our experiments across seven datasets. As shown in Figures 6 and 7, the pose representations exhibit noticeable variations in skeleton structure, motion patterns, and temporal dynamics across different datasets. In addition, we present representative video samples from the proposed CSL-OpenWorld dataset, as illustrated in Figure 8, to provide an intuitive understanding of its visual diversity and real-world complexity. Figure 6: Visualization of used datasets. Figure 7: Visualization of used datasets. We blurred the facial areas for privacy. Figure 8: More visualizations of the proposed CSL-OpenWorld dataset, illustrating its diversity in backgrounds, signers, and topics. We blurred the facial areas for privacy. 52, 34, 35, 71, 87, 29, 52, 92, 95, 48, 7774, 10, 94, 62, 32, 57, 86, 68, 78, 58, 85, 37, 36, 93, 75, 38, 61, 4169, 13, 13, 47, 12, 28, 81, 72, 65, 1, 31, 5, 44