Paper deep dive
Bridging the Gap Between Semantics and Reconstruction:Unifying Sign Language Translation and Production
Xiao Liu, Shiwei Gan, Yafeng Yin, Jiaxin Yin, Bowen Guo, Yaqi Sun, Zhiwei Jiang, Lei Xie
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/12/2026, 3:17:40 AM
Summary
The paper introduces Uni-SLTP, a unified framework for Sign Language Translation (SLT) and Sign Language Production (SLP). It addresses the 'Semantic-Reconstruction Gap' by proposing a shared sign tokenizer (SR-RVQ) that captures both semantic alignment and motion details, and a unified autoregressive model that performs bidirectional conditional sequence generation between sign and text modalities.
Entities (6)
Relation Signals (5)
Uni-SLTP → implements → Sign Language Translation
confidence 95% · Uni-SLTP is a unified framework for SLT and SLP
Uni-SLTP → implements → Sign Language Production
confidence 95% · Uni-SLTP is a unified framework for SLT and SLP
Uni-SLTP → uses → SR-RVQ
confidence 95% · we propose Uni-SLTP... with... a shared sign tokenizer... SR-RVQ
SR-RVQ → bridges → Semantic-Reconstruction Gap
confidence 90% · SR-RVQ... bridging the Semantic–Reconstruction Gap
Uni-SLTP → unifies → Sign Language Understanding
confidence 85% · can sign language understanding and production be unified within a single framework?
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advances in sign language (SL) research have shown a trend toward unifying multiple sign language understanding (SLU) subtasks, such as isolated sign language recognition (ISLR), continuous sign language recognition (CSLR), and sign language translation (SLT), within a single framework, leading to substantial progress. Meanwhile, sign language production (SLP), which generates sign sequences from text, has also attracted growing attention. This naturally raises an important question: can sign language understanding and production be unified within a single framework? Compared with unifying SLU subtasks, this problem is substantially more challenging. Existing SLU tasks largely share the same direction of mapping, namely from sign inputs to linguistic outputs, whereas SLT and SLP lie in opposite directions of sign-text mapping. A unified framework must therefore address two key challenges: (1) bridging the modality gap between continuous sign motions and discrete text tokens through a shared sign tokenizer that supports both linguistic abstraction and motion reconstruction; and (2) learning a single conditional autoregressive model that can take either sign or text as input and generate the corresponding target sequence in the opposite modality. To this end, we propose Uni-SLTP, a unified framework for SLT and SLP with two key components: (1) a shared sign tokenizer that converts sign sequences into discrete tokens and latent representations, capturing both semantic and reconstructive information; and (2) a unified autoregressive generation model that formulates both tasks as conditional sequence generation. Experiments on widely used public datasets show that Uni-SLTP achieves superior motion accuracy for SLP while maintaining competitive SLT performance.
Tags
Links
- Source: https://arxiv.org/abs/2608.09045v1
- Canonical: https://arxiv.org/abs/2608.09045v1
Trouble viewing inline? Open PDF directly →
Full Text
61,256 characters extracted from source content.
Expand or collapse full text
Bridging the Gap Between Semantics and Reconstruction: Unifying Sign Language Translation and Production Xiao Liu State Key Laboratory of Novel Software Technology, Nanjing University Suzhou, Jiangsu, China liuxiaox@smail.nju.edu.cn Shiwei Gan ∗ State Key Laboratory of Novel Software Technology, Nanjing University Nanjing, Jiangsu, China sw@nju.edu.cn Yafeng Yin ∗ State Key Laboratory of Novel Software Technology, Nanjing University Suzhou, Jiangsu, China yafeng@nju.edu.cn Jiaxin Yin State Key Laboratory of Novel Software Technology, Nanjing University Suzhou, Jiangsu, China jiaxin.yin@smail.nju.edu.cn Bowen Guo State Key Laboratory of Novel Software Technology, Nanjing University Suzhou, Jiangsu, China bowen@smail.nju.edu.cn Yaqi Sun State Key Laboratory of Novel Software Technology, Nanjing University Suzhou, Jiangsu, China yaqi@smail.nju.edu.cn Zhiwei Jiang State Key Laboratory of Novel Software Technology, Nanjing University Suzhou, Jiangsu, China jzw@nju.edu.cn Lei Xie State Key Laboratory of Novel Software Technology, Nanjing University Nanjing, Jiangsu, China lxie@nju.edu.cn Abstract Recent advances in sign language (SL) research have shown a trend toward unifying multiple sign language understanding (SLU) sub- tasks, such as isolated sign language recognition (ISLR), continuous sign language recognition (CSLR), and sign language translation (SLT), within a single framework, leading to substantial progress. Meanwhile, sign language production (SLP), which generates sign sequences from text, has also attracted growing attention. This naturally raises an important question: can sign language under- standing and production be unified within a single framework? Compared with unifying SLU subtasks, this problem is substan- tially more challenging. Existing SLU tasks largely share the same direction of mapping, namely from sign inputs to linguistic out- puts, whereas SLT and SLP lie in opposite directions of sign-text mapping. A unified framework must therefore address two key challenges: (1) bridging the modality gap between continuous sign motions and discrete text tokens through a shared sign tokenizer that supports both linguistic abstraction and motion reconstruc- tion; and (2) learning a single conditional autoregressive model that can take either sign or text as input and generate the corre- sponding target sequence in the opposite modality. To this end, we propose Uni-SLTP, a unified framework for SLT and SLP with two key components: (1) a shared sign tokenizer that converts sign sequences into discrete tokens and latent representations, capturing both semantic and reconstructive information; and (2) a unified autoregressive generation model that formulates both tasks as con- ditional sequence generation. Experiments on widely used public datasets show that Uni-SLTP achieves superior motion accuracy for SLP while maintaining competitive SLT performance. ∗ Corresponding authors. This work is licensed under a Creative Commons Attribution-NonCommercial- NoDerivatives 4.0 International License. M ’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2213-4/2026/11 https://doi.org/10.1145/3767308.3835086 CCS Concepts • Computing methodologies→Computer vision; Natural language processing. Keywords Sign Language Translation, Sign Language Production ACM Reference Format: Xiao Liu, Shiwei Gan, Yafeng Yin, Jiaxin Yin, Bowen Guo, Yaqi Sun, Zhi- wei Jiang, and Lei Xie. 2026. Bridging the Gap Between Semantics and Reconstruction: Unifying Sign Language Translation and Production. In Proceedings of the 34th ACM International Conference on Multimedia (M ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3767308.3835086 1 Introduction Sign language (SL) is the primary means of communication for the hearing-impaired community. Existing research primarily focuses on two directions: sign language understanding (SLU) [36,89], with sign language translation (SLT) receiving the most attention [16, 17,37,72,79] and sign language production (SLP) [1,57,60,74,87]. SLT aims to convert SL sequence into natural language text, making it easier for the general public to understand, while SLP seeks to translate natural language text into SL sequences, enabling the hard of hearing individuals to better access and understand information. Considering that the two tasks involve fundamentally different modality transformations, SLT maps continuous sign sequences to discrete text tokens, whereas SLP converts discrete text sequences into continuous, natural sign sequences, existing research typically models them with separate architectures. As shown in Figure 1 (a), for SLT, the de facto architecture typically begins with a visual backbone that extracts sign semantic features across both spatial and temporal dimensions, followed by a translation model (e.g., mBART [40] or GPT-2 [10]), to generate the corresponding text sentence. For SLP tasks shown in Figure 1 (b), current methods mainly follow two paradigms: (1) Diffusion-based approaches [1, 48]: A text-conditioned diffusion model generates continuous or discrete latent features learned by a VAE or VQ-VAE, which are arXiv:2608.09045v1 [cs.CL] 10 Aug 2026 M ’26, November 10–14, 2026, Rio de Janeiro, BrazilXiao Liu et al. (a) SL Translation Visual Backbone Translation Module Text Sign Sequence (b) SL Production Generative Module VAE (or VQ-VAE) Sign sequence Text (C) Ours: SL Translation and Production Sign SequenceText Tokenizer Autoregressive Generative Model Tokenizer Text Sign Sequence Figure 1: Previous task-specific SLT/SLP models vs Uni-SLTP. then decoded into target sign sequences. (2) Autoregressive (AR)- based methods [41,73,87]: A VQ-VAE converts the sign sequence into discrete IDs, allowing SLP to be modeled as text-conditioned sequence prediction. An AR model predicts these IDs, which the VQ-VAE decodes into the target sign sequence. Meanwhile, recent advances in SL understanding (SLU), such as Unisign [36], BEST [78], MSLU [84], have explored unifying multi- ple SLU tasks, including isolated SL recognition (ISLR) [23,35,89], continuous SL recognition (CSLR) [13,26,67], and SLT, within a sin- gle framework. These models significantly reduce individual-task modeling complexity through a unified paradigm. Beyond SLU, a more fundamental question arises: Can SLU (e.g., SLT) and SLP be unified within a single framework to enable shared to- ken representations and bidirectional generation? Our answer is that unifying these tasks is feasible, but not straightforward. (1) For unified SLU models [78,84], although different understanding tasks may have different output forms, they generally share the same mapping direction, i.e., from sign inputs to linguistic outputs such as glosses or text sequences. (2) In the unified SLU settings, the intermediate sign representations do not introduce conflicting objectives. All subtasks benefit from learning stronger semantic representations of SL, making the optimization of the shared sign backbone relatively straightforward. Unlike previous unified SLU models, unifying SLT and SLP is more challenging, considering: (1) The two tasks differ funda- mentally in their input and output modalities. Specifically, SLT maps sign sequences to text, whereas SLP maps text to sign se- quences, making it non-trivial to formulate a single unified task for joint modeling. A possible approach is to introduce a sign tokenizer using VQ-VAE (referred to as a sign tokenizer), which converts sign sequences into discrete intermediate representations. The relation- ship between these discrete sign representations and discrete text tokens can then be modeled in a bidirectional autoregressive man- ner; (2) However, simply adopting the same sign tokenizer to simultaneously capture low-level motion details required for SLP and high-level semantic representations required for SLT is inherently difficult. SLT and SLP impose conflicting requirements on sign representations: translation benefits from semantic abstraction and alignment with text, while production demands fine-grained motion accuracy to enable natural and accu- rate synthesis. We term this tension the Semantic–Reconstruction Gap, which makes it hard to learn discrete sign tokens that are both semantically aligned and highly decodable for generation. In this paper, we revisit SLT and SLP and aim to develop a general unified framework supporting both tasks with two main objectives. (1) A unified autoregressive model that operates on both sign and text representations, and can flexibly generate either text or sign token sequences, thereby enabling joint modeling of SLT and SLP within a single framework. (2) A unified sign tokenizer that provides effective intermediate representations of sign language, capturing both fine-grained motion details for accurate reconstruction and high-level semantic information for sign understanding. To achieve these goals, we propose Uni-SLTP, a unified frame- work that supports bidirectional mapping between sign and text within a single architecture. Specifically, to build a shared sign tokenizer that serves both SLT and SLP tasks, providing discrete representations that capture both fine-grained motion details for reconstruction and high-level semantic features for understanding, we introduce Semantic-Reconstruction guided Residual Vec- tor Quantization (SR-RVQ) as a shared discrete interface that decouples semantic alignment from detail reconstruction. To allow modeling SLT and SLP in the same framework, we cast both SLT and SLP as conditional next-token prediction in a unified pipeline via proposed SR-RVQ and AR modeling, enabling bidirectional learning with competitive semantic consistency or motion accuracy. •To the best of our knowledge, we are the first to formu- late SLT and SLP as bidirectional sign–text generation. We propose Uni-SLTP, a unified framework that casts both directions as conditional next-token prediction in an autore- gressive pipeline, enabling one model to perform both tasks. •We introduce Semantic-Reconstruction guided Residual Vector Quantization (SR-RVQ), a hierarchical tokenizer that decouples semantic alignment from motion-detail recon- struction via a semantic anchor token and residual motion tokens, bridging the Semantic–Reconstruction Gap. •Extensive experiments show that Uni-SLTP improves SLP quality while maintaining competitive SLT performance against task-specific baselines. 2 Related Work Sign Language Translation. SLT aims to translate sign language (SL) sequences into textual sentences. Existing approaches typi- cally employ a sign tokenizer (i.e., a visual backbone) to encode SL sequences into intermediate sign representations, which are then fed into a pretrained translation model (typically an autoregressive (AR) language model like mBART [6], GPT-2 [16]) to generate the target text. Despite the strong language modeling capabilities of language models (LMs), recent studies indicate that the primary performance bottleneck of SLT models lies in extracting effective and semantically accurate sign representations. Consequently, most prior work adopts a two-stage training paradigm and proposes var- ious strategies to pretrain sign tokenizers, including CTC-based gloss supervision [5,81], contrastive learning with text [30,37], pseudo-gloss supervision [15,20], and large-scale SL dataset pre- training [18,36,78]. Such pretraining strategies enable the sign tokenizer to learn more effective and sign-specific semantic repre- sentations, which in turn provide high-quality inputs to the transla- tion module, and are crucial for ensuring translation performance. Bridging the Gap Between Semantics and Reconstruction: Unifying Sign Language Translation and ProductionMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Sign Language Production. Sign language production (SLP) [39, 50–52, 54, 60, 69, 88] aims to generate sign sequences conditioned on spoken-language text. To avoid the high dimensionality and redundancy of raw videos, recent methods increasingly adopt pose sequences or parametric body trajectories as the generation tar- get [11,55]. Existing SLP approaches can be broadly grouped by whether they rely on gloss as an intermediate supervision signal. Gloss-based pipelines [56,57,60,74] leverage glosses to facilitate learning and enforce monotonic alignment, but they depend on costly and unevenly available gloss annotations. This has moti- vated a growing line of gloss-free, end-to-end text-to-pose gen- eration [1,41,73], where duration and temporal alignment are modeled either explicitly or implicitly within sequence generators. Early end-to-end text-to-pose methods largely adopt continuous regression-based encoder–decoder generators [51,53]. More re- cently, two paradigms have attracted attention: diffusion-based methods that synthesize pose sequences via iterative denoising under text conditioning [1], and discrete autoregressive approaches that tokenize poses and perform next-token prediction, enabling efficient decoding and easier integration with language models [87]. SLT and SLP have advanced with task-specific architectures, while SLU has begun to unify sign understanding objectives. Yet SLT and SLP remain largely separated, as they map between con- tinuous motion and discrete text in opposite directions, demanding language-aligned and faithfully decodable representations. By dis- cretizing signs into tokens, tokenizers provide a shared interface that narrows the modality gap between sign and text. This makes it possible to cast both directions as conditional next-token prediction under a single autoregressive framework. The remaining challenge is to learn a token space that reconciles semantic alignment for SLT with fine-grained reconstructability for SLP. 3 Method 3.1 Preliminaries We first define the key modules used in our framework: (1) Sign Tokenizer (ST): a sign encoderST 푒 that maps SL inputs into discrete IDs and embeddings, and a sign decoderST 푑 that re- constructs SL sequences from token IDs. (2) Text Tokenizer (T): a text encoderTT 푒 that maps text inputs into token IDs and embeddings 1 , and a text decoderTT 푑 that con- verts token IDs back into text sequences. (3) Autoregressive Module (AR): modeling the conditional gener- ation of target token IDs in an AR manner. SLT Task Formulation. SLT is formulated as a conditional se- quence generation task, aiming to generate a target text sentence W from a continuous SL sequence S. In practice, an SLT model first employs a sign tokenizer to map the input sequence into latent rep- resentations:E s = ST 푒 (S), and then uses the autoregressive module ARto generate the target token sequence W 푖푑 conditioned onE s . The conditional distribution is factorized as 푝 휃 (W 푖푑 | E s )= 푈 Ö 푢=1 푝 휃 푤 푖푑 푢 | W 푖푑 <푢 , E s ,(1) 1 Standard text tokenizers typically do not include embedding functions; we include them here for consistency and clarity. where W 푖푑 <푢 denotes the previously generated tokens. Finally, the text decoder reconstructs the output sentence from the token IDs: W= T 푑 (W 푖푑 ). SLP Task Formulation. SLP is commonly studied under either diffusion-based or autoregressive (AR)-based paradigms. To enable a unified formulation of SLP and SLT, we focus on the AR-based approach. In this framework, an SLP model first employs a text tokenizer to encode a textual input sequence into latent representa- tions: E 푡 = T 푒 (W), and then usesARmodule to generate discrete SL tokens, where the conditional distribution is factorized as 푝 휃 (S 푖푑 | E 푡 )= 푈 Ö 푢=1 푝 휃 푠 푖푑 푢 | S 푖푑 <푢 , E 푡 ,(2) Finally, theST 푑 reconstructs the SL sequence from the predicted token IDs: S= ST 푑 (S 푖푑 ). Unified Formulation of SLT and SLP.. From a unified perspective, both SLT and SLP can be formulated as AR sequence modeling problems over discrete tokens. Specifically, given an input sequence 푋and a target sequence푌, a source tokenizer푇=(푇 푒 ,푇 푑 )encodes 푋into source embeddings E=푇 푒 (푋), and a target tokenizer퐷= (퐷 푒 ,퐷 푑 ) encodes푌into target token IDs Y 푖푑 = 퐷 푒 (푌). The model is trained to learn the conditional distribution 푝 휃 (Y 푖푑 | E)= 푈 Ö 푢=1 푝 휃 (푦 푖푑 푢 | Y 푖푑 <푢 , E),(3) During inference, the AR predicted token sequence Y 푖푑 is decoded into the output sequence Y using the target tokenizer: Y= 퐷 푑 (Y 푖푑 ) . Under this unified formulation, SLT and SLP thus share a single modeling paradigm, with the input and output swapped between the two directions. •SLT: The input푋is a sign sequence S, and the output푌is a text sentence W. We use푇= STto encode S into source embeddings, and 퐷= T to decode predicted IDs into W. •SLP: The input푋is a text sentence W, and the output푌is a sign sequence S. We use푇= TTto encode W into source embeddings, and 퐷= ST to decode predicted IDs into S. In both SLT and SLP, we use the pretrained LM’s built-in text tokenizerTTfor text tokenization and decoding. To unify the two directions, the remaining challenge is to (i) learn a discrete sign tokenization that converts continuous sign into a sequence of pose token IDs, and (i) train a single AR backboneARto model both text tokens and pose tokens under a shared vocabulary. Next, we introduce our sign tokenizer and describe the unified AR modeling. 3.2 Semantic-Reconstruction Guided Sign Tokenizer We introduce the Semantic-Reconstruction Guided Residual Vector Quantizer (SR-RVQ), a pose tokenizer that discretizes continuous sign motion into pose tokens for unified AR modeling, as shown in Figure 2. Our goal is to learn a discrete pose representation that simultaneously (i) aligns with text for SLT and (i) preserves motion details for accurate reconstruction. This calls for tokens that are both text-aligned and reconstruction-accurate. However, jointly enforcing semantic alignment and reconstruction with a single discrete bottleneck is inherently unstable, as the two objectives M ’26, November 10–14, 2026, Rio de Janeiro, BrazilXiao Liu et al. Semantic head Pose Decoder 퓛 푪 퓛 푹 Pose Encoder VQ 1 VQ2 VQ N ... + + + Pose Features 퓛 푪 Text: It will snow tonight. Semantic Features Reconstruction Features Pose Sequence - - + Residual update - Accumulation Glosses: tonight snow SR-RVQ Figure 2: SR-RVQ tokenizer pretraining. A pose encoder maps sign pose sequences to latent features, which are quantized by SR-RVQ into semantic tokens푄 1 and residual detail tokens푄 2:푁 .푄 1 serves as a semantic anchor trained with gloss CTC and semantic contrastive supervision, while 푄 2:푁 are optimized with reconstruction loss to capture fine-grained motion details. may conflict during optimization [49]. To satisfy both requirements, we adopt an푁-stage residual vector quantizer (RVQ) [34] to build a coarse-to-fine hierarchy: the first stage produces an alignment- oriented semantic stream, while later stages encode residual motion details to progressively refine reconstruction. Pose Encoder. Inspired by previous SLT models [15], our pose encoder adopts a similar design. The inputs are processed by a ResNet1D to extract spatial features, which are then processed by a temporal convolution module to capture short-term tempo- ral dependencies and perform temporal downsampling. Finally, a BiLSTM is applied to model long-term temporal relationships. Specifically, given a SL pose sequence S∈R 푇×퐷 , the encoder out- puts continuous latent features Z=E(S) ∈R 푇 ′ ×푑 , where푑is the latent dimension and푇 ′ = ⌊푇/푠⌋is obtained by a fixed temporal downsampling factor푠. We intentionally bias the encoder toward semantics, since fine-grained motion details can be recovered by subsequent residual stages and the pose decoder ST 푑 . SR-RVQ.. We discretize latents with an푁-stage RVQ to obtain a coarse-to-fine hierarchy of token streams, where푁is the number of quantization stages. Given latent features Z=z 푡 푇 ′ 푡=1 , SR-RVQ produces푁index streamsQ 푛 푁 푛=1 with Q 푛 =푞 푛,푡 푇 ′ 푡=1 . We main- tain stage-wise token dictionariesC 푛 =c (푛) 푘 퐾 푛 푘=1 ⊂R 푑 , where퐾 푛 is the size of the푛-th dictionary. At each timestep푡, we initialize the residual r 0,푡 = z 푡 and iteratively quantize: 푞 푛,푡 = arg min 푘∈1,...,퐾 푛 r 푛−1,푡 − c (푛) 푘 2 2 , e (푛) 푡 = c (푛) 푞 푛,푡 , r 푛,푡 = r 푛−1,푡 − e (푛) 푡 , (4) where r 푛,푡 denotes the stage-푛residual after subtracting the first푛 token embeddings e (푛) 푡 . The quantized embedding is accumulated as ˆ z 푡 = Í 푁 푛=1 e (푛) 푡 (thus ˆ Z= ˆ z 푡 푇 ′ 푡=1 ). In training, we assign semantic supervision primarily to the first stream Q 1 , while the remaining streams Q 2:푁 to recover residual motion details for reconstruction. Pose Decoder. The pose decoder reconstructs the pose sequence from the quantized embeddings ˆ Zto produce the reconstructed sequence ˆ S= ST 푑 ( ˆ Z). It begins with a 1D convolutional layer to process the latent embeddings, followed by two upsampling stages to recover the temporal details. Finally, a Conv1D projection layer maps the features back to the pose parameter space, producing the reconstructed pose sequence ˆ S∈R 푇×퐷 . SR-RVQ Optimization Objectives. We train the tokenizer with the principle: let the coarse stream capture high-level semantics, and let the residual streams fill in motion details. This design leads to a two-part training objective: we (i) impose semantic supervi- sion on the first-stage quantized features to encourage semantic alignment, and (i) optimize reconstruction using the full quantized embeddings so that residual stages recover fine-grained motion details. (1) Semantic-side. We apply semantic supervision only to the first-stage quantized features ˆ Z (1) = ˆ z (1) 푡 푇 ′ 푡=1 with ˆ z (1) 푡 =e (1) 푡 selected by Q 1 . A lightweight CTC head on ˆ Z (1) predicts the gloss sequence g, yieldingL ctc =− log푝(g| ˆ Z (1) ). We further align pose semantics with the paired text sentence y using an InfoNCE loss. We compute a pooled pose embedding h 푠 = Pool( ˆ Z (1) )and a pooled text embedding h 푡 = Pool(T 푒 (y)), and use in-batch negatives: L con =− 1 퐵 퐵 ∑︁ 푖=1 log exp(sim(h (푖) 푠 , h (푖) 푡 )/휏) Í 퐵 푗=1 exp(sim(h (푖) 푠 , h (푗) 푡 )/휏) ,(5) wheresimis cosine similarity and휏is a temperature. (2) Reconstruction- side. To recover motion details while keeping the first-stage codes semantics-oriented, we let the residual stages (푄 2:푁 ) absorb most re- construction pressure. We optimize pose reconstruction withL rec = ∥S− ˆ S∥ 1 +휆 푣 ∥ΔS−Δ ˆ S∥ 1 , whereΔdenotes the first-order temporal difference to encourage temporally coherent dynamics. Concretely, we form the reconstruction latent as ˆ z rec 푡 = sg[e (1) 푡 ]+ Í 푁 푛=2 e (푛) 푡 , so that gradients fromL rec do not update the stage-1 tokens, while residual stages learn to encode fine-grained motion variations. Fol- lowing standard vector quantization, we use the straight-through estimator and the commitment lossL vq to learn the codebooks. The overall tokenizer objective is L tok = 휆 ctc L ctc + 휆 con L con + 휆 rec L rec + 휆 vq L vq .(6) Bridging the Gap Between Semantics and Reconstruction: Unifying Sign Language Translation and ProductionMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Framework Semantic head SR-RVQ Pose Decoder 퓛 푪 퓛 푹 Pose Encoder VQ 1 VQ2 VQ N ... + + + Pose Features TONIGHT SNOW (Gloss) (Text) 퓛 푪 It will snow tonight. Semantic Features Reconstruction Features (Pose Sequence) Auto-regressive Model SR-RVQ Tokenizer · Text tokens · What a nice day! It will snow tonight. SR-RVQ Tokenizer Lines for SLT Trainable weights Frozen weights - - + Residual update - Accumulation Lines for SLP Pose Tokens Text: It will snow tonight. Figure 3: Uni-SLTP: unified autoregressive modeling for SLT and SLP. SR-RVQ converts pose sequences into discrete pose tokens, enabling a single AR model to perform conditional next-token prediction in both directions: pose-to-text for SLT and text-to-pose for SLP. 3.3 Uni-SLTP: Unified Autoregressive Framework Uni-SLTP, as shown in Figure 3, aims to support both SLT and SLP with a single LM. The key is to discretize continuous poses into SR-RVQ tokens and model text and pose tokens in one shared token space, so that both directions can be trained as conditional next-token prediction. Unified vocabulary. We use the pretrained LM’s built-in text tokenizer for spoken language, yielding text token IDs from the original vocabularyV 푡 . For poses, the frozen SR-RVQ tokenizer (Sec. 3.2) maps a pose sequence to stage-wise indices푞 푛,푡 . To integrate pose tokens into the LM, we augment its vocabulary with a stage-aware pose vocabularyV 푝 . For each RVQ stage푛, we allocate a disjoint token blockV (푛) 푝 of size퐾 푛 , together with boundary tokens such as<sos>and<eos>. The unified vocabulary isV=V 푡 ∪V 푝 . Accordingly, we extend both the LM embedding and output layers, where the parameters associated withV 푡 are inherited from the pretrained LM, while those associated withV 푝 are newly initialized and learned. During decoding, we mask the output space to the valid subset: text decoding usesV 푡 , while pose decoding at stage 푛 usesV (푛) 푝 . Training and inference. Uni-SLTP is trained in two stages. In Stage 1, we pretrain the sign tokenizerSTwithL tok (Eq. 6) and then freezeST 푒 andST 푑 . In Stage 2, we fine-tune the pretrained seq2seq LM as the autoregressive moduleARunder the unified vocabulary V. Rather than directly mixing SLT and SLP from scratch, we adopt a progressive training schedule: we first optimize the model on SLP, then on SLT, and finally perform joint training on both tasks. SLT. In training, given pose–text pairs(S,W), we tokenize the pose input with the frozenST 푒 and use only the semantic-stage IDs Q 1 as the source sequence and the text tokens W 푖푑 as the target. We train with standard teacher-forced negative log-likelihood: L SLT =− 푈 ∑︁ 푢=1 log푝 휃 푤 푖푑 푢 | W 푖푑 <푢 , Q 1 .(7) At inference,ARautoregressively generates tokens conditioned on Q 1 , and the LM decoder converts predicted IDs back to the sentence. SLP. In training, given text–pose pairs(W,S), we encode text with the LM and tokenize the target pose with the frozenST 푒 to obtain RVQ indicesQ 푛 푁 푛=1 . We serialize the stage-wise indices into a single pose-token sequence P 푖푑 =[<sos>,푞 1,1 , . . .,푞 푁,1 , . . ., 푞 1,푇 ′ , . . .,푞 푁,푇 ′ , <eos>] , where each푞 푛,푡 is mapped to its stage-specific token ID inV (푛) 푝 . We optimize the pose-token likelihood with: L SLP =− 퐿 ∑︁ ℓ=1 log푝 휃 푝 푖푑 ℓ | P 푖푑 <ℓ , W .(8) At inference,ARgenerates pose tokens until<eos>; we then de- serialize them back into ˆ Q 푛 푁 푛=1 and reconstruct poses with the frozen decoder ST 푑 . 4 Experiments 4.1 Experimental Setup Datasets. We evaluate our unified framework for SLT and SLP on two widely used datasets: Phoenix14T [3] and CSL-Daily [81]. Phoenix14T, a German Sign Language dataset from weather fore- casts, contains 8,257 videos (7,096/519/642 train/dev/test) across 9 signers, with 1,066 glosses and 2,887 German words. CSL-Daily, a Chinese Sign Language dataset covering daily-life topics, contains 20,654 videos (18,401/1,077/1,176 train/dev/test) from 10 signers, with 2,000 glosses and 2,343 Chinese words. Following SOKE [87], we represent each SL motion sequence as S∈R 푇×푑 , where푇is the sequence length and푑=133 denotes the number of SMPL-X parameters, including 11 upper-body joints, 30 hand joints, and 10 expression parameters. Implementation Details. (1) SR-RVQ. We tokenize continuous pose sequences using a 3-level RVQ module. The pose encoder downsamples the temporal length by a factor of 4, yielding latent features of length푇 ′ = 푇/4. The RVQ module comprises three codebooks,푄 1 ,푄 2 ,푄 3 , each containing 512 entries. Under our semantic-reconstruction decomposition,푄 1 is regularized by CTC and contrastive supervision to encode coarse semantic structures, while푄 2 and푄 3 capture residual motion details for more accurate reconstruction. (2) Autoregressive Model. We adopt mBART-large- c25 [40] as the backbone for sequence modeling. It consists of 12 encoder-decoder layers with a hidden size of 1024. Evaluation Metrics. For SLT, we report ROUGE-L [38] and BLEU- 1/BLEU-4 [44] on the generated sentences. For SLP, we evaluate both semantic consistency and motion accuracy. (1) Back-translation M ’26, November 10–14, 2026, Rio de Janeiro, BrazilXiao Liu et al. Table 1: Comparison of SLT performance. SLT-FT denotes task-specific fine-tuning from the unified model. SLT ExtraPhoenix14TCSL-Daily DEVTESTDEVTEST PoseRGBROUGEBLEU1BLEU4ROUGEBLEU1BLEU4ROUGEBLEU1BLEU4ROUGEBLEU1BLEU4 RGB-based SLRT [4]✓–47.2622.38–46.6121.3237.9637.4711.8836.7437.3811.79 STN-SLT [63] ✓–49.1223.23–48.6123.65– STMC-T [83]✓48.2447.6024.0946.6546.9823.65– SignBT [81]✓50.2951.1124.4549.5450.8024.3249.4951.4620.8049.3151.4221.34 MMTLB [5]✓53.1053.9527.6152.6553.9728.3953.3853.8124.4253.2553.3123.92 BN-TIN-Transf. [81]✓–37.2940.6612.7337.6740.7413.19 COSLRT [14] ✓52.4752.2927.8352.2452.4827.88– TwoStream-SLT [6]✓54.0854.3228.6653.4854.9028.9555.1055.2125.7655.7255.4425.79 SignDINO [12]✓53.6153.4927.1753.7954.1527.1752.3653.6425.6252.7552.1325.46 MixSignGraph [15]✓55.7755.0129.0053.8454.9028.9754.5455.8725.7754.6755.4125.87 Pose-based Skeletor [29]✓32.6631.9710.9131.8031.8610.35– Signbert+ [22]✓45.5344.4519.8644.8944.3520.41– VAP [30]✓51.47 52.78 26.6251.28 53.07 26.1648.7250.4121.1648.5649.9920.85 Uni-SLTP✓50.9251.9925.0050.4752.2925.7550.1151.2723.0649.8251.9623.63 Uni-SLTP (SLT-FT)✓51.8852.9326.9452.3953.2526.8950.7351.4123.4050.2452.2723.86 Table 2: Comparison of SLP performance. * denotes reimplemented results; SLP-FT denotes fine-tuning from the unified model. SLP Phoenix14TCSL-Daily DEVTESTDEVTEST B-T↑DTW-PA-JPE↓B-T↑DTW-PA-JPE↓B-T↑DTW-PA-JPE↓B-T↑DTW-PA-JPE↓ ROUGE BLEU4 BodyHandROUGE BLEU4 BodyHandROUGE BLEU4 BodyHandROUGE BLEU4 BodyHand Text2Gloss2Pose PT [52]11.873.8814.3310.4713.174.3113.1510.267.680.6715.8513.977.540.4116.3213.29 Sign-IDD [59]27.978.429.163.0727.118.469.003.0514.702.9813.134.5214.112.7413.244.69 Text2Pose SignPR [39]30.849.126.141.9032.869.416.081.8715.023.3612.974.2114.433.0112.824.56 T2M-GPT* [76]28.538.408.952.9528.978.418.762.6025.126.718.952.3426.576.938.442.32 SOKE* [87]30.9410.156.391.9131.2310.436.161.8527.859.747.812.0528.2910.317.582.17 Uni-SLTP33.4111.045.941.7634.2711.945.801.7229.7411.536.991.8630.0211.416.321.71 Uni-SLTP (SLP-FT)34.0911.765.491.6935.7312.045.541.6730.1811.956.671.7330.3612.016.281.65 Table 3: Comparison with unified human motion methods on SLT and SLP tasks on the Phoenix14T dataset. Methods SLT TaskSLP Task ROUGE↑ BLEU4↑BLEU4↑ DTW-Body↓ DTW-Hand↓ MotionGPT [28]35.3610.046.0610.646.89 MotionGPT2 [65]37.9611.888.629.343.41 MG-MotionLLM [68]36.6711.398.469.783.67 Uni-SLTP50.4725.7511.945.801.72 (B-T): We translate the generated poses back to text by employing a translation model [15], and report ROUGE-L and BLEU-1/4 scores against the ground truth. (2) Motion metrics: following SOKE [87], to handle length mismatch between generated and reference signs, we report JPE/MPJPE in the original and Procrustes-aligned spaces, and their DTW counterparts (DTW-PA-JPE), where DTW temporally aligns the generated and reference pose sequences while computing the joint position error. 4.2 Comparisons Following [39], we train the SLT model [15] and use it to evaluate our generated signs. SLT Comparisons. As shown in Table 1, we evaluate Uni-SLTP on Phoenix14T and CSL-Daily against both RGB-based and pose- based SLT methods. Since Uni-SLTP is pose-only, it is expected to underperform RGB-based models that exploit richer visual cues for SLT, especially compared to MixSignGraph. Nevertheless, under the same pose-based setting, Uni-SLTP remains competitive: it achieves performance comparable to the baseline VAP on Phoenix14T and yields an improvement on CSL-Daily. We further evaluate a task- specific SLT fine-tuned variant initialized from the unified model. This further improves SLT performance, indicating that the unified model already provides a strong shared foundation across tasks, while additional task-specific adaptation can further specialize the model for SLT. Overall, these results show that Uni-SLTP remains Bridging the Gap Between Semantics and Reconstruction: Unifying Sign Language Translation and ProductionMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil aberabdienstagwirdeslangsamwiederfreundlicherundwärmer. (But startingTuesday,it willgraduallybecomenicerandwarmeragain.) SOKE Ours Ground Truth 我爸差点考进大学。(My dad almost got into college.) Figure 4: SLP qualitative results of our Uni-SLTP and baseline method SOKE on Phoenix14T (left) and CSL-Daily (right) datasets. Table 4: Ablation of the SR-RVQ tokenizer. PA-MPJPE measures reconstruction quality; WER is reported when CTC loss is used. Tokenizer Setting RVQSupervisionCSLRSLTReconstruction (Residuals)ReconCTCCLWER↓ROUGE↑BLEU-4↑PA-MPJPE-body↓PA-MPJPE-hand↓ (a) VQ-VAE (Recon)✓-19.714.0813.024.38 (b) Semantic-VQ (CTC+Con)✓27.6551.6826.7644.5214.94 (c) VQ-VAE (Recon+CTC+Con)✓44.6136.5513.9716.736.84 (d) SR-RVQ (w/o CTC)✓-21.244.6513.194.48 (e) SR-RVQ (w/o Con)✓36.7648.4722.9013.304.46 (f ) SR-RVQ (Ours)✓28.6350.4725.7513.684.69 Table 5: Effect of SR-RVQ stages on SLT and SLP. RVQ stages used SLT TaskSLP TaskEfficiency ROUGE↑BLEU-4↑DTW-Body↓DTW-Hand↓s/video↓ (a)푄 1 51.6826.7618.7615.601.06 (b)푄 2 ,푄 3 18.203.985.991.761.47 (c)푄 1 ,푄 2 ,푄 3 45.7819.235.801.721.68 competitive among pose-based SLT methods, while also providing a unified framework that also supports SLP. SLP Comparisons. Table 2 reports the performance comparison between Uni-SLTP and prior SLP methods on the Phoenix14T and CSL-Daily datasets. Experimental results show that Uni-SLTP out- performs prior methods in both the B-T metric for semantic con- sistency and the motion-related metrics for motion accuracy. Task- specific fine-tuning for SLP further improves performance. Comparisons with Unified Human Motion Methods. As shown in Table 3, we compare Uni-SLTP with representative unified human motion methods on the Phoenix14T dataset. Directly applying these models to sign language tasks results in worse SLT performance and lower SLP performance than our method. One possible reason is that the alignment between sign sequences and text is more semantically complex and often non-monotonic, making it difficult for methods designed for generic human motion to model the fine- grained linguistic structure of sign language effectively. Table 6: Effect of the choice of the pre-trained LM backbone. Backbone SLT TaskSLP Task ROUGE↑ BLEU4↑BLEU4↑ DTW-Body↓ DTW-Hand↓ Llama3.2 1B [62]49.6125.3311.586.021.75 Gemma [61]49.9925.3611.325.971.76 mT5 [70]50.9925.3611.975.781.70 mBART (ours)50.4725.7511.945.801.72 Table 7: Ablation study of different loss weight combinations. Loss WeightsSLT TaskReconstruction 휆 rec 휆 CTC 휆 Con ROUGE↑ BLEU-4↑PA-MPJPE-body↓ PA-MPJPE-hand↓ 1.001.021.244.6513.194.48 1.01.0048.4722.9013.304.46 1.01.01.046.2923.7415.595.56 1.00.20.825.287.6213.244.52 1.00.80.250.4725.7513.684.69 4.3 Qualitative Results We provide a qualitative comparison between our Uni-SLTP and the baseline method SOKE on the SLP task. As shown in Figure 4, the sign sequences generated by Uni-SLTP exhibit finer hand-level details that are closer to the ground truth, resulting in lower motion error and more natural signing dynamics. M ’26, November 10–14, 2026, Rio de Janeiro, BrazilXiao Liu et al. Table 8: Ablation on RVQ depth. Eff. #Codes (↑) denotes the effective number of utilized codes. RVQ Depth 푁Eff. #Codes↑ SLT TaskSLP Task ROUGE↑ BLEU4↑BLEU4↑ DTW-Body↓ DTW-Hand↓ 138536.55 13.978.647.712.06 2 41650.9125.0010.677.291.87 333850.4725.7511.945.801.72 428050.6325.5411.695.821.73 Table 9: Ablation on codebook size. Codebook Size 퐾 SLT TaskSLP Task ROUGE↑ BLEU4↑BLEU4↑ DTW-Body↓ DTW-Hand↓ 12846.4022.8711.685.701.74 25649.8225.6411.625.681.71 51250.4725.7511.945.801.72 102450.94 26.1111.056.071.83 4.4 Ablation Study All ablation experiments were conducted on the Phoenix14T dataset. Effect of SR-RVQ Tokenizer. To assess the impact of tokenizer design on SLT and reconstruction, we compare representative variants in Table 4. (a) VQ-VAE (reconstruction-only) achieves the best PA-MPJPE but poor language metrics, while (b) Semantic-VQ (CTC+Con) substantially improves CSLR/SLT yet severely degrades reconstruction. (c) VQ-VAE (Recon+CTC+Con) adds semantic loss to a single-bottleneck VQ-VAE but still fails to balance semantics and motion accuracy. Turning to our SR-RVQ variants, the results with residual codebooks enabled show that both semantic signals matter: removing CTC (d) notably hurts SLT, and removing contrastive learning (e) reduces CSLR/SLT. (f) SR-RVQ (Ours) combines residual quantization with both CTC and contrastive supervision, yielding the best overall trade-off, with near semantic-only performance and reconstruction close to the VQ-VAE baseline. Effect of SR-RVQ Stages. Table 5 examines how using different SR- RVQ stages at training and inference affects SLT, SLP, and efficiency. Using (a) only the semantic stage gives the best SLT results but poor SLP motion quality, while using (b) only residual stages improves SLP but severely degrades SLT. Using (c) all stages yields the best overall trade-off, achieving competitive SLP metrics with acceptable runtime overhead. Therefore, we use the semantic stage for SLT and all stages for SLP. Effect of the Choice of the Pre-trained LM Backbone. To study the impact of the pre-trained LM, we replace mBART with other open- source language backbones, including Llama3.2 1B, Gemma, and mT5. As shown in Table 6, different backbones lead to consistent performance with only minor variations, and no single model sub- stantially changes task balance. This suggests that our framework is not sensitive to the choice of pre-trained LM, and the improvements primarily come from our unified sign representation and training design rather than the backbone itself. Effect of Loss Weights in the SR-RVQ Tokenizer. To evaluate the effect of the loss weights in SR-RVQ on SLT and reconstruction, Table 10: Efficiency statistics of Uni-SLTP. PhaseModelTrainable #Params (M)Efficiency Tokenizer trainingSR-RVQ76.954 min/epoch Unified task trainingAR model375.8815 min/epoch SLT inferenceUni-SLTPN/A12.6 samples/s SLP inferenceUni-SLTPN/A0.60 samples/s we conduct a sensitivity analysis in Table 7. The results show that increasing the semantic-loss weights improves SLT performance but degrades reconstruction quality, while larger reconstruction weights lead to the opposite trend. In the experiments, we choose the setting(휆 rec = 1.0,휆 CTC = 0.8,휆 Con = 0.2). Effect of RVQ Depth. To study RVQ depth, Table 8 varies the number of stages푁and reports both downstream metrics and code usage. A single-stage VQ (푁=1), where semantic and reconstruction objectives share the same codebook, underperforms on SLT/SLP despite relatively high code usage, while increasing depth markedly improves performance.푁=3 gives the best overall trade-off, achiev- ing the strongest SLP quality with balanced code usage; moving to푁=4 yields only marginal gains, indicating diminishing returns. Accordingly, we adopt a three-stage SR-RVQ in all experiments. Effect of Codebook Size. Codebook size controls the representa- tional capacity of each RVQ stage. We therefore fix the RVQ depth to푁=3 and vary the codebook size퐾to evaluate both SLT and SLP (Table 9). A small codebook (퐾=128) underperforms on both tasks, indicating limited expressiveness. In contrast, an overly large code- book (퐾=1024) yields only marginal SLT gains but harms SLP, with lower BLEU-4 and higher DTW errors. Overall,퐾=512 provides the best trade-off between semantic modeling and motion detail recovery, achieving the strongest overall performance. Model Size and Training/Inference Speed. As shown in Table 10, we report the trainable parameters, training and inference efficiency of our model on Phoenix14T using a single NVIDIA A6000 GPU. Training time is measured on the training set, while inference speed is evaluated on test samples with an average sequence length of 250 frames, averaged over 100 runs. Overall, Uni-SLTP enables efficient unified training, while SLP inference is slower due to autoregressive pose generation. 5 Conclusion We revisit SLT and SLP as two inverse directions of sign and text mapping, and show that unifying them is feasible only if the sign representation supports both semantic alignment for SLT and mo- tion accuracy for SLP. We propose Uni-SLTP with SR-RVQ, a hier- archical pose tokenizer that converts continuous pose sequences into discrete multi-stage tokens. It separates a semantic anchor token from residual motion detail tokens to bridge the Semantic- Reconstruction Gap. With the resulting shared token space, both tasks reduce to conditional next token prediction and can be han- dled by a single language model. Experiments on two widely used datasets show stronger SLP performance while keeping pose-based SLT competitive with task-specific baselines. Bridging the Gap Between Semantics and Reconstruction: Unifying Sign Language Translation and ProductionMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Acknowledgments This work is supported in part by National Natural Science Founda- tion of China under Grant Nos. 62172208, 92467202, 62272216; Key Projects of Jiangsu Provincial Basic Research Program under Grant No. BK20243040; JiangSu Natural Science Foundation under Grant No. BK20251989. This work is partially supported by Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (No. JYB2025XDXM118); the “111 Center” (No. B26023); Collaborative Innovation Center of Novel Software Technology and Industrialization. References [1]Vasileios Baltatzis, Rolandos Alexandros Potamias, Evangelos Ververas, Guanx- iong Sun, Jiankang Deng, and Stefanos Zafeiriou. 2024. Neural Sign Actors: A diffusion model for 3D sign language production from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1985–1995. [2]Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, and Richard Bowden. 2017. Subunets: End-to-end hand shape and continuous sign language recognition. In ICCV. IEEE, 3075–3084. [3] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. 2018. Neural sign language translation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7784–7793. [4]Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020. Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation. In CVPR. 10023–10033. [5]Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. 2022. A simple multi-modality transfer learning baseline for sign language translation. In CVPR. 5120–5130. [6] Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak. 2022. Two-stream network for sign language recognition and translation. Advances in Neural Information Processing Systems 35 (2022), 17043–17056. [7]Ka Leong Cheng, Zhaoyang Yang, Qifeng Chen, and Yu-Wing Tai. 2020. Fully convolutional networks for continuous sign language recognition. In ECCV. Springer. [8]Runpeng Cui, Hu Liu, and Changshui Zhang. 2017. Recurrent convolutional neural networks for continuous sign language recognition by staged optimization. In CVPR. 7361–7369. [9] Runpeng Cui, Hu Liu, and Changshui Zhang. 2019. A deep neural framework for continuous sign language recognition by iterative training. TMM 21, 7 (2019), 1880–1891. [10]Kawin Ethayarajh. 2019. How contextual are contextualized word represen- tations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 conference on empirical methods in natural language pro- cessing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 55–65. [11]Sen Fang, Chunyu Sui, Xuedong Zhang, and Yapeng Tian. 2023. Signdiff: Learning diffusion models for american sign language production. arXiv e-prints (2023), arXiv–2308. [12] Shiwei Gan, Xiao Liu, Yafeng Yin, Nan Liu, Kuizhuang Liu, Desibieer Tuerdaken, Zhiwei Jiang, Lei Xie, Sanglu Lu, and Hongkai Wen. 2026. Learning effective sign features without text for gloss-free sign language translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9827–9836. [13]Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Hongkai Wen, Lei Xie, and Sanglu Lu. 2024. Signgraph: A sign sequence is worth graphs of nodes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13470–13479. [14]Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Kang Xia, Lei Xie, and Sanglu Lu. 2023. Contrastive Learning for Sign Language Recognition and Translation.. In IJCAI, Vol. 23. 763–772. [15]Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Lei Xie, Sanglu Lu, and Hongkai Wen. 2025. MixSignGraph: A Sign Sequence is Worth Mixed Graphs of Nodes. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [16]Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, and Jun Liu. 2024. Llms are good sign language translators. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18362–18372. [17]Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, and Karen Livescu. 2025. Signmusketeers: An efficient multi-stream approach for sign language translation at scale. Findings of the Association for Computational Linguistics: ACL 2025 (2025). [18]Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, Karen Livescu, and Alexan- der H. Liu. 2025. SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction. In Proceedings of the 63rd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vienna, Austria, 28792–28810. https://aclanthology.org/2025.acl-long.1397/ [19]Dan Guo, Wengang Zhou, Houqiang Li, and Meng Wang. 2018. Hierarchical lstm for sign language translation. In AAAI, Vol. 32. [20]Jianyuan Guo, Peike Li, and Trevor Cohn. 2025. Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation. arXiv preprint arXiv:2505.15438 (2025). [21]Aiming Hao, Yuecong Min, and Xilin Chen. 2021. Self-Mutual Distillation Learn- ing for Continuous Sign Language Recognition. In ICCV. 11303–11312. [22]Hezhen Hu, Weichao Zhao, Wengang Zhou, and Houqiang Li. 2023. SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understand- ing. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023). [23]Hezhen Hu, Wengang Zhou, and Houqiang Li. 2021. Hand-Model-Aware Sign Language Recognition. In Proceedings of the AAAI Conference on Artificial Intelli- gence, Vol. 35. 1558–1566. [24]Lianyu Hu, Liqing Gao, Zekang Liu, and Wei Feng. 2022. Temporal lift pooling for continuous sign language recognition. In European conference on computer vision. Springer, 511–527. [25]Lianyu Hu, Liqing Gao, Zekang Liu, and Wei Feng. 2023. Continuous Sign Language Recognition with Correlation Network. In CVPR. 2529–2539. [26]Lianyu Hu, Liqing Gao, Zekang Liu, and Wei Feng. 2023. Self-emphasizing network for continuous sign language recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 854–862. [27] Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li, and Weiping Li. 2018. Video-based sign language recognition without temporal segmentation. In AAAI. [28]Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. 2023. Mo- tiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems 36 (2023), 20067–20079. [29]Tao Jiang, Necati Cihan Camgoz, and Richard Bowden. 2021. Skeletor: Skeletal transformers for robust body-pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3394–3402. [30] Peiqi Jiao, Yuecong Min, and Xilin Chen. 2024. Visual alignment pre-training for sign language translation. In European Conference on Computer Vision. Springer, 349–367. [31]Peiqi Jiao, Yuecong Min, Yanan Li, Xiaotao Wang, Lei Lei, and Xilin Chen. 2023. CoSign: Exploring Co-occurrence Signals in Skeleton-based Continuous Sign Language Recognition. In ICCV. 20676–20686. [32]Oscar Koller, Cihan Camgoz, Hermann Ney, and Richard Bowden. 2019. Weakly supervised learning with multi-stream CNN-LSTM-HMMs to discover sequential parallelism in sign language videos. TPAMI (2019). [33] Oscar Koller, Sepehr Zargaran, and Hermann Ney. 2017. Re-sign: Re-aligned end-to-end sequence modelling with deep recurrent CNN-HMMs. In CVPR. 4297– 4305. [34]Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. 2022. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 11523–11532. [35]Dongxu Li, Xin Yu, Chenchen Xu, Lars Petersson, and Hongdong Li. 2020. Trans- ferring cross-domain knowledge for video sign language recognition. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition. 6205–6214. [36] Zecheng Li, Wengang Zhou, Weichao Zhao, Kepeng Wu, Hezhen Hu, and Houqiang Li. 2025. Uni-sign: Toward unified sign language understanding at scale. arXiv preprint arXiv:2501.15187 (2025). [37]Han Liang, Chengyu Huang, Yuecheng Xu, Cheng Tang, Weicai Ye, Juze Zhang, Xin Chen, Jingyi Yu, and Lan Xu. 2024. LLaVA-SLT: Visual Language Tuning for Sign Language Translation. arXiv preprint arXiv:2412.16524 (2024). [38]Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd annual meeting of the association for computational linguistics (ACL-04). 605–612. [39]Xiao Liu, Shiwei Gan, Yafeng Yin, Bowen Guo, Zhiwei Jiang, Shunmei Meng, Lei Xie, and Sanglu Lu. 2026. SignPR: A Progressive Vector-Quantized Diffu- sion Framework for Sign Language Production. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2198–2208. [40]Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvinine- jad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics 8 (2020), 726–742. [41]Jian Ma, Wenguan Wang, Yi Yang, and Feng Zheng. 2024.Ms2sl: multi- modal spoken data-driven continuous sign language production. arXiv preprint arXiv:2407.12842 (2024). [42] Yuecong Min, Aiming Hao, Xiujuan Chai, and Xilin Chen. 2021. Visual alignment constraint for continuous sign language recognition. In ICCV. 11542–11551. [43]Zhe Niu and Brian Mak. 2020. Stochastic fine-grained labeling of multi-state sign glosses for continuous sign language recognition. In ECCV. Springer, 172–186. [44]Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In ACL. 311–318. [45]Maria Parelli, Katerina Papadimitriou, Gerasimos Potamianos, Georgios Pavlakos, and Petros Maragos. 2022. Spatio-temporal graph convolutional networks for continuous sign language recognition. In ICASSP. IEEE. M ’26, November 10–14, 2026, Rio de Janeiro, BrazilXiao Liu et al. [46]Junfu Pu, Wengang Zhou, Hezhen Hu, and Houqiang Li. 2020. Boosting Con- tinuous Sign Language Recognition via Cross Modality Augmentation. In M. 1497–1505. [47] Junfu Pu, Wengang Zhou, and Houqiang Li. 2018. Dilated convolutional network with iterative optimization for continuous sign language recognition.. In IJCAI, Vol. 3. 7. [48] Fan Qi, Yu Duan, Huaiwen Zhang, and Changsheng Xu. 2024. Signgen: End-to- end sign language video generation with latent diffusion. In European Conference on Computer Vision. Springer, 252–270. [49]Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. 2025. Tokenflow: Unified image tokenizer for multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference. 2545–2555. [50]Razieh Rastgoo, Kourosh Kiani, Sergio Escalera, and Mohammad Sabokrou. 2021. Sign language production: A review. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3451–3461. [51]Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. 2020.Adver- sarial training for multi-channel sign language production. arXiv preprint arXiv:2008.12405 (2020). [52]Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. 2020. Progressive transformers for end-to-end sign language production. In European Conference on Computer Vision. Springer, 687–705. [53]Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. 2021. Continuous 3d multi-channel sign language production via progressive transformers and mixture density networks. International journal of computer vision 129, 7 (2021), 2113–2135. [54]Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. 2021. Mixed signals: Sign language production via a mixture of motion primitives. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1919–1929. [55] Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. 2022. Signing at scale: Learning to co-articulate signs for large-scale photo-realistic sign language production. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5141–5151. [56]Stephanie Stoll, Necati Cihan Camgöz, Simon Hadfield, and Richard Bowden. 2018. Sign language production using neural machine translation and generative adversarial networks. In Proceedings of the 29th British Machine Vision Conference (BMVC 2018). British Machine Vision Association. [57]Stephanie Stoll, Armin Mustafa, and Jean-Yves Guillemaut. 2022. There and back again: 3d sign language generation from text using back-translation. In 2022 International Conference on 3D Vision (3DV). IEEE, 187–196. [58] Shengeng Tang, Dan Guo, Richang Hong, and Meng Wang. 2021. Graph-based multimodal sequential embedding for sign language translation. TMM (2021). [59]Shengeng Tang, Jiayi He, Dan Guo, Yanyan Wei, Feng Li, and Richang Hong. 2025. Sign-idd: Iconicity disentangled diffusion for sign language production. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 7266–7274. [60]Shengeng Tang, Feng Xue, Jingjing Wu, Shuo Wang, and Richang Hong. 2025. Gloss-driven conditional diffusion models for sign language production. ACM Transactions on Multimedia Computing, Communications and Applications 21, 4 (2025), 1–17. [61] Gemma Team. 2025. Gemma 3. (2025). https://goo.gle/Gemma3Report [62] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). [63]Andreas Voskou, Konstantinos P Panousis, Dimitrios Kosmopoulos, Dimitris N Metaxas, and Sotirios Chatzis. 2021. Stochastic transformer networks with linear competing units: Application to end-to-end sl translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 11946–11955. [64]Harry Walsh, Abolfazl Ravanshad, Mariam Rahmani, and Richard Bowden. 2024. A data-driven representation for sign language production. In 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG). IEEE, 1–10. [65]Yuan Wang, Di Huang, Yaqi Zhang, Wanli Ouyang, Jile Jiao, Xuetao Feng, Yan Zhou, Pengfei Wan, Shixiang Tang, and Dan Xu. 2024. Motiongpt-2: A general- purpose motion-language model for motion generation and understanding. arXiv preprint arXiv:2410.21747 (2024). [66]Chengcheng Wei, Jian Zhao, Wengang Zhou, and Houqiang Li. 2020. Semantic Boundary Detection with Reinforcement Learning for Continuous Sign Language Recognition. TCSVT 31, 3 (2020), 1138–1149. [67] Fangyun Wei and Yutong Chen. 2023. Improving continuous sign language recognition with cross-lingual signs. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 23612–23621. [68] Bizhu Wu, Jinheng Xie, Keming Shen, Zhe Kong, Jianfeng Ren, Ruibin Bai, Rong Qu, and Linlin Shen. 2025. Mg-motionllm: A unified framework for motion comprehension and generation across multiple granularities. In Proceedings of the Computer Vision and Pattern Recognition Conference. 27849–27858. [69]Pan Xie, Qipeng Zhang, Peng Taiying, Hao Tang, Yao Du, and Zexian Li. 2024. G2p-ddm: Generating sign pose sequence from gloss sequence with discrete diffusion model. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 6234–6242. [70]Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: Human language technologies. 483–498. [71]Zhaoyang Yang, Zhenmei Shi, Xiaoyong Shen, and Yu-Wing Tai. 2019. SF-Net: Structured Feature Network for Continuous Sign Language Recognition. arXiv preprint arXiv:1908.01341 (2019). [72]Jinhui Ye, Xing Wang, Wenxiang Jiao, Junwei Liang, and Hui Xiong. 2024. Improv- ing Gloss-free Sign Language Translation by Reducing Representation Density. In NeurIPS. [73] Aoxiong Yin, Haoyuan Li, Kai Shen, Siliang Tang, and Yueting Zhuang. 2024. T2S-GPT: Dynamic vector quantization for autoregressive sign language produc- tion from text. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 3345–3356. [74]Aoxiong Yin, Tianyun Zhong, Li Tang, Weike Jin, Tao Jin, and Zhou Zhao. 2023. Gloss attention for gloss-free sign language translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2551–2562. [75]Huaiwen Zhang, Zihang Guo, Yang Yang, Xin Liu, and De Hu. 2023. C2st: Cross-modal contextualized sequence transduction for continuous sign language recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 21053–21062. [76] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. 2023. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 14730–14740. [77]Zhihao Zhang, Junfu Pu, Liansheng Zhuang, Wengang Zhou, and Houqiang Li. 2019. Continuous sign language recognition via reinforcement learning. In ICIP. IEEE, 285–289. [78] Weichao Zhao, Hezhen Hu, Wengang Zhou, Jiaxin Shi, and Houqiang Li. 2023. BEST: BERT pre-training for sign language recognition with coupling tokeniza- tion. In Proceedings of the AAAI conference on artificial intelligence, Vol. 37. 3597– 3605. [79]Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang. 2023. Gloss-free sign language translation: Improving from visual-language pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 20871–20881. [80]Hao Zhou, Wengang Zhou, and Houqiang Li. 2019. Dynamic pseudo label decoding for continuous sign language recognition. In ICME. IEEE, 1282–1287. [81]Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. 2021. Improv- ing Sign Language Translation with Monolingual Data by Sign Back-Translation. In CVPR. 1316–1325. [82]Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. 2020. Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition.. In AAAI. [83]Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. 2021. Spatial-temporal multi-cue network for sign language recognition and translation. TMM (2021). [84]Wengang Zhou, Weichao Zhao, Hezhen Hu, Zecheng Li, and Houqiang Li. 2025. Scaling up multimodal pre-training for sign language understanding. IEEE Trans- actions on Pattern Analysis and Machine Intelligence (2025). [85]Qidan Zhu, Jing Li, Fei Yuan, and Quan Gan. 2024. Multiscale temporal network for continuous sign language recognition. Journal of Electronic Imaging 33, 2 (2024), 023059–023059. [86]Ronglai Zuo and Brian Mak. 2022. C2SLR: Consistency-Enhanced Continuous Sign Language Recognition. In CVPR. 5131–5140. [87]Ronglai Zuo, Rolandos Alexandros Potamias, Evangelos Ververas, Jiankang Deng, and Stefanos Zafeiriou. 2025. Signs as Tokens: A retrieval-enhanced multilingual sign language generator. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 23806–23816. [88]Ronglai Zuo, Fangyun Wei, Zenggui Chen, Brian Mak, Jiaolong Yang, and Xin Tong. 2024. A simple baseline for spoken language to sign language translation with 3d avatars. In European Conference on Computer Vision. Springer, 36–54. [89]Ronglai Zuo, Fangyun Wei, and Brian Mak. 2023. Natural language-assisted sign language recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 14890–14900.