Paper deep dive
SeRV: Semantic-Aligned Residual Vector Quantization for American Sign Language Generation
Hongyu Wu, Xu Wu, Tianhao Wu, Jiawei Yu, Phuc Nguyen, Jian Liu, Yi Wu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/9/2026, 4:51:56 AM
Summary
The paper introduces SeRV (Semantic-Aligned Residual Vector Quantization), a framework for generating 3D American Sign Language (ASL) motion from text. SeRV utilizes a semantic-aligned RVQ tokenizer that aligns motion tokens with text semantics at both sentence and token levels, enabling a Hierarchical GPT to predict residual motion tokens in a coarse-to-fine manner. The authors also construct a large-scale 3D ASL motion-text benchmark by reconstructing motion from YouTube-ASL videos. Experiments show SeRV achieves state-of-the-art pose accuracy on How2Sign and YouTube-ASL datasets.
Entities (10)
Relation Signals (8)
SeRV → evaluatedon → How2Sign
confidence 98% · Experiments across 375 hours of ASL video show that SeRV achieves state-of-the-art pose accuracy on both How2Sign and YouTube-ASL datasets
SeRV → evaluatedon → YouTube-ASL
confidence 98% · Experiments across 375 hours of ASL video show that SeRV achieves state-of-the-art pose accuracy on both How2Sign and YouTube-ASL datasets
SeRV → uses → RVQ-VAE
confidence 95% · SeRV is built on a Residual Vector-Quantized Variational AutoEncoder (RVQ-VAE)
SeRV → uses → T5
confidence 92% · a frozen T5 encoder Raffel et al. (2020) provides both sentence-level and token-level text features
SeRV → enables → Hierarchical GPT
confidence 90% · Building on this tokenizer, a Hierarchical GPT predicts residual motion tokens
YouTube-ASL → usedtoconstruct → 3D ASL motion-text benchmark
confidence 90% · We further construct a large-scale reconstructed 3D ASL motion-text benchmark by recovering paired 3D motion from YouTube-ASL videos.
SeRV → outperforms → SOKE
confidence 85% · SeRV achieves state-of-the-art pose accuracy... while producing semantically consistent 3D ASL motion
SeRV → outperforms → MaDiS
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:American Sign Language (ASL) generation remains challenging due to limited paired text-ASL motion data and the difficulty of learning motion representations both precise for reconstruction and predictable from linguistic input. Existing methods rely on motion tokenizers optimized for reconstruction, without explicit semantic supervision from paired text. As a result, the learned tokens remain limited in supporting semantically consistent and fine-grained ASL motion generation. To address this limitation, we propose SeRV (Semantic-Aligned Residual Vector Quantization), a semantic-aligned RVQ tokenizer for ASL generation. SeRV learns a semantically structured residual token space by combining sentence-level motion-text alignment with token-level text-conditioned supervision. Building on this tokenizer, a Hierarchical GPT predicts residual motion tokens in a coarse-to-fine manner, generating structurally coherent and semantically aligned 3D ASL motion. We further construct a large-scale reconstructed 3D ASL motion-text benchmark by recovering paired 3D motion from YouTube-ASL videos. Experiments across 375 hours of ASL video show that SeRV achieves state-of-the-art pose accuracy on both How2Sign and YouTube-ASL datasets, while producing semantically consistent 3D ASL motion directly from text.
Tags
Links
- Source: https://arxiv.org/abs/2609.05742v1
- Canonical: https://arxiv.org/abs/2609.05742v1
Trouble viewing inline? Open PDF directly →
Full Text
67,507 characters extracted from source content.
Expand or collapse full text
SeRV: Semantic-Aligned Residual Vector Quantization for American Sign Language Generation Hongyu Wu Xu Wu Tianhao Wu Jiawei Yu Phuc Nguyen Jian Liu Yi Wu Affiliation: University of Oklahoma University of Georgia University of Massachusetts Amherst Abstract American Sign Language (ASL) generation remains challenging due to limited paired text-ASL motion data and the difficulty of learning motion representations both precise for reconstruction and predictable from linguistic input. Existing methods rely on motion tokenizers optimized for reconstruction, without explicit semantic supervision from paired text. As a result, the learned tokens remain limited in supporting semantically consistent and fine-grained ASL motion generation. To address this limitation, we propose SeRV (Semantic-Aligned Residual Vector Quantization), a semantic-aligned RVQ tokenizer for ASL generation. SeRV learns a semantically structured residual token space by combining sentence-level motion-text alignment with token-level text-conditioned supervision. Building on this tokenizer, a Hierarchical GPT predicts residual motion tokens in a coarse-to-fine manner, generating structurally coherent and semantically aligned 3D ASL motion. We further construct a large-scale reconstructed 3D ASL motion–text benchmark by recovering paired 3D motion from YouTube-ASL videos. Experiments across 375 hours of ASL video show that SeRV achieves state-of-the-art pose accuracy on both How2Sign and YouTube-ASL datasets, while producing semantically consistent 3D ASL motion directly from text. Figure 1: Left: ASL generated by SeRV from text for two example sentences, rendered as 3D avatars alongside the reference signer. Right: comparison with state-of-the-art methods on six metrics across How2Sign and YouTube-ASL datasets (body/hand DTW-JPE, lower better; SiBLEU-4, higher better; all axes oriented so larger area = better). SeRV achieves the best result on five of six metrics. Note MaDiS Zuo et al. (2026) is concurrent work reporting How2Sign only; its YouTube-ASL axes are drawn at the center to indicate results are unavailable, not zero. 1 Introduction American Sign Language (ASL) is the primary communication method of Deaf and Hard-of-Hearing (DHH) communities, but the lack of fluent signers in the United States creates substantial communication barriers. To bridge this gap, recent AI research has focused on Sign Language Translation (SLT) Jang et al. (2025); Li et al. (2025); Wong et al. (2024) and Sign Language Generation (SLG) Zuo et al. (2025); Yu et al. (2024); Fang et al. (2025); Yin et al. (2024). Some SLG methods formulate the task as visual content generation Saunders et al. (2022); Fang et al. (2023), which often underrepresents the fine-grained motion patterns that carry critical linguistic information. Motivated by the linguistic nature of sign language, tokenization-based SLG has recently attracted growing interest Yin et al. (2024); Zuo et al. (2025), where continuous sign motions are converted into discrete tokens. However, many existing methods still rely on an external sign dictionary and a retrieval step to assemble motion Zuo et al. (2025), which limits generation to a fixed vocabulary rather than producing sign motion directly from text. A more fundamental limitation lies in the tokenizer itself. ASL meaning is carried by fine-grained hand articulation, where small differences distinguish different signs. Existing tokenizers are trained purely for reconstruction, so the token space has no alignment to the meaning of the conditioning text. The token space is thus hard to predict from text: two signs with distinct meanings can map to nearly the same tokens, and the generator must learn a tangled text-to-token mapping, producing motion that looks natural but conveys the wrong sign. Current 3D ASL resources Duarte et al. (2021); Yu et al. (2024) are also limited in scale, vocabulary, and motion diversity. In this work, we address these limitations by redesigning the motion tokenizer and downstream generator for 3D ASL generation. We propose SeRV (Semantic-Aligned Residual Vector Quantization), which reshapes the residual quantization hierarchy with linguistic supervision rather than reconstruction alone, anchoring fine-grained hand articulation to text-relevant semantics so that these residual codes become predictable from language. To be specific, SeRV is built on a Residual Vector-Quantized Variational AutoEncoder (RVQ-VAE) Lee et al. (2022); Guo et al. (2024), where ASL motion is represented by a coarse-to-fine hierarchy of residual codebooks. Each level progressively refines the motion representation from coarse global structure to fine-grained hand and upper-body articulations, reducing approximation error while preserving motion detail. SeRV trains the encoder with two semantic alignment objectives: a sentence-level alignment on the base-level (coarsest) tokens, and a token-level motion–text alignment on the fully quantized latent. This drives coarse tokens to encode sentence-level semantics while finer residuals refine articulatory details. Building on this token space, we further propose Hierarchical GPT, an RVQ-aware autoregressive generator that mirrors SeRV’s coarse-to-fine hierarchy. At each residual level, it predicts tokens conditioned on per-token text features and the previously decoded lower-level motion tokens. Unlike flat single-sequence prediction Guo et al. (2025b), which inflates sequence length and weakens the residual hierarchy, or two-stage schemes Guo et al. (2024) that decouple coarse and fine prediction, this progressive design preserves the hierarchy and produces structurally coherent, semantically consistent ASL motion. To support scalable token learning, we further expand ASL training resources by reconstructing paired 3D sign motion from YouTube-ASL Uthus et al. (2023) videos using a state-of-the-art motion reconstruction pipeline Sun et al. (2024), followed by quality-based cleaning and filtering. To the best of our knowledge, this results in the largest reconstructed 3D ASL motion-text benchmark to date, providing richer supervision for token learning. As illustrated in Figure 1, SeRV generates 3D ASL motions with faithful hand articulation and achieves the best result on five of six reported metrics across How2Sign and YouTube-ASL. Note MaDiS Zuo et al. (2026) is concurrent work released shortly before our submission; as its code and checkpoints are not publicly available, we cannot evaluate it on the larger YouTube-ASL benchmark under the same protocol. Our contributions are summarized as follows: • We propose SeRV, a semantic-aligned RVQ framework for text-to-3D ASL generation. SeRV combines sentence-level and token-level motion–text alignment to learn a residual token space that is precise for reconstruction and predictable from text, and instantiates a hierarchy-aware autoregressive generator to synthesize ASL motion following the learned coarse-to-fine token hierarchy. • We construct the largest reconstructed 3D ASL motion-text benchmark to date by reconstructing 3D sign motion from YouTube-ASL videos and curating it at scale, providing broad signer and linguistic diversity for training generative ASL models. • Extensive experiments on How2Sign and YouTube-ASL show that SeRV substantially outperforms prior text-to-3D ASL generation methods, achieving state-of-the-art pose accuracy on both benchmarks and strong sign-token consistency. 2 Related Work Sign Language Generation (SLG). Early SLG methods often adopt cascaded pipelines, bridging text and sign motion via glosses, skeletons, or pose sequences Stoll et al. (2020); Zelinka and Kanis (2020). These intermediate representations provide linguistic or structural guidance but require costly annotations and risk information bottlenecks or error accumulation Saunders et al. (2021); Zuo et al. (2025); Zuo et al. (2026). Others generate continuous sign poses or videos directly from text via recurrent networks Saunders et al. (2020); Saunders et al. (2021), Transformers Saunders et al. (2022); Qi et al. (2024); Fang et al. (2025), GANs Stoll et al. (2020), or diffusion models Fang et al. (2023). However, operating in a continuous output space makes long-horizon generation harder to stabilize and offers limited discrete structure for scalable autoregressive modeling. Recent SLG methods tokenize 3D sign motion into discrete codes. Early efforts use pose-level VQ-VAE Xie et al. (2024b) or VQ-GAN-style video tokenizers Esser et al. (2021); Xie et al. (2024a). More recent work targets 3D motion: SignAvatars Yu et al. (2024) pairs a large-scale 3D holistic sign dataset with a VQ-VAE codebook; T2S-GPT Yin et al. (2024) uses dynamic vector quantization for adaptive-length codes; SOKE Zuo et al. (2025) tokenizes upper-body, left-hand, and right-hand separately and generates multilingual signs via a pretrained LM with dictionary retrieval; MaDiS Zuo et al. (2026) combines part-wise tokens with masked diffusion language modeling and tri-level cross-modal pretraining; and SignViP Wang et al. (2025) extends tokenization to sign video generation with multi-condition tokens. ASL Datasets. How2Sign Duarte et al. (2021) offers 80+ hours of in-studio continuous ASL with aligned RGB, depth, and English transcripts. YouTube-ASL Uthus et al. (2023) scales to 984 hours of in-the-wild video from 2,500+ signers, broadening vocabulary and visual diversity. To address these limitations, we construct a large-scale reconstructed 3D ASL motion–text benchmark and propose SeRV, a semantic-aligned RVQ tokenizer that learns residual tokens both precise for reconstruction and predictable from text, enabling fine-grained articulation control for ASL generation. 3 Methodology The overview of our framework is shown in Figure 2 and Figure 3. Our method has two stages: SeRV (Semantic-Aligned Residual Vector Quantization), a text-aligned RVQ tokenizer, and Hierarchical GPT, a coarse-to-fine token generator. SeRV converts continuous 3D ASL motion into residual discrete tokens while aligning the motion representations with the paired transcript. Hierarchical GPT then predicts these residual tokens from text in a coarse-to-fine autoregressive order, and the predicted tokens are decoded by the learned motion decoder to reconstruct the final 3D ASL motion. 3.1 Preliminaries of RVQ-VAE Given a sign motion sequence S, a motion encoder ℰE produces a continuous latent sequence =ℰ()∈ℝT′×Cz=E(S) ^T × C, with temporal length T′T and feature dimension C. A codebook =mm=0K−1C=\e_m\_m=0^K-1 is a learnable set of K embedding vectors; each latent vector is quantized to its nearest code, and the selected index serves as a discrete token. We adopt residual vector quantization with L codebooks ll=0L−1\C_l\_l=0^L-1 Guo et al. (2024); Lee et al. (2022). Starting from (0)=r^(0)=z, stage l computes (l)=Q(l)((l))q^(l)=Q^(l)(r^(l)) and updates the residual (l+1)=(l)−(l)r^(l+1)=r^(l)-q^(l), where Q(l)(⋅)Q^(l)(·) is nearest-neighbor quantization with lC_l. The selected index kt(l)k_t^(l) at position t is the discrete motion token. The final quantized latent is ^=∑l=0L−1(l) z= _l=0^L-1q^(l), decoded to ^=(^) S=D( z) by the motion decoder D. The RVQ-VAE is trained with a reconstruction loss and a per-level embedding constraint: ℒRVQ=ℒrec+β∑l=0L−1‖(l)−sg[(l)]‖22,L_RVQ=L_rec+β _l=0^L-1 \|r^(l)-sg\! [q^(l) ] \|_2^2, (1) where ℒrec=‖−^‖1L_rec=\|S- S\|_1, sg[⋅]sg[·] is the stop-gradient operator, and β weights the embedding constraint. 3.2 SeRV Model Architecture Figure 2: Overview of SeRV. We introduce a global-level loss ℒglobalL_global on the level-0 motion representation and a token-level loss ℒtokL_tok on the fully quantized latent sequence. Together with the standard RVQ tokenizer objective ℒRVQL_RVQ, these objectives encourage the learned discrete motion tokens to preserve motion fidelity while retaining text-relevant semantics. Unlike a reconstruction-only RVQ-VAE, SeRV adds semantic supervision from paired transcripts during tokenizer training. As illustrated in Figure 2, a frozen T5 encoder Raffel et al. (2020) provides both sentence-level and token-level text features from the same transcript. We align the sentence-level embedding with the level-0 quantized latent (0)q^(0), a coarse content representation of the ASL motion, and use the token-level features in a motion–text late interaction loss over quantized motion latents. These losses inject global and token-level textual supervision into the tokenizer. Global-Level Sign–Text Alignment. We align (0)q^(0) with the paired text representation so that the learned ASL motion tokens retain text-relevant semantics. Given the paired transcript c, a frozen pretrained T5 encoder Raffel et al. (2020) ℰtextEtext produces token-level text features =ℰtext()∈ℝLt×dcF=Etext(c) ^L_t× d_c, where LtL_t is the tokenized text length and dcd_c is the T5 hidden dimension. We use the average hidden state over non-padded tokens as the sentence-level representation ∈ℝdcf ^d_c. On the motion side, we summarize (0)q^(0) into a global motion embedding (0)g^(0) using temporal mean pooling. The motion and text embeddings are then projected into a shared semantic space and ℓ2 _2-normalized. For a mini-batch of size B, we apply a symmetric global-level alignment loss ℒglobal=12(ℒm→tglobal+ℒt→mglobal)L_global= 12 (L_m→ t^global+L_t→ m^global ). Let sij(0)=sim(^i(0),^j)s_ij^(0)=sim ( g_i^(0), f_j ) denote the cosine similarity between the i-th motion embedding and the j-th text embedding. The two directional losses are then defined as ℒm→tglobal _m→ t^global =−1B∑i=1Blogexp(sii(0)/τ)∑j=1Bexp(sij(0)/τ), =- 1B _i=1^B \! (s_i^(0)/τ ) _j=1^B \! (s_ij^(0)/τ ), (2) ℒt→mglobal _t→ m^global =−1B∑i=1Blogexp(sii(0)/τ)∑j=1Bexp(sji(0)/τ). =- 1B _i=1^B \! (s_i^(0)/τ ) _j=1^B \! (s_ji^(0)/τ ). (3) Here, ^i(0) g_i^(0) and ^i f_i denote the projected and ℓ2 _2-normalized level-0 motion and text embeddings, and τ is the temperature. The matched motion–text pair is treated as the positive pair, while the other samples in the mini-batch serve as negatives. Figure 3: Hierarchical GPT for coarse-to-fine ASL motion generation. Given the text feature from the frozen condition encoder, the generator autoregressively predicts SeRV tokens over time and RVQ levels. For each time step, a coarse decoder first predicts the level-00 token, while residual decoders progressively predict higher-level tokens conditioned on the temporal history, the generated coarser tokens, and the text feature. The predicted multi-level tokens are then decoded by the shared motion decoder to reconstruct the ASL motion sequence. Token-Level Sign–Text Alignment. Sentence level contrastive loss provides global alignment but leaves token-level text–motion interactions implicit. We therefore introduce a late-interaction loss Yao et al. (2021) that averages token-wise maximum similarities between text tokens and quantized ASL motion latents. For text features, we reuse the T5 token hidden states ∈ℝLt×dcF ^L_t× d_c; for motion features, we use the fully quantized latent ^=∑l=0L−1(l)∈ℝT′×C z= _l=0^L-1q^(l) ^T × C. Separate token-level projection heads map them into a shared normalized space, producing ~∈ℝLt×da F ^L_t× d_a and ~∈ℝT′×da Z ^T × d_a. For each ASL motion–text pair, we compute a token-wise cosine similarity matrix ∈ℝLt×T′A ^L_t× T , where t,m=sim(~t,~m)A_t,m=sim( F_t, Z_m) measures the similarity between the t-th text token and the m-th motion latent. Let ΩF _F and ΩZ _Z denote valid text-token and motion-latent positions. We define the late-interaction scores as sF→Z=1|ΩF|∑u∈ΩFmaxv∈ΩZu,vs_F→ Z= 1| _F| _u∈ _F _v∈ _ZA_u,v and sZ→F=1|ΩZ|∑v∈ΩZmaxu∈ΩFu,v.s_Z→ F= 1| _Z| _v∈ _Z _u∈ _FA_u,v. Each valid token is therefore matched to its closest token in the other modality. Given a mini-batch of size B, let sijm→ts_ij^m→ t and sijt→ms_ij^t→ m denote the motion-to-text and text-to-motion scores between the motion sequence of sample i and the text sequence of sample j. We apply a symmetric token-level InfoNCE loss ℒtok=12(ℒm→ttok+ℒt→mtok)L_tok= 12(L_m→ t^tok+L_t→ m^tok). The two directional losses are defined as ℒm→ttok _m→ t^tok =−1B∑i=1Blogexp(siim→t/τ′)∑j=1Bexp(sijm→t/τ′), =- 1B _i=1^B (s_i^m→ t/τ ) _j=1^B (s_ij^m→ t/τ ), (4) ℒt→mtok _t→ m^tok =−1B∑i=1Blogexp(siit→m/τ′)∑j=1Bexp(sjit→m/τ′). =- 1B _i=1^B (s_i^t→ m/τ ) _j=1^B (s_ji^t→ m/τ ). (5) Here, τ′τ is the token-level temperature. Training Objective. The overall training objective of SeRV is defined as ℒSeRV=ℒRVQ+λglobalℒglobal+λtokℒtok.L_SeRV=L_RVQ+ _globalL_global+ _tokL_tok. (6) Here, ℒRVQL_RVQ denotes the standard RVQ tokenizer objective, ℒglobalL_global aligns the coarse level-0 motion representation with sentence-level text semantics, while ℒtokL_tok encourages fine-grained interactions between text tokens and quantized ASL motion latents. The coefficients λglobal _global and λtok _tok balance the two semantic alignment objectives. 3.3 Hierarchical GPT Hierarchy-aligned generation. We propose Hierarchical GPT, an RVQ-aware autoregressive generator that predicts SeRV tokens over both time and quantization depth. We reuse the frozen text encoder and denote its output as =ℰcond()f=E_cond(c), where c is the input text. Let 1:Tq0:L−1k_1:T_q^0:L-1 be the multi-level token hierarchy produced by SeRV, where L is the number of RVQ levels, TqT_q is the token sequence length, and ktlk_t^l is the token at time t and level l. Following the text-aligned hierarchy learned by SeRV, the generator predicts lower-level coarse tokens before higher-level residual tokens: p(1:Tq0:L−1∣)=∏t=1Tq∏l=0L−1p(ktl∣<t0:L−1,t0:l−1,),p(k_1:T_q^0:L-1 )= _t=1^T_q _l=0^L-1p\! (k_t^l _<t^0:L-1,k_t^0:l-1,f ), (7) where p(⋅)p(·) is the categorical distribution over codebook entries, <t0:L−1k_<t^0:L-1 denotes all tokens before time t, and t0:l−1k_t^0:l-1 denotes coarser tokens already generated. This factorization preserves temporal causality and coarse-to-fine dependencies. Coarse-to-fine decoding. We implement Eq. 7 with a coarse token decoder followed by residual token decoders. At each time step t, the coarse decoder predicts the level-00 token kt0k_t^0 from the previous token history and the text feature f, capturing the coarse text-conditioned motion pattern. The coarse decoder is a causal Transformer built with LLaMA-style blocks, including RMSNorm, SwiGLU, and rotary positional embeddings (RoPE) Touvron et al. (2023). For each higher level l>0l>0, a residual decoder predicts ktlk_t^l conditioned on the previous token history <t0:L−1k_<t^0:L-1, the coarser tokens t0:l−1k_t^0:l-1 already generated at the current time step, and the text feature f. The embeddings of these coarser tokens are fed to the residual decoder, allowing higher levels to refine the current coarse representation rather than generate the motion state from scratch. Since residual decoders mainly model refinement details, we use fewer Transformer layers while retaining attention layers for fine-grained articulation modeling. This decoding process follows the coarse-to-fine hierarchy learned by SeRV. Training objective. Let ^tl p_t^\,l denote the predicted token distribution at level l and time step t, and let ktlk_t^l be the corresponding ground-truth token from the frozen SeRV tokenizer. The generator is trained with a level-wise cross-entropy loss: ℒHGPT=∑l=0L−1λl1Tq∑t=1TqCE(^tl,ktl),L_HGPT= _l=0^L-1 _l 1T_q _t=1^T_qCE ( p_t^\,l,k_t^l ), (8) where TqT_q is the token length, λl _l is the loss weight for level l, and CE(⋅,⋅)CE(·,·) denotes cross-entropy. 4 Performance Evaluation Table 1: Comparison with state-of-the-art methods on How2Sign and YouTube-ASL. Lower is better for DTW-JPE, and higher is better for SiBLEU-4. Method How2Sign YouTube-ASL Body↓ Hands↓ SiBLEU-4↑ Body↓ Hands↓ SiBLEU-4↑ S-MotionGPT∗ Jiang et al. (2023) 12.41 13.74 2.32 8.95 10.12 1.84 MoMask++∗ Guo et al. (2025a) 9.06 11.34 1.53 6.54 8.35 1.67 SOKE† Zuo et al. (2025) 7.75 10.08 2.37 5.42 7.44 2.18 MaDiS Zuo et al. (2026) 6.59 9.45 4.47 – – – Ours 6.51 9.22 4.12 4.98 6.70 3.13 Notes. Body and Hands are DTW-JPE errors; Hands denotes the average over left and right hands. ∗ denotes general motion-generation baselines adapted to sign motion generation, and † denotes methods using external sign dictionaries. How2Sign results for prior methods are taken from reported results Zuo et al. (2025); Zuo et al. (2026). YouTube-ASL results are obtained under the same SMPL-X representation, split, and evaluation protocol. MaDiS is not reported on YouTube-ASL because its official code and checkpoints are unavailable. 4.1 Experimental Methodology Datasets. We evaluate on two ASL datasets, How2Sign Duarte et al. (2021) and YouTube-ASL Uthus et al. (2023). For How2Sign, we use its SMPL-X motion annotations and the official train/validation/test split. For YouTube-ASL, we reconstruct 3D SMPL-X signing motion from these videos and filter out low-quality clips with unreliable reconstructions or text-motion misalignment. For both datasets, we represent sign motion as sequences of SMPL-X parameters covering upper-body and hand joints. Full preprocessing, filtering criteria, and dataset statistics are provided in Appendix A.5. Evaluation metrics & Implementation. Since generated sign sequences may differ in length from the ground truth, we follow prior 3D sign language generation work and use Dynamic Time Warping over Joint Position Error (DTW-JPE) Zuo et al. (2025); Cai et al. (2019) to measure sequence-level motion distance. DTW temporally aligns the generated and reference sequences before computing joint-position errors. We report DTW-JPE for the body, left hand, and right hand, and compute Hand Avg. by averaging the left- and right-hand errors. For tokenizer reconstruction, where the generated motion is frame-aligned with the input, temporal alignment is unnecessary; we instead report PA-MPJPE, the Mean Per-Joint Position Error after rigid (Procrustes) Alignment. Following MaDiS Zuo et al. (2026), we also report SiBLEU-4 to measure token-level consistency in the sign-token space. We discretize both generated and ground-truth motions with the same frozen evaluator tokenizer and compute BLEU-4 over the resulting token sequences. Formal definitions of all metrics are provided in Appendix A.6. We train the framework in two stages. SeRV uses L=3L=3 RVQ levels, latent dimension C=512C=512, and codebook size K=512K=512 per level. Hierarchical GPT uses a hidden dimension of 10241024, 99 Transformer layers, and 1616 attention heads. Full details are in Appendix A.7. Figure 4: Qualitative comparison of text-to-3D ASL generation. For each of six sentences, we show the reference signer video (top) and the 3D motion generated by SOKE and by our method (SeRV). The top three examples are from How2Sign; the bottom three are from YouTube-ASL (faces masked for privacy). 4.2 Comparison with State-of-the-Art Methods We compare our method against both general motion-generation models adapted to sign motion and recent sign-specific generation methods. The general motion generators include S-MotionGPT Jiang et al. (2023), an autoregressive token-based model, and MoMask++ Guo et al. (2025a), a masked generative token model, both adapted to 3D sign motion under our SMPL-X representation. The sign-specific methods include SOKE Zuo et al. (2025), a strong autoregressive baseline with part-wise motion tokenization and external dictionary retrieval, and MaDiS Zuo et al. (2026), a recent masked diffusion model for sign generation. Since MaDiS is not open-sourced, we are unable to evaluate it on YouTube-ASL, and we therefore report only its How2Sign results from the original paper. As shown in Table 1, general motion generators adapted to sign perform substantially worse than sign-specific methods, confirming that ASL generation requires modeling tailored to its fine-grained articulatory structure. On YouTube-ASL, our method achieves the best results on all metrics, lowering the body and hand DTW-JPE of the strongest baseline, SOKE, by 8.1% and 9.9%, and raising SiBLEU-4 from 2.18 to 3.13. On How2Sign, our model attains the lowest body and hand errors among all methods; MaDiS reports a higher SiBLEU-4 (4.47 vs. 4.12), although, as noted above, it cannot be assessed on the larger YouTube-ASL benchmark due to the lack of a public release. Notably, our model surpasses SOKE on every metric without relying on an external sign dictionary, indicating that the improvements stem from the learned semantic-aligned token space rather than from retrieval. These results demonstrate the benefit of semantic-aligned residual tokenization for ASL generation. Performance of Tokenizer Reconstruction. We further evaluate whether the proposed tokenizer preserves the input 3D motion. As shown in Table 2, SeRV achieves the lowest reconstruction errors on both How2Sign and YouTube-ASL. Compared with the standard VQ-VAE and the part-wise Multi-VAE tokenizer used in SOKE, SeRV consistently reduces body and hand reconstruction errors, indicating that the proposed semantic alignment does not compromise motion fidelity. The gains are especially important for downstream generation, as a more accurate residual token space provides cleaner targets for predicting both coarse body motion and fine-grained hand articulation. Table 2: Reconstruction performance of different motion tokenizers on How2Sign and YouTube-ASL. Method How2Sign(PA-MPJPE↓ ) YouTube-ASL(PA-MPJPE↓ ) Body↓ Hands↓ Body↓ Hands↓ VQ-VAE 24.07 7.96 19.53 4.45 Multi-VAE 19.37 6.65 16.96 3.62 SeRV 16.23 6.13 13.91 3.31 4.3 Qualitative Comparison We provide qualitative comparisons in Figure 4. For each input transcript, we show the 3D signing motion generated by SOKE and SeRV. Compared with SOKE, SeRV produces more accurate hand configurations, especially for the fine-grained handshapes that carry important linguistic information; for instance, in the “Clean tools” example, the baseline outputs leave the hands in a weakly articulated, near-static pose, whereas our method reproduces the intended handshape. Our method also yields more plausible upper-body posture with fewer self-contact artifacts. 4.4 Ablation Studies Text–sign retrieval analysis. We further evaluate whether SeRV learns a more text-aligned token space through bidirectional text–sign retrieval on both How2Sign and YouTube-ASL. Text inputs are represented by frozen T5 global embeddings, while sign motions are represented by pooled latents from the frozen tokenizer encoder. Under the same per-batch retrieval protocol with batch size 64, SeRV improves Recall@1, Recall@5, and Recall@10 in both Text→ and Sign→ directions, as shown in Table 3. This suggests that semantic supervision improves cross-modal alignment in the learned sign-token space. Table 3: Per-batch bidirectional text–sign retrieval analysis with batch size 64. Higher is better for all recall metrics. Dataset Tokenizer Direction R@1↑ R@5↑ R@10↑ How2Sign RVQ T → S 11.61 37.28 59.04 SeRV 14.52 46.97 73.21 RVQ S → T 6.73 27.05 47.10 SeRV 8.42 34.08 58.41 YouTube-ASL RVQ T → S 7.55 24.23 38.38 SeRV 9.73 31.50 49.01 RVQ S → T 4.37 17.58 30.62 SeRV 5.64 22.85 39.10 Effect of SeRV Tokenization. We first examine whether semantic supervision improves learned motion tokens. We compare a standard RVQ tokenizer trained only with reconstruction losses, a variant with sentence-level motion–text alignment on the level-0 quantized latent, a variant with token-level late-interaction alignment on the fully quantized latent sequence, and the full SeRV tokenizer with both objectives. All variants use the same generator architecture. As shown in Table 4 and Figure 6 in Appendix A.9, on both How2Sign and YouTube-ASL, semantic supervision consistently improves generation accuracy, with larger gains on hand motion, and raises SiBLEU-4, indicating better token-level consistency with the input text. Combining sentence-level and token-level alignment performs best across all metrics, suggesting that global semantic alignment and token-level late interaction provide complementary supervision for learning text-predictable motion tokens. Table 4: Effect of SeRV tokenizer objectives on How2Sign and YouTube-ASL. We report DTW-JPE (lower is better) and SiBLEU-4 (higher is better). All variants use the same generator. Tokenizer Objective How2Sign YouTube-ASL Body↓ Hands↓ SiB4.↑ Body↓ Hands↓ SiB4.↑ Baseline (RVQ) 6.82 9.63 3.20 5.39 7.20 2.56 + ℒglobalL_global 6.67 9.50 3.53 5.24 7.01 2.62 + ℒtokL_tok 6.53 9.31 3.80 5.15 6.85 2.87 + ℒglobalL_global + ℒtokL_tok 6.51 9.22 4.12 4.98 6.70 3.13 Effect of Hierarchical Generation. We evaluate whether the proposed generator benefits from explicitly modeling the residual token hierarchy. We compare three generation designs using the same SeRV tokenizer. The flat autoregressive baseline flattens all residual tokens into a single sequence and predicts them left-to-right. The two-stage baseline first predicts coarse tokens and then predicts the remaining residual tokens in a separate refinement stage. Our Hierarchical GPT predicts residual tokens in a coarse-to-fine order while conditioning each level on previously generated lower-level tokens. As shown in Table 5, on both How2Sign and YouTube-ASL, Hierarchical GPT consistently outperforms both the flat autoregressive (Flat AR) and two-stage baselines on body and hand motion, suggesting that explicitly preserving the residual token hierarchy is important for accurate ASL motion generation. Table 5: Ablation on generator architectures on How2Sign and YouTube-ASL. All variants use the same SeRV tokenizer. Lower is better for DTW-JPE. Generator How2Sign (DTW-JPE↓ ) YouTube-ASL (DTW-JPE↓ ) Body Hands Body Hands Flat AR 7.73 10.41 7.59 9.25 Two-stage 7.21 9.86 6.92 8.33 Hierarchical GPT 6.51 9.22 4.98 6.70 5 Conclusion We present SeRV, a semantic-aligned RVQ tokenizer for text-to-3D ASL generation. Unlike reconstruction-only tokenizers, SeRV uses sentence-level and token-level motion–text alignment to learn residual tokens that are both precise and text-predictable. Building on SeRV, Hierarchical GPT predicts residual tokens over time and quantization depth while preserving the coarse-to-fine hierarchy. To support scalable training and evaluation, we construct a large-scale reconstructed 3D ASL motion-text benchmark from YouTube-ASL. Experiments on How2Sign and YouTube-ASL show state-of-the-art performance, with consistent gains in body and hand motion accuracy; ablations confirm the contributions of both semantic alignment and hierarchy-aligned generation. 6 Limitations Despite the effectiveness of our current framework, several limitations remain. First, the current autoregressive architecture still generates tokens sequentially, which limits inference parallelism and makes long-sequence generation inefficient. Future work could explore masked, semi-autoregressive, or diffusion-based token generation to improve decoding speed while preserving temporal coherence. Second, although semantic alignment improves hand motion, our current objective does not explicitly supervise linguistic handshape correctness, palm orientation, or sign-level articulatory categories. Since hand motion carries much of the linguistic information in sign language, future models should introduce hand-specific objectives, higher-resolution hand tokenization, or dedicated hand refinement modules. Third, our current representation does not sufficiently model facial expression and emotion, which are essential non-manual components of sign language. Incorporating facial expression parameters, affective cues, and emotion-aware supervision may lead to more expressive and natural signing avatars. Finally, our current setting relies only on text as the input modality. Future work could extend the framework to multi-modal conditions, such as speech, prosody, glosses, reference videos, or signer-specific style prompts, to support more controllable and expressive sign generation. References Cai et al. (2019) X. Cai, T. Xu, J. Yi, J. Huang, and S. Rajasekaran DTWNet: a dynamic time warping network. Advances in neural information processing systems 32. Cited by: §4.1. Duarte et al. (2021) A. Duarte, S. Palaskar, L. Ventura, D. Ghadiyaram, K. DeHaan, F. Metze, J. Torres, and X. Giro-i-Nieto How2sign: a large-scale multimodal dataset for continuous american sign language. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 2735–2744. Cited by: §A.5, Table 13, §1, §2, §4.1. Esser et al. (2021) P. Esser, R. Rombach, and B. Ommer Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 12873–12883. Cited by: §2. Fang et al. (2025) S. Fang, C. Chen, L. Wang, C. Zheng, C. Sui, and Y. Tian Signllm: sign language production large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 6622–6634. Cited by: §1, §2. Fang et al. (2023) S. Fang, C. Sui, X. Zhang, and Y. Tian SignDiff: learning diffusion models for american sign language production. arXiv preprint arXiv:2308.16082. Cited by: §1, §2. Guo et al. (2025a) C. Guo, I. Hwang, J. Wang, and B. Zhou Snapmogen: human motion generation from expressive texts. arXiv preprint arXiv:2507.09122. Cited by: §4.2, Table 1. Guo et al. (2025b) C. Guo, I. Hwang, J. Wang, and B. Zhou Snapmogen: human motion generation from expressive texts. Advances in Neural Information Processing Systems 38, p. 99939–99955. Cited by: §1. Guo et al. (2024) C. Guo, Y. Mu, M. G. Javed, S. Wang, and L. Cheng Momask: generative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 1900–1910. Cited by: §1, §3.1. Jang et al. (2025) Y. Jang, H. Raajesh, L. Momeni, G. Varol, and A. Zisserman Lost in translation, found in context: sign language translation with contextual cues. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8742–8752. Cited by: §1. Jiang et al. (2023) B. Jiang, X. Chen, W. Liu, J. Yu, G. Yu, and T. Chen Motiongpt: human motion as a foreign language. Advances in Neural Information Processing Systems 36, p. 20067–20079. Cited by: §4.2, Table 1. Lee et al. (2022) D. Lee, C. Kim, S. Kim, M. Cho, and W. Han Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 11523–11532. Cited by: §1, §3.1. Li et al. (2025) Z. Li, W. Zhou, W. Zhao, K. Wu, H. Hu, and H. Li Uni-sign: toward unified sign language understanding at scale. arXiv preprint arXiv:2501.15187. Cited by: §1. Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. Cited by: §A.7. Lugaresi et al. (2019) C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C. Chang, M. G. Yong, J. Lee, W. Chang, W. Hua, M. Georg, and M. Grundmann Mediapipe: a framework for building perception pipelines. arXiv preprint arXiv:1906.08172. Cited by: §A.5. Potamias et al. (2025) R. A. Potamias, J. Zhang, J. Deng, and S. Zafeiriou Wilor: end-to-end 3d hand localization and reconstruction in-the-wild. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 12242–12254. Cited by: §A.5. Qi et al. (2024) F. Qi, Y. Duan, C. Xu, and H. Zhang SignGen: end-to-end sign language video generation with latent diffusion. In European Conference on Computer Vision, Cited by: §2. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21 (140), p. 1–67. Cited by: §A.7, §3.2, §3.2. Saunders et al. (2020) B. Saunders, N. C. Camgoz, and R. Bowden Progressive transformers for end-to-end sign language production. In European Conference on Computer Vision, p. 687–705. Cited by: §2. Saunders et al. (2021) B. Saunders, N. C. Camgoz, and R. Bowden Continuous 3d multi-channel sign language production via progressive transformers and mixture density networks. International journal of computer vision 129 (7), p. 2113–2135. Cited by: §2. Saunders et al. (2022) B. Saunders, N. C. Camgoz, and R. Bowden Signing at scale: learning to co-articulate signs for large-scale photo-realistic sign language production. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 5141–5151. Cited by: §1, §2. Stoll et al. (2020) S. Stoll, N. C. Camgoz, S. Hadfield, and R. Bowden Text2Sign: towards sign language production using neural machine translation and generative adversarial networks. International Journal of Computer Vision 128 (4), p. 891–908. Cited by: §2. Sun et al. (2024) Q. Sun, Y. Wang, A. Zeng, W. Yin, C. Wei, W. Wang, H. Mei, C. Leung, Z. Liu, L. Yang, and Z. Cai Aios: all-in-one-stage expressive human pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 1834–1843. Cited by: §A.5, §1. Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §3.3. Uthus et al. (2023) D. Uthus, G. Tanzer, and M. Georg Youtube-asl: a large-scale, open-domain american sign language-english parallel corpus. Advances in Neural Information Processing Systems 36, p. 29029–29047. Cited by: §A.5, §1, §2, §4.1. Wang et al. (2025) C. Wang, Z. Deng, Z. Jiang, Y. Yin, F. Shen, Z. Cheng, S. Ge, S. Gan, and Q. Gu Advanced sign language video generation with compressed and quantized multi-condition tokenization. arXiv preprint arXiv:2506.15980. Cited by: §2. Wong et al. (2024) R. Wong, N. C. Camgoz, and R. Bowden Sign2GPT: leveraging large language models for gloss-free sign language translation. In International Conference on Learning Representations, Cited by: §1. Xie et al. (2024a) P. Xie, T. Peng, Y. Du, and Q. Zhang Sign language production with latent motion transformer. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 3024–3034. Cited by: §2. Xie et al. (2024b) P. Xie, Q. Zhang, P. Taiying, H. Tang, Y. Du, and Z. Li G2p-ddm: generating sign pose sequence from gloss sequence with discrete diffusion model. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 6234–6242. Cited by: §2. Yao et al. (2021) L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu Filip: fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783. Cited by: §3.2. Yin et al. (2024) A. Yin, H. Li, K. Shen, S. Tang, and Y. Zhuang T2S-gpt: dynamic vector quantization for autoregressive sign language production from text. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3345–3356. Cited by: §1, §2. Yu et al. (2024) Z. Yu, S. Huang, Y. Cheng, and T. Birdal Signavatars: a large-scale 3d sign language holistic motion dataset and benchmark. In European Conference on Computer Vision, p. 1–19. Cited by: Table 13, §1, §2. Zelinka and Kanis (2020) J. Zelinka and J. Kanis Neural sign language synthesis: words are our glosses. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, p. 3395–3403. Cited by: §2. Zuo et al. (2026) R. Zuo, R. A. Potamias, Q. Sun, E. Ververas, J. Deng, and S. Zafeiriou MaDiS: taming masked diffusion language models for sign language generation. arXiv preprint arXiv:2601.19577. Cited by: Figure 1, Figure 1, §1, §2, §4.1, §4.2, Table 1, Table 1. Zuo et al. (2025) R. Zuo, R. A. Potamias, E. Ververas, J. Deng, and S. Zafeiriou Signs as tokens: a retrieval-enhanced multilingual sign language generator. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 23806–23816. Cited by: §A.5, Table 13, §1, §2, §4.1, §4.2, Table 1, Table 1. Appendix A Appendix A.1 Additional Controlled Ablation Studies We conduct additional controlled experiments to isolate the contributions of the semantic objectives and analyze the behavior of the residual representation hierarchy. Unless otherwise specified, all compared variants use the same model architecture, data split, training schedule, and evaluation protocol. A.1.1 Controlled Tokenizer-Objective Ablation To isolate the contribution of semantic supervision from the use of residual vector quantization, we train four tokenizer variants under a strictly controlled setting. All variants use the same RVQ architecture, codebooks, decoder, temporal downsampling ratio, reconstruction objective, data split, and training protocol. They differ only in whether the sentence-level alignment objective ℒglobalL_global and the token-level text-conditioned objective ℒtokL_tok are enabled. When enabled, each semantic objective is assigned a weight of 0.0050.005. For every tokenizer, we independently train a Hierarchical GPT from scratch using identical generator settings. Table 6: Controlled tokenizer-objective ablation. All variants share the same RVQ architecture, reconstruction objective, data split, and training protocol. A separate Hierarchical GPT is trained from scratch for each tokenizer under identical settings. Lower is better for PA-MPJPE and DTW-JPE, while higher is better for SiBLEU-4. Tokenizer H2S Recon. B/H YT Recon. B/H H2S Gen. B/H, SiB4 YT Gen. B/H, SiB4 RVQ only 17.65 / 6.85 15.60 / 3.85 6.82 / 9.63, 3.20 5.39 / 7.20, 2.56 RVQ +ℒglobal+L_global 17.05 / 6.55 15.05 / 3.62 6.67 / 9.50, 3.53 5.24 / 7.01, 2.62 RVQ +ℒtok+L_tok 16.65 / 6.35 14.45 / 3.45 6.53 / 9.31, 3.80 5.15 / 6.85, 2.87 Full SeRV 16.23 / 6.13 13.91 / 3.31 6.51 / 9.22, 4.12 4.98 / 6.70, 3.13 As shown in Table 6, both semantic objectives consistently improve reconstruction and generation over the RVQ-only baseline. On How2Sign, adding ℒglobalL_global improves SiBLEU-4 from 3.20 to 3.53, while adding ℒtokL_tok improves it to 3.80. Combining both objectives produces the best SiBLEU-4 score of 4.12 and the lowest body and hand DTW-JPE. The same trend is observed on YouTube-ASL, where Full SeRV obtains the best reconstruction and generation results across all reported metrics. Because the tokenizer architecture and downstream generator protocol are fixed across all variants, these improvements cannot be attributed solely to replacing VQ-VAE with RVQ. Instead, they demonstrate the contribution of semantic supervision to the learned residual motion representation. A.1.2 Effect of Cumulative-Latent Supervision We further investigate which residual representation should receive token-level semantic supervision. Let (ℓ)z^( ) denote the latent contribution of RVQ level ℓ . The cumulative latent up to level k is defined as (≤k)=∑ℓ=0k(ℓ).z^(≤ k)= _ =0^kz^( ). (9) We train otherwise identical tokenizers by applying ℒtokL_tok to the Level-0 latent, the cumulative latent from Levels 0–1, or the full cumulative latent from all RVQ levels. Table 7: Effect of the supervision target for ℒtokL_tok on How2Sign. Applying token-level semantic supervision to progressively more complete cumulative RVQ representations improves both generation and text-to-sign retrieval. Supervision target SiBLEU-4 DTW-JPE B/H T2S R@1 Level 0 3.70 6.62 / 9.45 12.65 Levels 0–1 3.91 6.56 / 9.32 13.80 Full latent 4.12 6.51 / 9.22 14.52 As shown in Table 7, applying supervision to increasingly complete cumulative representations consistently improves semantic retrieval and generation quality. SiBLEU-4 increases from 3.70 with Level-0 supervision to 3.91 with Levels 0–1 and 4.12 with the full latent. T2S R@1 similarly increases from 12.65 to 13.80 and 14.52. This result is consistent with the design of the Hierarchical GPT. Each refinement level is generated conditioned on the text and lower-level motion tokens. Supervising the cumulative latent therefore encourages the representation available at each refinement stage to remain compatible with this conditional generation process. This finding does not imply that later residual levels independently encode word-level semantic alignment. As shown by the per-token temporal analysis in Appendix , Level 0 and the full latent exhibit similar temporal dispersion, suggesting that the later levels mainly refine the representation established by the coarse level. A.1.3 Cumulative Contribution of RVQ Levels To characterize the role of each residual level, we evaluate progressively accumulated latent representations from the trained Full SeRV tokenizer. For each setting, we reconstruct motion and evaluate text-to-sign and sign-to-text retrieval using the cumulative latent available up to the corresponding RVQ level. Table 8: Cumulative contribution of RVQ levels. Level 0 captures most of the cross-modal retrieval signal, while subsequent residual levels progressively improve geometric reconstruction and provide smaller but consistent semantic gains. Representation PA-MPJPE B/H T2S / S2T R@1 Level 0 20.84 / 8.06 13.61 / 7.79 Levels 0–1 17.42 / 6.71 14.18 / 8.16 Full latent 16.23 / 6.13 14.52 / 8.42 Table 8 shows that Level 0 already captures most of the cross-modal retrieval signal, achieving T2S and S2T R@1 scores of 13.61 and 7.79, respectively. Adding subsequent residual levels substantially improves reconstruction, reducing body and hand PA-MPJPE from 20.84/8.0620.84/8.06 to 16.23/6.1316.23/6.13. Retrieval performance also improves gradually to 14.52/8.4214.52/8.42. These results indicate a coarse-to-fine division of labor across the RVQ hierarchy. The first level represents the dominant motion structure and most of the text-related information, whereas subsequent levels primarily refine geometric details while preserving and slightly strengthening cross-modal alignment. A.1.4 Semantic-Loss Weight Sensitivity We analyze the sensitivity of SeRV to the semantic-loss weight λsem _sem. All model components and training settings are fixed while varying only λsem _sem. Table 9: Sensitivity to the semantic-loss weight on How2Sign. The reported weight of 0.0050.005 provides the best downstream generation performance. A smaller value slightly favors reconstruction, whereas an excessively large value degrades both reconstruction and generation. λsem _sem Recon. PA-MPJPE B/H Gen. DTW-JPE B/H SiBLEU-4 0.05 18.71 / 7.02 6.73 / 9.58 3.63 0.005 16.23 / 6.13 6.51 / 9.22 4.12 0.0005 15.77 / 6.05 6.66 / 9.40 3.82 As shown in Table 9, a relatively small weight of 0.00050.0005 slightly favors reconstruction, achieving body and hand PA-MPJPE of 15.77/6.0515.77/6.05. However, it produces weaker downstream generation than the reported setting. The weight λsem=0.005 _sem=0.005 achieves the best generation performance, with body and hand DTW-JPE of 6.51/9.226.51/9.22 and SiBLEU-4 of 4.12. Increasing the weight to 0.050.05 degrades both reconstruction and generation. The results reveal a trade-off between geometric reconstruction and semantic predictability. We therefore use λsem=0.005 _sem=0.005 in the main experiments because it provides the strongest downstream generation performance, rather than selecting the setting with the lowest tokenizer reconstruction error alone. A.2 Human Perceptual Evaluation We conduct a human perceptual evaluation to complement the automatic generation metrics. Five participants independently rank anonymized outputs generated by SOKE, S-MotionGPT, and SeRV for 30 videos. For each example, the corresponding ground-truth signer video is provided as the reference. Participants are instructed to rank the generated motions according to their visual similarity to the reference, considering body movement, hand motion, temporal alignment, and overall motion correspondence. The participants are not fluent ASL signers. Therefore, this experiment evaluates perceptual motion similarity rather than linguistic correctness, sign intelligibility, or grammatical accuracy. Table 10: Human perceptual evaluation results. SeRV is ranked first most frequently and obtains the lowest mean rank. Pairwise preference rates are derived from the valid rankings. Method Ranked first Mean rank SeRV pairwise win SOKE 23.3% 2.19 72.9% S-MotionGPT 17.8% 2.28 74.4% SeRV 58.9% 1.53 – The evaluation contains 5×30=1505× 30=150 collected rankings. After excluding responses containing duplicated ranks, 129 rankings remain valid. As shown in Table 10, SeRV is ranked first in 58.9% of the valid comparisons, compared with 23.3% for SOKE and 17.8% for S-MotionGPT. SeRV also obtains a mean rank of 1.53, substantially better than 2.19 for SOKE and 2.28 for S-MotionGPT. In pairwise comparisons derived from the rankings, SeRV is preferred over SOKE in 72.9% of cases and over S-MotionGPT in 74.4% of cases. These findings indicate that the improvements in automatic metrics are accompanied by stronger perceptual correspondence to the reference motions. A.3 YouTube-ASL Data Statistics and Quality Audit A.3.1 Data Processing Statistics The original YouTube-ASL metadata contains 611,533 clip records. Among them, 561,145 clips are successfully downloaded. We remove 187,880 clips containing multiple visible signers and 136,728 clips for which no reliable signer can be identified or the AiOS reconstruction confidence is below 0.6. This produces 236,537 retained ASL–text pairs. Table 11: Summary of the YouTube-ASL data processing pipeline. Processing stage Number of clips Metadata records 611,533 Successfully downloaded 561,145 Removed: multiple signers 187,880 Removed: unreliable signer or low confidence 136,728 Retained ASL–text pairs 236,537 The retained dataset contains 8,119 source videos and approximately 295.1 hours of signing motion. The number 61,153 reported in an earlier version of Appendix A.2 was a typographical error; the correct number of successfully downloaded clips is 561,145. A.3.2 Human Audit Protocol To assess the quality of the reconstructed 3D motion, four raters independently audit 100 clips randomly sampled from the retained YouTube-ASL pairs, resulting in 400 ratings. For every sample, the reconstructed motion is displayed side-by-side with its source video. Raters evaluate detailed correspondence between the reconstruction and the source video, including hand shape, finger articulation, arm trajectory, and temporal alignment. Each sample is assigned one of four labels: good, partially correct, poor, or uncertain. Table 12: Human audit of reconstructed YouTube-ASL motion. Four raters independently evaluate 100 randomly sampled clips, producing 400 ratings. Rating Percentage Number of ratings Good 85.75% 343 Partially correct 11.25% 45 Poor 2.25% 9 Uncertain 0.75% 3 At least partially correct 97.0% 388 As shown in Table 12, 85.75% of the ratings classify the reconstructions as good, 11.25% as partially correct, 2.25% as poor, and 0.75% as uncertain. Overall, 97.0% of the ratings consider the reconstructed motion to be at least partially correct. After excluding the three uncertain ratings, the average audit score is 1.84 out of 2 when good, partially correct, and poor ratings are assigned scores of 2, 1, and 0, respectively. These results suggest that the filtering procedure removes most severe reconstruction failures. Nevertheless, the retained dataset is not noise-free. Remaining errors primarily involve fine-grained finger articulation, rapid or occluded hand motion, arm trajectories, temporal alignment, and motion smoothness. Representative successful, partially correct, and failed reconstruction examples are provided in Fig. . A.4 Comparison with Existing 3D Sign Resources Recent 3D sign language resources have made important progress toward avatar-based sign language understanding and generation. Early 3D ASL resources such as How2Sign provide aligned videos, depth, and English transcripts, but their scale and signer coverage remain limited for training high-capacity text-to-motion generators. More recent SLG systems further curate SMPL-X poses for existing continuous sign language benchmarks, including How2Sign, CSL-Daily, and Phoenix-2014T, enabling token-based 3D sign generation across multiple languages. In parallel, SignAvatars introduces a large-scale holistic 3D sign language dataset with SMPL-X, MANO, and 2D/3D keypoint annotations, covering both isolated and continuous signs with multiple prompt types. However, existing resources are either relatively limited in ASL scale, constructed from studio-recorded benchmarks, or designed for general multi-prompt/multilingual 3D sign production rather than large-scale paired text-to-3D ASL generation from in-the-wild videos. Table 13 summarizes the comparison. Our processed YouTube3D dataset contains 236,537 ASL–text pairs from 8,119 in-the-wild videos, totaling 295.1 hours of signing motion. Table 13: Comparison with existing 3D sign language motion resources. “Text paired” indicates whether spoken-language text is paired with the 3D motion. Dataset Lang. Clips / Pairs Duration Text Paired How2Sign-3D Duarte et al. (2021) ASL 35K 80+h ✓ SOKE Curated 3D Poses Zuo et al. (2025) ASL / CSL / DGS 35K / 20K / 8K – ✓ SignAvatars Yu et al. (2024) Multi-SL 70K 117h ✓ Reconstructed YouTube-ASL (Ours) ASL 236,537 295.1h ✓ A.5 Data Preparation Existing 3D ASL datasets Duarte et al. (2021) remain limited in scale, signer diversity, vocabulary coverage, and recording duration, making it difficult to train high-capacity generative models that generalize well. To support scalable 3D ASL generation, we build a reconstructed 3D motion–text dataset from YouTube-ASL Uthus et al. (2023), a large-scale in-the-wild ASL video corpus with English text annotations. For privacy protection and to respect the rights of video owners, YouTube-ASL releases YouTube video IDs rather than raw video files. We therefore download the available videos using the released IDs. In total, 61,153 available video clips are successfully downloaded and used as the initial pool for 3D motion reconstruction. Following prior work on in-the-wild 3D sign motion reconstruction Zuo et al. (2025); Sun et al. (2024); Potamias et al. (2025), we recover expressive 3D signing motion from RGB videos, including upper-body pose, hand articulation, and facial expression. Each reconstructed motion sequence is represented as ∈ℝT×dS ^T× d, where T denotes the sequence length and d=133d=133 is the dimensionality of the SMPL-X motion parameters, consisting of 3D rotations for 11 upper-body joints and 30 hand joints, plus 10 facial expression coefficients. Each reconstructed ASL motion sequence is paired with its corresponding text transcript. To improve data quality, we apply a multi-stage filtering pipeline. First, we use MediaPipe-based person detection Lugaresi et al. (2019) to remove clips containing multiple visible signers, since multi-signer clips make it ambiguous which person should be paired with the transcript. Second, we discard clips in which no reliable signer can be detected. Third, after 3D reconstruction, we filter out samples whose reconstruction confidence is below a predefined threshold, which removes clips with severe reconstruction failures, poorly tracked hands, or unreliable body estimates. The remaining samples are normalized into a consistent body representation and coordinate system. After downloading, reconstruction, filtering, and normalization, our processed YouTube3D dataset contains 236,537 ASL–text pairs from 8,119 videos, totaling 295.1 hours of signing motion. This pipeline converts large-scale in-the-wild ASL videos into structured paired 3D motion–text data, providing broader signer diversity, richer motion variation, and larger linguistic coverage for training both the semantic-aligned tokenizer and the hierarchical autoregressive generator. A.6 Evaluation Metric Definitions We provide formal definitions of the metrics used in Section 4.1. Notation. Let ^=^ii=1T S=\ p_i\_i=1 T denote a generated sign motion sequence and =jj=1TS=\p_j\_j=1^T the corresponding reference, where each frame contains J 3D joints and (k)∈ℝ3p^(k) ^3 is the k-th joint position. The per-frame joint position error between generated frame i and reference frame j is d(i,j)=1J∑k=1J‖^i(k)−j(k)‖2.d(i,j)= 1J _k=1^J \| p_i^(k)-p_j^(k) \|_2. (10) DTW-JPE. Since the generated and reference sequences may differ in length, we align them with dynamic time warping. Let Π be the set of monotonic warping paths π=(i,j)π=\(i,j)\ satisfying the standard boundary, monotonicity, and continuity constraints. DTW selects the path with minimum cumulative cost, π⋆=argmin∑(i,j)∈π∈Πd(i,j),π = _π∈ _(i,j)∈πd(i,j), (11) and DTW-JPE is the cost along the optimal path normalized by its length: DTW-JPE=1|π⋆|∑(i,j)∈π⋆d(i,j).DTW -JPE= 1|π | _(i,j)∈π d(i,j). (12) We compute Eq. (10) over the body, left-hand, and right-hand joint sets separately, and report Hand Avg. as the mean of the left- and right-hand errors. PA-MPJPE. For tokenizer reconstruction the output is frame-aligned with the input (T^=T T=T), so no temporal warping is required. Before computing the error, we apply a rigid Procrustes alignment that estimates the optimal rotation R, translation t, and scale s minimizing the residual between the two frames: (⋆,⋆,s⋆)=argmin∑k=1J,,s‖s^t(k)+−t(k)‖22.(R ,t ,s )= _R,t,s _k=1^J \|s\,R\, p_t^(k)+t-p_t^(k) \|_2^2. (13) PA-MPJPE is then the per-joint error averaged over all frames after alignment: PA-MPJPE=1T∑t=1T1J∑k=1J‖s⋆⋆^t(k)+⋆−t(k)‖2.PA -MPJPE= 1T _t=1^T 1J _k=1^J \|s R p_t^(k)+t -p_t^(k) \|_2. (14) SiBLEU-4. Let (⋅)T(·) be a frozen evaluator tokenizer mapping a motion sequence to a discrete token sequence. Denote the generated and reference token sequences as ^=(^) u=T( S) and =()u=T(S). SiBLEU-4 is the BLEU-4 score between them: SiBLEU-4=BP⋅exp(∑n=14wnlogpn),SiBLEU -4=BP· \! ( _n=1^4w_n p_n ), (15) where pnp_n is the modified n-gram precision over token sequences, wn=14w_n= 14 are uniform weights, and BPBP is the brevity penalty BP=1,c>r,exp(1−r/c),c≤r,BP= cases1,&c>r,\\[2.0pt] \! (1-r/c ),&c≤ r, cases (16) with c and r the total lengths of the generated and reference token sequences, respectively. A higher SiBLEU-4 indicates better token-level consistency. A.7 Implementation details. We train the framework in two stages: SeRV tokenizer training followed by Hierarchical GPT training. In the first stage, the SeRV tokenizer is trained on the training split with the RVQ reconstruction objective and two semantic alignment objectives: the global-level motion–text alignment loss and the token-level motion–text late-interaction loss. Unless otherwise specified, we use L=3L=3 residual quantization levels, a latent dimension of C=512C=512, and a codebook size of K=512K=512 at each level. The motion encoder downsamples the input sequence by a factor of 44, so each discrete token corresponds to four motion frames. For semantic supervision, we use a frozen T5 encoder Raffel et al. (2020) to encode the paired transcript. The mean-pooled T5 hidden states are used as sentence-level text embeddings for the global alignment loss, where they are aligned with the temporally pooled level-0 quantized motion representation. The token-level T5 hidden states are used in the late-interaction loss, where they are matched with the fully quantized motion latent sequence. The SeRV training objective is ℒSeRV=ℒRVQ+λglobalℒglobal+λtokℒtokL_SeRV=L_RVQ+ _globalL_global+ _tokL_tok. We optimize the tokenizer with AdamW Loshchilov and Hutter (2017) for 500500 epochs using a batch size of 128128, an initial learning rate of 4×10−44× 10^-4, a weight decay of 0.10.1, and cosine learning-rate decay. In the second stage, Hierarchical GPT is trained to autoregressively predict the residual token hierarchy produced by the frozen SeRV tokenizer. Given frozen T5 text features, the generator predicts tokens over time and RVQ depth in a coarse-to-fine order. This preserves the residual hierarchy learned by SeRV instead of flattening all residual tokens into a single sequence. Coarse Motion Decoder. The level-0 decoder predicts coarse motion tokens from the text condition and previous temporal history. Since the level-0 RVQ code carries the dominant motion structure, this decoder is responsible for generating the coarse text-conditioned signing trajectory. We implement it as a causal Transformer with LLaMA-style blocks, including RMSNorm, SwiGLU feed-forward layers, rotary positional embeddings, and prefix-based conditioning on the T5 text features. The T5 token features are projected to the generator hidden dimension and prepended as conditioning tokens, allowing the decoder to autoregressively predict coarse motion tokens while attending to the transcript representation. Residual Refinement Decoder. Higher-level tokens are predicted by residual refinement decoders. Each residual decoder predicts refinement tokens conditioned on the text feature, previous temporal tokens, and the lower-level tokens already generated at the current time step. Rather than generating motion from scratch, these decoders refine the representation produced by lower RVQ levels. Concretely, the level-1 decoder is conditioned on the level-0 token embeddings, while the level-2 decoder is conditioned on the additive coarse context formed by level-0 and level-1 token embeddings. This design follows the additive structure of RVQ, where later codebooks encode residual details over the coarse motion state. The generator uses a hidden dimension of 10241024, 1616 attention heads, a dropout rate of 0.10.1, and a maximum token length of 100100. The coarse motion decoder contains 99 Transformer layers, while the two residual refinement decoders contain 66 and 33 layers, respectively. We train the generator for 150150 epochs with AdamW, using a batch size of 1616, a learning rate of 2×10−42× 10^-4, a weight decay of 0.010.01, and cosine learning-rate decay. At inference time, tokens are generated in the same coarse-to-fine order until an EOS token is predicted or the maximum length is reached. We use autoregressive sampling with a temperature of 1.01.0. The predicted multi-level token streams are decoded by the frozen SeRV decoder into continuous SMPL-X motion features, which are then converted into joints or vertices for evaluation. All experiments are conducted on three NVIDIA L40S GPUs. A.8 Additional Qualitative Results for Tokenizer Reconstruction Figure 5: Qualitative comparison of tokenizer reconstruction results. Our tokenizer preserves motion details more effectively, especially for hand articulations. A.9 Additional Qualitative Results on Ablation Study Figure 6: Qualitative comparison of generated 3D ASL motions on two examples. For each example, the top row shows the baseline and the bottom row shows SeRV. With semantic-aligned RVQ, SeRV better captures hand information and produces more faithful signing motions.