Paper deep dive
SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMs
Shiwei Gan, Xiao Liu, Yafeng Yin, Zhiwei Jiang, Bowen Guo, Lie Xie, Sanglu Lu, Hongkai Wen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/12/2026, 3:12:12 AM
Summary
The paper introduces SignLlama, a model for Gloss-Free Sign Language Translation (GFSLT) that adapts Large Language Models (LLMs) by addressing the distributional gap between visual and textual inputs. It proposes Filtered Pseudo-Gloss CTC Pretraining (FPG-CTC) to align visual features with text and Visual-Prioritized Distillation (VPD) to ensure the model relies more on visual cues than textual priors during generation.
Entities (10)
Relation Signals (9)
SignLlama → usesmethod → Filtered Pseudo-Gloss CTC Pretraining
confidence 95% · we propose a simple yet effective method named Filtered Pseudo-Gloss CTC Pretraining... To address the first challenge
SignLlama → usesmethod → Visual-Prioritized Distillation
confidence 95% · we introduce a Visual-Prioritized Distillation training strategy... To tackle the second issue
SignLlama → solvestask → Gloss-Free Sign Language Translation
confidence 92% · The proposed SignLlama achieves very competitive performance on multiple datasets for GFSLT tasks
SignLlama → evaluatedon → OpenASL
confidence 90% · We evaluate our model on several publicly available SLT datasets, including... OpenASL
SignLlama → evaluatedon → CSL-Daily
confidence 90% · We evaluate our model on several publicly available SLT datasets, including... CSL-Daily
SignLlama → evaluatedon → PHOENIX14T
confidence 90% · We evaluate our model on several publicly available SLT datasets, including the widely used Phoenix14T
SignLlama → evaluatedon → How2Sign
confidence 90% · We evaluate our model on several publicly available SLT datasets, including... How2Sign
Filtered Pseudo-Gloss CTC Pretraining → usesloss →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) have achieved remarkable success across a wide range of tasks. However, fine-tuning LLMs for Gloss-Free Sign Language Translation (GFSLT) remains a challenge. In this paper, we investigate how to effectively adapt LLMs to the GFSLT task. We show that there are two key issues that need to be solved: (1) the inherent distributional gap between visual feature inputs and text feature inputs makes it difficult for LLMs to interpret visual inputs; and (2) existing approaches typically concatenate visual and textual features in an autoregressive framework, which leads to the model overemphasizing textual inputs and deprioritizing visual cues, as LLMs are pretrained predominantly on text-centric data. To address the first challenge, we propose a simple yet effective method named Filtered Pseudo-Gloss CTC Pretraining, which leverages filtered pseudo-gloss sequences generated from text sequences to supervise the training of the visual backbone. To tackle the second issue, we introduce a Visual-Prioritized Distillation training strategy. Specifically, we define a visual-only prediction path in which text inputs are masked, and the model is required to generate the target sequence relying solely on visual inputs. To guide this path, the outputs from the standard visual-textual prediction are then distilled into the visual-only prediction path, encouraging the model to prioritize visual features. Comprehensive experiments and qualitative analyses demonstrate the effectiveness of the proposed model. The proposed SignLlama achieves very competitive performance on multiple datasets for GFSLT tasks, without using any extra modalities or external sign language datasets for pretraining.
Tags
Links
- Source: https://arxiv.org/abs/2608.09006v1
- Canonical: https://arxiv.org/abs/2608.09006v1
Trouble viewing inline? Open PDF directly →
Full Text
61,600 characters extracted from source content.
Expand or collapse full text
SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMs Shiwei Gan ∗ State Key Laboratory of Novel Software Technology, Nanjing University Nanjing, Jiangsu, China sw@nju.edu.cn Xiao Liu ∗ State Key Laboratory of Novel Software Technology, Nanjing University Suzhou, Jiangsu, China liuxiaox@smail.nju.edu.cn Yafeng Yin † State Key Laboratory of Novel Software Technology, Nanjing University Suzhou, Jiangsu, China yafeng@nju.edu.cn Zhiwei Jiang State Key Laboratory of Novel Software Technology, Nanjing University Suzhou, Jiangsu, China jzw@nju.edu Bowen Guo State Key Laboratory of Novel Software Technology, Nanjing University Suzhou, Jiangsu, China 602024720001@smail.nju.edu.cn Lei Xie State Key Laboratory of Novel Software Technology, Nanjing University Nanjing, Jiangsu, China lxie@nju.edu.cn Sanglu Lu State Key Laboratory of Novel Software Technology, Nanjing University Nanjing, Jiangsu, China sanglu@nju.edu.cn Hongkai Wen University of Warwick Coventry, United Kingdom hongkai.wen@warwick.ac.uk Abstract Large Language Models (LLMs) have achieved remarkable success across a wide range of tasks. However, fine-tuning LLMs for Gloss- Free Sign Language Translation (GFSLT) remains a challenge. In this paper, we investigate how to effectively adapt LLMs to the GFSLT task. We show that there are two key issues that need to be solved: (1) the inherent distributional gap between visual feature inputs and text feature inputs makes it difficult for LLMs to interpret visual inputs; and (2) existing approaches typically concatenate visual and textual features in an autoregressive framework, which leads to the model overemphasizing textual inputs and deprioritizing visual cues, as LLMs are pretrained predominantly on text-centric data. To address the first challenge, we propose a simple yet effective method named Filtered Pseudo-Gloss CTC Pretraining, which lever- ages filtered pseudo-gloss sequences generated from text sequences to supervise the training of the visual backbone. To tackle the sec- ond issue, we introduce a Visual-Prioritized Distillation training strategy. Specifically, we define a visual-only prediction path in which text inputs are masked, and the model is required to generate the target sequence relying solely on visual inputs. To guide this path, the outputs from the standard visual-textual prediction are then distilled into the visual-only prediction path, encouraging the model to prioritize visual features. Comprehensive experiments and qualitative analyses demonstrate the effectiveness of the proposed model. The proposed SignLlama achieves very competitive perfor- mance on multiple datasets for GFSLT tasks, without using any extra modalities or external sign language datasets for pretraining. ∗ Both authors contributed equally to this research. † Yafeng Yin is the corresponding author. This work is licensed under a Creative Commons Attribution-NonCommercial- NoDerivatives 4.0 International License. M ’26, Rio de Janeiro, Brazil © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2213-4/2026/11 https://doi.org/10.1145/3767308.3835002 CCS Concepts • Computing methodologies→Activity recognition and un- derstanding; Natural language generation;• Human-centered computing→ Accessibility technologies. Keywords Sign Language Translation, Large Language Models ACM Reference Format: Shiwei Gan, Xiao Liu, Yafeng Yin, Zhiwei Jiang, Bowen Guo, Lei Xie, Sanglu Lu, and Hongkai Wen. 2026. SignLlama: Enhancing Gloss-free Sign Lan- guage Translation by Prioritizing Visual Features for LLMs. In Proceedings of the 34th ACM International Conference on Multimedia (M ’26), November 10–14, 2026, Rio de Janeiro, Brazil. ACM, New York, NY, USA, 15 pages. https://doi.org/10.1145/3767308.3835002 1 Introduction Current sign language (SL) understanding tasks mainly include Continuous SL Recognition (CSLR) [11] and SL Translation (SLT). CSLR aims to recognize a sequence of signs and convert it into the corresponding gloss sequence, while SLT [14,38] focuses on trans- lating sign sequences into spoken language. As SLT produces more natural and fluent language outputs, it has attracted increasing at- tention in recent research. The de facto architecture for current SLT tasks typically employs a 2D or 3D CNN-based backbone to extract visual features [12], followed by temporal modeling modules (e.g., 1D CNNs or LSTMs) to capture dynamic changes across sign frames. Finally, a translation model (e.g., mBART [9,10,15], GPT-2 [44]) is used to generate the corresponding spoken language sentence. SLT research has branched into two distinct paths: gloss-based SLT (GBSLT) and gloss-free SLT (GFSLT). Previous state-of-the-art (SOTA) GBSLT models [4,12] have emphasized that pretraining the visual encoder with gloss labels is critical for achieving strong SLT performance. Due to the reliance on gloss labels, SLT has in- creasingly shifted its focus towards GFSLT [15,23,44], which aims to boost SLT performance without gloss supervision. Meanwhile Large Language Models (LLMs) have revolutionized the field of natural language processing, setting new benchmarks arXiv:2608.09006v1 [cs.CV] 10 Aug 2026 M ’26, November 10–14, 2026, Rio de Janeiro, BrazilShiwei Gan et al. across a broad spectrum of tasks [1,16,18,29]. Benefiting from the vast pretraining datasets, LLMs demonstrate extraordinary gener- alization capabilities, and as a result, fine-tuning LLMs for down- stream tasks has emerged as a prevailing trend [29,33]. However, adapting LLMs for the GFSLT task remains understudied [17,44], primarily due to the following two key challenges. 1) There exists a fundamental distributional gap between textual features and SL video features. Textual inputs, which are the primary input modal- ity during LLM pretraining, are typically tokenized into discrete, symbolic units, forming clearly segmented sequences. In contrast, SL video features are inherently continuous, with subtle variations between consecutive frames. This discrepancy in representation poses a significant challenge in effectively tokenizing SL videos and providing suitable inputs for LLMs, which is crucial for fine- tuning LLMs for GFSLT. 2) Existing Vision-Language Models (e.g., MiniGPT-4 [1]) and LLM-based SLT models [23] typically adopt a straightforward strategy by concatenating visual and textual fea- tures directly in an autoregressive manner. However, this approach often leads to the deprioritization of visual features and biases the model toward fitting the distribution of text tokens rather than visual ones [45], as LLMs are mainly pretrained on text-centric data, and visual inputs constitute an out-of-distribution modality. To tackle the first challenge, a common approach in vision- language models (VLMs) is to align visual and textual features by employing a pre-trained image encoder and leveraging large-scale image-text datasets with objectives such as Image-Text Contrastive (ITC) loss and Image-Text Matching (ITM) loss [30,47]. In addi- tion, some works have explored using attention mechanisms [48], vector quantization [17], and pseudo glosses [15,44] to tokenize SL videos. As for the second challenge, the effectiveness of con- catenating visual and textual features in an autoregressive way remains underexplored. CAL [45] addresses this by focusing on reweighting text tokens based on their visual relevance, thereby enhancing the alignment between modalities. However, it over- looks the enhancement of visual representations themselves. In contrast, our approach aims to prioritize the contribution of visual features during prediction, ensuring that the model more effectively leverages visual information. In this paper, we focus on the Gloss-Free SLT task by adopting open-source LLMs (e.g., the Llama family [40]). First, inspired by recent works [15,17] and GBSLT approaches where CTC loss is commonly used to pretrain the visual encoder, we propose a simple yet effective method to bridge the distributional gap between video and textual features, namely filtered pseudo-gloss CTC Pretrain- ing. Specifically, we generate filtered pseudo-gloss sequences from the target text by applying lemmatization, removing prepositions and conjunctions, and randomly dropping tokens. The resulting pseudo-gloss sequences are then used with CTC loss to pretrain the visual encoder. To further prioritize the model’s reliance on visual features, we propose Vision-Prioritized Distillation (VPD). This training strategy distills knowledge from a ‘Visual-Textual Prediction’ path into a ‘Visual-Only Prediction’ path, enabling the LLMs to generate text with greater reliance on visual features. By prioritizing visual features, VPD can boost the ability of text gen- eration from visual inputs, thereby improving the performance of GFSLT models. We make the following contributions: •We introduce a simple yet effective pretraining method, Fil- tered Pseudo-gloss CTC Pretraining (FPG-CTC), which generates filtered pseudo-gloss sequences to supervise the visual encoder. We show that FPG-CTC can effectively dis- cretize and align SL features, leading to substantial perfor- mance improvements on the GFSLT task. • We propose a Vision-Prioritized Distillation (VPD) train- ing method, which distills the ‘Visual-Textual Prediction’ into the ‘Visual-Only Prediction’ to enhance GFSLT perfor- mance. The proposed VPD can boost text generation and alleviate the exposure bias by prioritizing visual features. •We contribute an array of pre-trained SignLlama models to the community, ranging from 1 billion to 13 billion param- eters. Extensive experiments on public SL datasets demon- strate the effectiveness of our models, which achieve very competitive performance on the GFSLT task. 2 Related Work LLMs for Visual Understanding. Open-source LLMs have revo- lutionized a wide range of downstream tasks, such as image un- derstanding [3,21,26,43] and video understanding [46]. These methods typically use a pretrained visual backbone (e.g., ViT [7]) to extract image or video features, which are then concatenated with textual inputs and fed into LLMs for text generation. The train- ing of these models typically follows a two-stage paradigm [22]. First, they are tuned on large-scale video-text data to align the visual features with textual features. The second stage involves instruction tuning with data such as videos with descriptions and question-answering pairs. Such a paradigm allows the visual back- bone to effectively tokenize images or videos and provide suitable inputs for LLMs. Although these methods have achieved promis- ing results, transferring these pretrained backbones directly to SL tasks remains challenging. Unlike visual understanding tasks that often focus on scene-level changes, SL tasks involve much finer- grained visual cues, such as subtle hand movements, hand shapes, and facial expressions, which are often overlooked in general vi- sual understanding. Moreover, a key limitation lies in the common practice of concatenating video and text features for autoregressive fine-tuning, which often leads the model to over-rely on textual features, deprioritize visual information, and ultimately degrade model performance. Sign Language Tasks. Current approaches in CSLR [5,11,55] and SLT [13,27,41,53] have established a de facto architecture, which typically involves employing a visual module to capture vi- sual features, followed by a temporal module (e.g., 1D CNNs, LSTM, or transformer layers) to model temporal dependencies. For CSLR tasks, a CTC loss is applied to compute the probabilities over all possible alignment paths. For SLT tasks, an additional translation model is employed to generate textual sequences from video fea- tures. Consequently, current CSLR research primarily focuses on improving the modeling of SL-related visual features and temporal relationships by designing various visual backbones (e.g., VAC [31], TwoStream [4], SignGraph [11]and MixSignGraph [15]) and tempo- ral modules [51,54]. Benefiting from CSLR, gloss-based SLT [12,13] leverages pretrained backbones from the CSLR task and incorpo- rates a translation model [28,34] to generate text sequences. These SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMsMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Visual Encoder Text Embedding Llama amtag... (LoRa) ... 풗 ퟏ 풗 ퟎ 풗 ퟎ 풗 ퟏ 풕 ퟎ 풕 ퟏ 풕 ퟎ 풕 ퟏ 풕 ퟐ 풔 풗 ퟏ 풗 ퟎ 풗 ퟎ 풗 ퟏ 풕 ퟎ 풕 ퟏ 풕 ퟎ 풕 ퟏ 풕 ퟐ Visual-TextualMask 풔 풗 ퟎ 풗 ퟏ 풕 ퟎ 풕 ퟏ 풕 ퟐ CE Loss Visual-Only Mask Distillation ...istessonnigimlaufedes tageskom meneinigewolkendazu. (a) SignLlamaArchitecture(b) Visual-Prioritized Distillation 풔 풔 풗 ퟎ 풗 ퟏ 풕 ퟎ 풕 ퟏ 풕 ퟐ CE Loss Figure 1: The proposed SignLlama with Visual-Prioritized Distillation pretrained backbones effectively align SL video features with gloss sequences, functioning as a tokenizer for SL videos. However, in GF- SLT, a key challenge lies in how to effectively tokenize SL videos and align visual features with textual features to provide suitable inputs for LLMs, which is crucial for fine-tuning LLMs for SLT. Existing GFSLT models typically adopt attention mechanisms [48] or con- trastive language-image pretraining [20,23] to enhance alignment and improve performance. Recently, SignLLM [17] attempted to address this by training two discrete codebooks to align sign video features with text. There are also some works focusing on designing pseudo gloss [9,15] to pretrain their backbone, while our methods provides a simpler, more straightforward, and more effective way to pretrain SL backbones. Nonetheless, these methods primarily focus on how to effectively tokenize SL video inputs. In addition, recent GFSLT models incorporating LLMs also typically adopt the same strategy, i.e., , concatenating video and textual features as input for autoregressive fine-tuning. However, this approach tends to prioritize textual features over visual ones. In this paper, we aim to develop a simple and effective training paradigm for effectively tokenizing SL video, along with a strategy that enhances the GFSLT model performance by prioritizing visual features. 3 Method Overall Framework. For an SL video with휃frames푓=푓 푖 휃 푖=1 , the goal of GFSLT is to generate푡=푡 푖 휍 푖=1 with휍words based on input푓. As illustrated in Figure 1, our model consists of a visual encoder and a Llama model. Note that the text embedding module is part of Llama. We show it explicitly here for clarity. The visual encoderVEprocesses frames to get visual features푣=VE(푓)with 푚vectors, where푣= 푣 푖 ∈ R 푑 퐼 푚 푖=1 (푚 ≤ 휃) and푑 퐼 is the input dimension of Llama. The tokenizer converts the target text into푛 token IDs푢 푖 푛 푖=1 , and the text embedding module maps them to text embeddings푒=푒 푖 ∈ R 푑 퐼 푛 푖=1 . (푛 ≥ 휍). During training stage, the model predicts token ˆ 푢 푖 based on푣and previous ground truth tokens:푝( ˆ 푢 푖 |푣,푢 푗 푖−1 푗=0 ). During inference, the model predicts token ˆ 푡 푖 based on 푣 and previous predicted tokens: 푝( ˆ 푢 푖 |푣, ˆ 푢 푗 푖−1 푗=0 ). 3.1 Filtered Pseudo-gloss CTC Pretraining For GBSLT [4,15], the visual encoder is typically trained with the CSLR task, in which CTC loss is used to optimize the visual encoder to learn semantic segmentation and alignment with gloss sequences 푔. A translation model is then integrated and fine-tuned for SLT, taking the output of the encoder as its input. Let푅denote the recog- nition network (with the visual encoder as a subcomponent) and 푇denote the translation network, whileΘ 푅 andΘ 푇 denote the pa- rameters of the recognition and translation networks, respectively. L 퐶푇퐶 andL 퐶퐸 denote the CTC loss and the cross-entropy loss. The hyperparameters훼and훽are used to balance the two objectives. The training process can be formalized as: min Θ 푅 L 퐶푇퐶 (푅(푓),푔)(1) min Θ 푅 ,Θ 푇 (훼L 퐶푇퐶 (푅(푓),푔)+ 훽L 퐶퐸 (푇(푣),푡))(2) For gloss-free SLT, the absence of gloss annotations makes it challenging to pretrain the visual encoder. Unlike previous pseudo gloss based GFSLT models that adopt contrastive learning for vi- sual backbone pretraining, we aim to develop a simple yet effective method that enables the visual encoder to effectively tokenize SL videos and align visual features with textual representations. We draw inspiration from previous work [15] and propose the Filtered Pseudo-gloss CTC Pretraining (FPG-CTC) method. Specifically, we generate a pseudo-gloss sequence푔 푝 from the target text by apply- ing lemmatization and filtering out prepositions and conjunctions. Considering that certain words in the text may not be explicitly expressed in the corresponding SL video, we further apply random dropping to remove a fixed proportion of words, resulting in the M ’26, November 10–14, 2026, Rio de Janeiro, BrazilShiwei Gan et al. final pseudo-gloss sequence푔 푝 . The FPG-CTC training process can be formalized as: min Θ 푅 L 퐶푇퐶 (푅(푓),푔 푝 )(3) min Θ 푅 ,Θ 푇 (훼L 퐶푇퐶 (푅(푓),푔 푝 )+ 훽L 퐶퐸 (푇(푣),푡))(4) 3.2 Visual-Prioritized Distillation Training for LLM Fine-Tuning After obtaining visual features, a widely adopted approach is to con- catenate visual and textual features directly, followed by next-token prediction, which we refer to as ‘Visual-Textual Prediction’, for- malized as:푝( ˆ 푡 푖 |푣,푡 푗 푖−1 푗=0 ). However, this training paradigm suffers from a critical limitation: visual deprioritization. LLMs tend to pri- oritize fitting the distribution of text tokens over that of the newly introduced visual tokens. This is because LLMs are pretrained on text-centric data, and visual inputs constitute an out-of-distribution modality, leading to the deprioritization of visual information dur- ing generation. During inference, the model no longer has access to ground-truth tokens푡 푗 푖−1 푗=0 and instead conditions on its own predictions:푝( ˆ 푡 푖 |푣, ˆ 푡 푗 푖−1 푗=0 ). This mismatch between training and testing leads to cumulative errors. When the model is overly reliant on generated text, errors in early predictions can quickly accumu- late, leading to degraded performance. To encourage the LLMs to prioritize visual features, we propose a novel training paradigm termed Visual-Prioritized Distillation. Specifically, we perform two separate forward passes through the LLM: one for Visual-Textual Prediction, where visual and textual features are concatenated as input; and another for Visual-Only Prediction, where the textual inputs are masked out using a ‘Visual- Only Mask’ (as illustrated in Figure 1 (b)), and the model is required to generate the full target sequence relying solely on visual in- puts. To ensure that the ‘Visual-Only Prediction’ path can converge and perform well, we distill the logits from the ‘Visual-Textual Prediction’ path into the ‘Visual-Only Prediction’ one. This dis- tillation guides the learning process of the visual-only prediction path, encouraging it to align with the output distribution of the visual-textual prediction path while learning to rely exclusively on visual features. The training loss can be formalized as: L 푉푃퐷 =L 푣푡 푇 +L 푣 푇 +훾L 퐾퐷 = 푛 ∑︁ 푖=1 L 퐶퐸 (푇(푣,푡 푗 푖−1 푗=0 ),푡 푖 ) + 푛 ∑︁ 푖=1 L 퐶퐸 (푇(푣),푡 푖 )+ 휆KL(푇(푣,푡)||푇(푣)) (5) where훾is a hyperparameter and푛denotes the number of tokens. L 푣푡 푇 ,L 푣 푇 are cross-entropy losses of ‘Visual-Textual Prediction’ and ‘Visual-Only Prediction’, andL 퐾퐷 (푇(푣,푡),푇(푣))denotes the distillation loss. Training and Inference for GFSLT. The final training process of our model begins with pretraining the visual backbone using pseudo-gloss supervision, as described below. L 푅 = min Θ 푅 L 퐶푇퐶 (푅(푓),푔 푝 )(6) The entire model is then fine-tuned using the VPD objective to- gether with the FPG-CTC loss. min Θ 푅 ,Θ 푇 (훼L 푅 + 훽(L 푣푡 푇 +L 푣 푇 + 휆L 퐾퐷 ))(7) During inference, the visual encoder takes the SL video as input and extracts SL video features, which are then fed into the Llama model. The Llama model follows the standard autoregressive process to generate the target text sequence. Neither pseudo-gloss supervision nor the VPD training procedure is required during inference. 4 Theoretical Insights into VPD The difference between VPD and Knowledge Distillation. Although VPD shares a superficial similarity with knowledge distillation (KD), it is conceptually distinct in both purpose and design. The key nov- elty lies in how VPD prioritizes visual features between visual-only and visual-textual paths, mitigating the tendency of the AR model to over-rely on textual cues and underutilize visual information, which is a critical bottleneck that has not been addressed in prior work. Thus, VPD is not merely an adaptation of KD, but a novel mechanism that explicitly prioritizes visual features in SLT. The theoretical intuition of VPD. Here, we also provide a brief theoretical intuition of VPD at the gradient level. Let 푣 denote the visual input encoded by parameters휃 푣 ,푡the optional textual input, 푇(푣,푡)the teacher (visual-textual path) with logits푧 푣푡 and softmax probabilities푝 푣푡 , and푆(푣)the student (visual-only path) with logits 푧 푣 and softmax probabilities 푝 푣 . The simplified VPD loss is: L= L 푣푡 |z visual-textual CE + L 푣 |z visual-only CE +훾 L 퐾퐷 |z distillation (KL) ,(8) where L 푣푡 =− ∑︁ 푖 푦 푖 log푝 푣푡,푖 ,(9) L 푣 =− ∑︁ 푖 푦 푖 log푝 푣,푖 ,(10) L 퐾퐷 = KL(푝 푣푡 ∥ 푝 푣 )= ∑︁ 푖 푝 푣푡,푖 log 푝 푣푡,푖 푝 푣,푖 .(11) Where y denote the one-hot target distribution. We are interested in the gradient ofL with respect to 휃 푣 : 휕L 휕휃 푣 = 휕L 푣푡 휕휃 푣 + 휕L 푣 휕휃 푣 | z CE gradients +훾 ∑︁ 푖 (푝 푣,푖 − 푝 푣푡,푖 ) 휕푧 푣,푖 휕휃 푣 .(12) Intuitively, when the teacher assigns a higher probability to the correct class, the KL term provides an additional corrective gradient that pushes푝 푣 toward푝 푣푡 , supplementing the supervision provided by the visual-only CE loss. This provides an additional training signal to the visual encoder, encouraging the visual-only path to produce predictions closer to those of the visual-textual path. 5 Experiments Datasets. We evaluate our model on several publicly available SLT datasets, including the widely used Phoenix14T and CSL-Daily, as well as two large-scale ASL datasets: How2Sign and OpenASL. (1) Phoenix14T [2] is a German Sign Language dataset annotated SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMsMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Table 1: Effects of our training strategy in Phoenix14T. FPG-CTC VPD TEST ROUGE BLEU1 BLEU2 BLEU3 BLEU4 20.0918.128.566.465.46 ✓49.9349.2236.5328.6123.47 ✓ ✓ 52.8152.5440.9632.8726.74 Table 2: Effects of VPD loss in Phoenix14T. L 푣푡 푇 L 푣 푇 L 퐾퐷 TEST ROUGE BLEU1 BLEU2 BLEU3 BLEU4 ✓49.9349.2236.5328.6123.47 ✓49.7249.7536.5128.3023.25 ✓ ✓ 51.0850.1638.0829.5123.82 ✓ ✓ ✓ 52.8152.5440.9632.8726.74 with both glosses and translations. It consists of 7096 training, 519 validation, and 642 test samples from 9 signers, with a vocabulary of 1066 glosses and 2877 German words. (2) CSL-Daily [53] is a Chinese SL dataset containing 18401, 1077, and 1176 videos for training, validation, and testing from 10 signers. It provides 2000 glosses and 2343 words for translation. (3) How2Sign [8] is a large- scale American Sign Language (ASL) dataset comprising over 80 hours of multi-view video and multimodal data. We use only the frontal-view RGB videos, including 31128, 1741, and 2322 samples for training, validation, and testing, respectively. (4) OpenASL [37] is another large ASL dataset with over 280 hours of video from more than 200 signers. It includes 96476 training, 997 validation, and 999 test samples. Data Preprocessing. Following prior work [11,15,31], we adopt the same preprocessing pipeline to ensure fair comparisons. During training, we apply standard data augmentations, including resiz- ing frames to 256×256 pixels, random cropping to 224×224 pixels, random horizontal flipping with a probability of 0.5, and random temporal scaling within±20%. During inference, frames are resized to 256×256 and center-cropped to 224×224. Implementation Details. Our architecture consists of following key components: (1) Visual Encoder: In our baseline setting, we choose PoolFormer [49] with 1D CNN layers used in previous work [11,31]. (2) Llama: We adopt the Llama-3.2-1B model 1 in our baseline setting. We choose the base pretrained model rather than instruction-tuned variants (e.g., Llama-3.1-1B-Instruct), as the effects of prompt engineering are beyond the scope of this work. Instead, our focus is on improving the performance of base LLMs in the context of gloss-free SLT. For LoRA used in Llama, we apply LoRA [19] with a rank of 4 and a scaling factor (‘LoRA alpha’) of 32. LoRA adapters are inserted into the ‘q-proj’, ‘v-proj’, and ‘o-proj’ layers. In addition, we fully fine-tune both the input embedding layer and the output layer in Llama, which are included in the list of trainable modules. (3) Training Setting: We train the model using 1 https://huggingface.co/meta-llama/Llama-3.2-1B Table 3: Effect of pseudo-gloss. POS means Part-Of-Speech. ModelROUGEBLEU1BLEU4 w/o lemmatizer50.1951.6224.74 w/o stopword50.0050.0424.16 POS-only33.0031.4118.34 Ours 52.8152.5426.74 Figure 2: Effect of different휆in VPD and different drop rates in FPG-CTC the Adam optimizer with a weight decay of 0.0001 for 50 epochs on three GeForce RTX 3090 GPUs. The initial learning rate is set to 1e-4 for the recognition model and 1e-5 for the translation model, and is decayed by a factor of 0.5 at epochs 10, 20 and 30. The batch size is set to 3. For FPG-CTC pretraining, the random dropout rate is set to 0.2. The loss weights훼,훽in Equation 7 are set to 1, and휆 is set to 5. To ensure the model can be trained end-to-end within 24GB of GPU memory, the entire model is trained in half-precision. Evaluation Metrics. To evaluate our model, we adopt the ROUGE- L F1 Score [24], BLEU-1,2,3,4 [32] and BLEURT [36] for gloss-free SLT, which are common metrics in existing work [4, 13]. 6 Ablation Study Following previous work [2,4,11], we perform ablation studies on the Phoenix14T dataset to verify the effectiveness of our proposed SignLlama model. Effect of the Proposed Training Strategies. To validate the effec- tiveness of our proposed FPG-CTC pretraining and VPD training strategies, we compare our approach against baselines that are trained via direct fine-tuning. The results are shown in Table 1. The direct fine-tuning baseline model performs poorly, achieving only 5.46 BLEU-4 on the ‘Test’ sets. In contrast, applying FPG-CTC sig- nificantly boosts performance, demonstrating that our pseudo-gloss CTC pretraining effectively trains the visual encoder and facilitates better alignment between visual and textual features. Combining FPG-CTC with VPD yields the best results, which suggests that prioritizing visual features leads to better performance. Effect of VPD Loss. The proposed VPD loss comprises three com- ponents:L 푣푡 푇 ,L 푣 푇 , andL 퐾퐷 . As shown in Table 2, training with L 푣푡 푇 alone (i.e., standard autoregressive learning with both visual and textual inputs) achieves BLEU-4 scores of 23.47 on the test set. In comparison, using onlyL 푣 푇 , which relies solely on visual inputs, yields slightly lower BLEU-4 score of 23.25. When the distillation lossL 퐾퐷 is introduced, the model achieves substantial performance M ’26, November 10–14, 2026, Rio de Janeiro, BrazilShiwei Gan et al. Table 4: Effect of different Llama pretrained model. Model TEST ROUGEBLEU1BLEU4 Llama2 7B52.6152.1426.26 Llama2 13B52.7352.5426.54 Llama3.2 1B52.8152.5426.74 Llama3.2 3B 52.1552.6126.97 Llama3.1 8B52.9852.5526.85 Llama3 8B 52.4052.2426.82 gains, reaching 26.74 BLEU-4 on the test set. These results highlight the effectiveness of VPD in enhancing visual feature utilization via KD, thereby improving overall translation quality. Effect of Pseudo Gloss. To evaluate the effect of different prepro- cessing steps on FPG-CTC, we conduct ablation studies, as shown in Table 3. Removing either the lemmatization or stopword filtering results in a small drop in performance. Using only POS information significantly degrades performance, showing that part-of-speech information alone is insufficient. These results indicate that our approach benefits from the combined preprocessing steps of lemma- tization and stopword removal, while the overall trends remain consistent across different settings. Effect of the Hyperparameter휆in VPD. We introduce a hyper- parameter휆in the VPD loss to control the contribution of the distillation term. To analyze its impact, we conduct experiments with varying values of휆. As illustrated in Figure 2, the best BLEU-4 performance is observed when휆is set to 5. Based on this observa- tion, we set 휆= 5 in all experiments for better performance. Effect of Drop Rate in FPG-CTC. We apply random dropping dur- ing pseudo-gloss generation in FPG-CTC and further investigate the impact of varying dropout rates. As shown in Figure 2, model per- formance improves with moderate dropout rates but degrades when the dropout rate is too high, as excessive word removal hinders effective alignment learning between SL videos and pseudo-gloss sequences during FPG-CTC pretraining. Accordingly, we use a drop rate of 0.2 in all experiments. Effect of Different Llama Sizes. Llama is a family of LLMs ranging from 1B to 405B parameters. Given our limited GPU resources, we scale our model from the 1B model to larger variants up to 13B, including both the Llama 2 and Llama 3 series. As shown in Table 4, interestingly, increasing the size of the pretrained model yields only marginal improvements. This may be attributed to the fact that emergent behaviors often associated with LLMs tend to manifest at larger model scales. This conclusion is consistent with the findings in [50]. Since scaling the Llama model size does not yield significant improvements, we report results using the Llama 3.2 1B model in all subsequent comparisons, unless otherwise specified. Training and Inference Speed. To evaluate VPD’s impact on train- ing time, we conduct experiments on three GeForce RTX 3090 GPUs with a batch size of 3, and limit the maximum training sample to Figure 3: Visualization of feature distributions using t- SNE [42]. Blue and green points represent the visual features without/with FPG-CTC. Red and black points represent the corresponding gloss and text sequence embeddings of Llama. Table 5: Training speed. ModelTraining Time w VPD22min/epoch w/o VPD20 min/epoch Table 6: Model inference speed. Inference(s) LLama Model Size 1B3B8B13B 250 frames0.852.142.554.23 280 frames on Phoenix14T. As shown in Table 5, VPD introduces only a slight overhead, increasing the training time per epoch from 20 minutes to 22 minutes. This demonstrates that VPD brings neg- ligible additional cost while providing performance gains. Here, we also report the inference latency on a single NVIDIA RTX 3090 GPU with approximately 250 frames for one video. As shown in Ta- ble 6, even the 13B model maintains reasonable inference efficiency, while the 1B version achieves 0.85s/video with one 3090 GPU. 7 Qualitative Results FPG-CTC Visualization. We visualize the visual feature distribu- tion of one test sample from Phoenix14T, trained with and without FPG-CTC, alongside the corresponding gloss/text sequence embed- dings from Llama. As shown in Figure 3, the gloss/text inputs are tokenized into discrete tokens. In contrast, the visual feature dis- tribution trained without FPG-CTC exhibits a non-discriminative and continuous pattern. When trained with FPG-CTC, the visual features are better tokenized and exhibit a more discriminative distribution, which closely resembles that of the textual input. This demonstrates that FPG-CTC effectively tokenizes SL videos and provides suitable inputs for the LLMs. VPD Visualization. To investigate the effect of VPD, we visual- ize one of the attention heads from the final layer of the Llama model (other heads and layers exhibit similar patterns) under mod- els trained with and without VPD. As shown in Figure 4, ‘s’ denotes the start-of-sequence token, ‘v0–v11’ represent visual tokens, and SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMsMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Table 7: Comparison of SLT performance on Phoenix14T datasets. Gloss free SLT Phoenix14T DEVTEST ROUGEBLEU1BLEU2BLEU3BLEU4ROUGEBLEU1BLEU2BLEU3BLEU4 SignCL [47]49.0449.7636.8529.9722.74 Sign2GPT[44]-----48.9049.5435.9628.8322.52 GFSLT-VLP-SignCL [47] -----49.0449.7636.8529.9722.74 LLaVA-SLT [23]-----50.4451.2037.5129.3923.43 FLa-LLM [6] -----45.2746.2935.3328.0323.09 C2RL [5]-----50.9652.8140.2032.2026.75 SignLLM [17] 44.4946.8836.5929.9125.2547.2345.2134.7828.0523.40 MixSignGraph [15]51.7151.0737.9729.9824.8751.1450.0138.0429.9524.02 SignDINO [9] 52.3653.6438.6530.4925.6252.7552.1339.6433.7325.46 SignLlama 1B52.5352.1338.8830.6125.7352.8152.5440.9632.8726.74 Table 8: Comparison of SLT performance on CSL-Daily datasets. Gloss-free SLT CSL-Daily DEVTEST ROUGEBLEU1BLEU2BLEU3BLEU4ROUGEBLEU1BLEU2BLEU3BLEU4 GASLT [48]----20.3519.909.945.984.07 GFSLT [52]35.1637.6023.3014.899.9235.4237.6923.2814.939.88 GFSLT-VLP [52]36.4439.2025.0216.3511.0736.7039.3724.9316.2611.00 SignCL [47] 48.9247.4732.5322.6216.16 Sign2GPT[44]-----42.3641.7528.7320.6015.40 SignLLM [17]39.1842.4526.8817.9012.2339.9139.5528.1320.0715.75 GFSLT-VLP-SignCL [47]-----48.9247.4732.5322.6216.16 FLa-LLM [6]-----37.2537.1325.1218.3814.20 C2RL [5]-----48.2149.3236.2827.5421.61 LLaVA-SLT [23]-----51.2652.1536.2426.4720.42 MixSignGraph [15]49.1649.9836.4226.8920.4349.9350.2436.9127.5420.78 SignLlama51.6252.1241.2333.2324.8351.4352.3839.7833.9124.76 Figure 4: Visualization of the attention weights of Llama. ‘t0–t7’ indicate text tokens. The model trained without VPD primar- ily relies on previous text tokens for next-token prediction, with limited attention to visual inputs. In contrast, the model trained with VPD allocates significantly more attention to the visual tokens. This demonstrates that VPD effectively prioritizes visual features for LLMs, thereby improving translation performance. 8 Comparisons For a fair comparison, we only compare with image-based GFSLT methods that do not use external SL datasets for pretraining. Evaluation on Phoenix14T Dataset. As shown in Table 7, we com- pare the GFSLT performance of our model with existing approaches on the Phoenix14T dataset. Our model achieves excellent results, surpassing most of previous methods and improving the BLUE4 score by 1.28 points over SignDINO [9], reaching a 52.81 ROUGE-L score and 26.74 BLEU-4 score on the test set. Evaluation on CSL-Daily Dataset. As shown in Table 8, we also evaluate the GFSLT performance of our model on the CSL-Daily dataset. To better support Chinese translation, we replace the origi- nal Llama with the Chinese-Llama-2-1.3B model. As shown in the table, our model achieves strong performance with a BLEU-4 score of 24.76, achieving competitive performance. Evaluation on How2Sign Dataset. How2Sign is a larger American SL dataset, only provides text labels. As shown in Table 9, we com- pare the GFSLT performance of our proposed model with existing M ’26, November 10–14, 2026, Rio de Janeiro, BrazilShiwei Gan et al. Table 9: Comparison of GFSLT performance on How2Sign. Gloss-free SLT How2Sign DEVTEST ROUGEBLEU1BLEU2BLEU3BLEU4ROUGEBLEU1BLEU2BLEU3BLEU4 YouTube-SLT [41]------14.965.112.261.22 YouTube-SLT-P [41] ------37.8224.1316.9212.39 SSVP-SLT [35]-----25.7030.2016.7010.507.00 퐶 2 RL [5] -----27.8129.8118.9913.279.66 FLa-LLM [6]-----27.8129.8118.9913.279.66 GloFE-VN [25] 12.9815.217.384.072.3712.6114.947.273.932.24 SLT-IV [39]-35.2020.6213.258.89-34.0119.312.188.03 MixSignGraph [15]25.4126.8216.7011.488.3625.7126.6516.5511.688.69 SignLlama 1B30.3430.9617.3212.249.3130.2630.5119.9516.929.89 Table 10: Comparison of SLT performance on OpenASL. Gloss-free SLT OpenASL DEVTEST ROUGEBLEU1BLEU2BLEU3BLEU4ROUGEBLEU1BLEU2BLEU3BLEU4 OpenASL [37]25.3124.3514.9410.728.3924.8323.8714.089.907.54 GloFE-VN [25]21.3721.0612.348.686.6821.7521.5612.749.057.06 MixSignGraph [15] 25.4126.8216.7011.488.3625.7126.6516.5511.688.69 퐶 2 RL [5]-----31.3631.4621.8516.5813.21 SignLlama 1B35.2835.1126.4419.0216.5937.7437.3725.9717.7015.16 models. Our method, incorporating FPG-CTC and VPD, achieves a BLEU4 score of 14.47 on the test set, outperforming the previous SOTA model. Evaluation on OpenASL Dataset. Similar to How2Sign, the Ope- nASL dataset is a large-scale benchmark with a large vocabulary and text-only annotations. Our model achieves excellent performance across all evaluation metrics on both the development and test sets. Compared to the previous best-performing model, C2RL [5], our model achieves substantial improvements, particularly in BLEU scores. On the test set, it improves BLEU-4 from 13.21 to 15.16, and BLEU-1 from 31.46 to 37.37, indicating more accurate and fluent generation. 9 Conclusion In this paper, we focus on fine-tuning large language models (LLMs) for gloss-free sign language translation (GFSLT). We reveal two key challenges in this setting: (1) the inherent distributional gap between SL video features and textual features, which makes align- ment difficult; and (2) the common practice of simply concatenating visual and textual inputs in autoregressive training, which causes the model to overemphasize textual cues, deprioritize visual infor- mation, and further amplify exposure bias during inference. To address these challenges, we propose two key contributions: Fil- tered Pseudo-Gloss CTC Pretraining (FPG-CTC), which leverages automatically generated pseudo-gloss sequences to supervise the training of the visual backbone; and Visual-Prioritized Distillation (VPD), a training strategy designed to encourage prioritizing vi- sual features. Extensive experiments across multiple benchmark datasets demonstrate the effectiveness of our approach. Our model achieves competitive results on the gloss-free SLT task. 10 Limitations And Discussions Here, we list some potential ideas that can be further explored to improve performance. First, our FPG-CTC approach generates pseudo-gloss sequences using simple rule-based NLP techniques. More discussion of effective pseudo-gloss generation strategies can be further investigated in future work. Second, VPD requires dual forward passes for visual-text prediction and visual-only predic- tion, increasing training cost. More efficient approaches to enhance visual feature utilization are worth investigating. In addition, we highlight the potential negative social impacts. Our model, like other deep learning approaches, requires significant computational resources, which could raise issues related to energy use and en- vironmental impact. Moreover, since our method is data-driven, it may inherit potential biases from the training data. Careful dataset selection and balanced data are important to reduce such risks. Scaling to Larger Models. Due to hardware limitations, we scale the LLM from 1B up to 13B in our experiments. In future work, we plan to explore larger pretrained LLMs, as emergent capabilities observed at greater scales may lead to further gains in GFSLT. SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMsMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Acknowledgments This work is supported in part by National Natural Science Founda- tion of China under Grant Nos. 62172208, 92467202, 62272216; Key Projects of Jiangsu Provincial Basic Research Program under Grant No. BK20243040; JiangSu Natural Science Foundation under Grant No. BK20251989. This work is partially supported by Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (No. JYB2025XDXM118); the “111 Center” (No. B26023); Collaborative Innovation Center of Novel Software Technology and Industrialization. References [1]Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. 2024. MiniGPT4-Video: Advancing Mul- timodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens. arXiv preprint arXiv:2404.03413 (2024). [2]Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. 2018. Neural sign language translation. In CVPR. 7784–7793. [3]Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023. MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478 (2023). [4]Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak. 2022. Two-stream network for sign language recognition and translation. Advances in Neural Information Processing Systems 35 (2022), 17043–17056. [5]Zhigang Chen, Benjia Zhou, Yiqing Huang, Jun Wan, Yibo Hu, Hailin Shi, Yanyan Liang, Zhen Lei, and Du Zhang. 2025. C 2 RL: Content and Context Represen- tation Learning for Gloss-free Sign Language Translation and Retrieval. IEEE Transactions on Circuits and Systems for Video Technology (2025). [6]Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, and Guoqing Zhao. 2024. Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). 7071–7081. [7]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR (2021). [8]Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. 2021. How2Sign: a large-scale multimodal dataset for continuous American sign language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2735–2744. [9]Shiwei Gan, Xiao Liu, Yafeng Yin, Nan Liu, Kuizhuang Liu, Desibieer Tuerdaken, Zhiwei Jiang, Lei Xie, Sanglu Lu, and Hongkai Wen. 2026. Learning Effective Sign Features without Text for Gloss-free Sign Language Translation. (2026). [10] Shiwei Gan, Lichen Wang, Xiao Liu, Yafeng Yin, Kuizhuang Liu, Sanglu Lu, and Lei Xie. 2026. Sign Language Question Answering: A New Task, Benchmark, and Baseline for Sign Language Understanding. arXiv preprint arXiv:2607.27826 (2026). [11] Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Hongkai Wen, Lei Xie, and Sanglu Lu. 2024. SignGraph: A Sign Sequence is Worth Graphs of Nodes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13470–13479. [12]Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Kang Xia, Lei Xie, and Sanglu Lu. 2023. Contrastive learning for sign language recognition and translation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI- 23. 763–772. [13]Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Lei Xie, and Sanglu Lu. 2021. Skeleton- Aware Neural Sign Language Translation. In M. 4353–4361. [14]Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Lei Xie, and Sanglu Lu. 2023. Towards Real- Time Sign Language Recognition and Translation on Edge Devices. In Proceedings of the 31st ACM International Conference on Multimedia. 4502–4512. [15]Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Lei Xie, Sanglu Lu, and Hongkai Wen. 2025. MixSignGraph: A Sign Sequence is Worth Mixed Graphs of Nodes. [16]Peter Gaskell, Jana Pavlasek, Tom Gao, Abhishek Narula, Stanley Lewis, and Odest Chadwicke Jenkins. 2024. MBot: A modular ecosystem for scalable robotics education. In 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 18294–18300. [17]Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, and Jun Liu. 2024. Llms are good sign language translators. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18362–18372. [18]Michael Guevarra, Indronil Bhattacharjee, Srijita Das, Christabel Wayllace, Car- rie Demmans Epp, Matthew E Taylor, and Alan Tay. 2025. An LLM-Guided Tutoring System for Social Skills Training. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 29643–29645. [19]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.2022. Lora: Low-rank adaptation of large language models. ICLR 1, 2 (2022), 3. [20]Peiqi Jiao, Yuecong Min, and Xilin Chen. 2024. Visual alignment pre-training for sign language translation. In European Conference on Computer Vision. Springer, 349–367. [21] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. PMLR, 19730–19742. [22]KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 (2023). [23]Han Liang, Chengyu Huang, Yuecheng Xu, Cheng Tang, Weicai Ye, Juze Zhang, Xin Chen, Jingyi Yu, and Lan Xu. 2024. LLaVA-SLT: Visual Language Tuning for Sign Language Translation. arXiv preprint arXiv:2412.16524 (2024). [24] Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81. [25]Kezhou Lin, Xiaohan Wang, Linchao Zhu, Ke Sun, Yi Yang, et al.2023. Gloss- Free End-to-End Sign Language Translation. In The 61st Annual Meeting Of The Association For Computational Linguistics. [26]Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning. Advances in neural information processing systems 36 (2023), 34892–34916. [27] Xiao Liu, Shiwei Gan, Yafeng Yin, Bowen Guo, Zhiwei Jiang, Shunmei Meng, Lei Xie, and Sanglu Lu. 2026. SignPR: A Progressive Vector-Quantized Diffu- sion Framework for Sign Language Production. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2198–2208. [28] Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvinine- jad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics 8 (2020), 726–742. [29]Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2024. Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). [30]Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2024. VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding. arxiv (2024). https://arxiv.org/abs/2406.09418 [31] Yuecong Min, Aiming Hao, Xiujuan Chai, and Xilin Chen. 2021. Visual alignment constraint for continuous sign language recognition. In ICCV. 11542–11551. [32]Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In ACL. 311–318. [33]Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat-Seng Chua, Yueting Zhuang, and Siliang Tang. 2024. Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning. arXiv:2402.11435 [cs.CV] [34]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9. [35] Phillip Rust, Bowen Shi, Skyler Wang, Necati Cihan Camgöz, and Jean Maillard. 2024. Towards privacy-aware sign language translation at scale. arXiv preprint arXiv:2402.09611 (2024). [36] Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. BLEURT: Learning Robust Metrics for Text Generation. In Proceedings of ACL. [37] Bowen Shi, Diane Brentari, Gregory Shakhnarovich, and Karen Livescu. 2022. Open-Domain Sign Language Translation Learned from Online Video. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing. 6365–6379. [38]Ozge Mercanoglu Sincan, Necati Cihan Camgoz, and Richard Bowden. 2023. Is context all you need? scaling neural sign language translation to large domains of discourse. In Proceedings of the IEEE/CVF international conference on computer vision. 1955–1965. [39]Laia Tarrés, Gerard I Gállego, Amanda Duarte, Jordi Torres, and Xavier Giró-i Nieto. 2023. Sign language translation from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5624–5634. [40]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). [41]Dave Uthus, Garrett Tanzer, and Manfred Georg. 2023. Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus. Advances in Neural Information Processing Systems 36 (2023), 29029–29047. [42]Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008). [43]Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al.2024. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems M ’26, November 10–14, 2026, Rio de Janeiro, BrazilShiwei Gan et al. 37 (2024), 121475–121499. [44]Ryan Wong, Necati Cihan Camgoz, and Richard Bowden. 2024. SIGN2GPT: leveraging large language models for gloss-free sign language translation. In ICLR 2024: The Twelfth International Conference on Learning Representations. [45]Xin Xiao, Bohong Wu, Jiacong Wang, Chunyuan Li, Haoyuan Guo, et al.2024. See- ing the image: Prioritizing visual correlation by contrastive alignment. Advances in Neural Information Processing Systems 37 (2024), 30925–30950. [46]Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont- Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. 2023. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10714– 10726. [47] Jinhui Ye, Xing Wang, Wenxiang Jiao, Junwei Liang, and Hui Xiong. 2024. Improv- ing Gloss-free Sign Language Translation by Reducing Representation Density. In NeurIPS. [48]Aoxiong Yin, Tianyun Zhong, Li Tang, Weike Jin, Tao Jin, and Zhou Zhao. 2023. Gloss attention for gloss-free sign language translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2551–2562. [49]Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. 2022. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10819–10829. [50]Biao Zhang, Garrett Tanzer, and Orhan Firat. 2024. Scaling sign language transla- tion. Advances in neural information processing systems 37 (2024), 114018–114047. [51]Jihai Zhang, Wengang Zhou, and Houqiang Li. 2014. A threshold-based hmm-dtw approach for continuous sign language recognition. In ICIMCS. 237–240. [52] Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang. 2023. Gloss-free sign language translation: Improving from visual-language pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 20871–20881. [53]Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. 2021. Improv- ing Sign Language Translation with Monolingual Data by Sign Back-Translation. In CVPR. 1316–1325. [54]Qidan Zhu, Jing Li, Fei Yuan, and Quan Gan. 2024. Multiscale temporal network for continuous sign language recognition. Journal of Electronic Imaging 33, 2 (2024), 023059–023059. [55]Ronglai Zuo and Brian Mak. 2022. C2SLR: Consistency-Enhanced Continuous Sign Language Recognition. In CVPR. 5131–5140. SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMsMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil A Datasets Details of the Datasets. We provide detailed information on the five datasets used in our paper in Table 11. The datasets we down- loaded may slightly differ from the official versions described in their respective papers. We also provide the token vocabulary ob- tained by the Llama tokenizer and the vocabulary in FPG-CTC of processed text for reference. B Model Details Our FPG-CTC module generates a pseudo-gloss sequence from the target text by applying lemmatization and removing function words such as prepositions and conjunctions. Since certain words in the text may not be explicitly expressed in the corresponding SL video, we further introduce random dropping to remove a fixed proportion of tokens, producing the final pseudo-gloss labels. The NLTK toolkit is used for pseudo-gloss preprocessing steps. C Ablation Study on Hyperparameters Effect of Different휆. In the main paper, we illustrated the impact of different휆values on our model’s performance using a line chart (Figure 2). Here, we provide the detailed results in Table 12. As shown, when휆=0 (i.e., without the VPD loss), the model achieves the lowest performance. Increasing휆leads to notable improvements across ROUGE and BLEU metrics, peaking around휆=5 to휆=10. Therefore, we set휆to 5 in our final model to balance performance and stability. Effect of Different Drop Rates. In the main paper, we illustrated the impact of different drop rates in FPG-CTC using a line chart (Figure 2). Here, we provide detailed quantitative results in Table 13. The model performance improves as the drop rate increases from 0 to 0.2, reaching the best results at a drop rate of 0.2 across all ROUGE and BLEU metrics. However, further increasing the drop rate beyond 0.2 leads to significant performance degradation, as excessive word dropping hinders effective alignment learning. D Ablation Study on Proposed Module Effect of FPG-CTC.. We evaluate the effectiveness of our pro- posed FPG-CTC module by comparing the SLT performance on four benchmark datasets with and without FPG-CTC pretraining. As shown in Table 14, incorporating FPG-CTC consistently yields significant improvements across all datasets and evaluation metrics. For instance, on the Phoenix14T dataset, ROUGE and BLEU-4 scores improve from 20.09/5.46 to 52.81/26.74 on the test set, indicating that the pseudo-gloss supervision helps align sign video features with text features more effectively. Additionally, we report the CSLR performance based on the pseudo-gloss sequences generated by FPG-CTC (right side of Ta- ble 14). Although the pseudo-gloss labels are not ground truth and the resulting WERs are suboptimal, they still demonstrate a reason- able level of recognition performance, suggesting that the pseudo sequences preserve coherent structure and temporal alignment. Overall, these results demonstrate that FPG-CTC offers strong su- pervisory signals for improving gloss-free SLT, and lays a solid foundation for high-quality translation without relying on ground- truth gloss annotations. E Visualization SLT Qualitative Results. Figure 5 presents a qualitative analysis of our SignLlama model on the gloss-free SLT task, with examples selected from the test sets of Phoenix14T, CSL-Daily, How2Sign, and OpenASL. The results show that SignLlama, when equipped with both FPG-CTC and VPD, produces the most accurate transla- tions. In contrast, the end-to-end fine-tuning model performs worse, highlighting the effectiveness of our proposed FPG-CTC and VPD. Visualization of Attention Maps of Llama. Here, we provide more attention maps from the last layer of Llama models trained with and without VPD. As shown in Figure 6 and 7, VPD significantly enhances the model’s ability to prioritize relevant visual features, thereby improving its overall accuracy. M ’26, November 10–14, 2026, Rio de Janeiro, BrazilShiwei Gan et al. Table 11: Details of datasets used in our paper. Voc: vocabulary size. PG: pseudo gloss. DatasetToken Voc in TextPG Voc in FPG-CTC Video Samples TrainTestValidation Phoenix14T212130,017,096642519 CSL-Daily7277237718,4011,1761,077 How2Sign 95341534730,90423281713 OpenASL19,25025,40296,476975966 Table 12: Effect of different 휆 in Equation 7. 휆 DEVTEST ROUGEBLEU1BLEU2BLEU3BLEU4ROUGEBLEU1BLEU2BLEU3BLEU4 050.9650.6835.9828.7523.8351.0850.1638.0829.5123.52 1 51.0351.7338.0729.2625.1551.4150.5039.1631.2725.04 552.5352.1338.8830.6125.7352.8152.5440.9632.8726.74 1052.1051.2138.7929.9725.7052.6151.9139.7231.6126.20 1551.1351.1638.6529.7725.5451.3149.8737.3129.3226.07 2051.7151.6638.3629.5225.2751.4849.7537.2929.3326.05 2551.5751.8738.5129.6225.3551.2050.1137.5829.5826.28 Table 13: Effect of different drop rate in FPG-CTC. DropRate DEVTESTDevTest ROUGEBLEU1BLEU2BLEU3BLEU4ROUGEBLEU1BLEU2BLEU3BLEU4WERDel/InsWERDel/Ins 048.7747.5334.5726.5623.4049.0548.2835.5927.6824.5669.5553.67/ 5.2369.3553.76/ 4.65 0.150.3548.6336.0328.0624.7250.3149.3936.8828.9425.7472.9563.67/ 0.4472.7763.03/ 0.45 0.252.5352.1338.8830.6125.7352.8152.5440.9632.8726.7478.7774.77/ 0.1578.9573.68/0.18 0.348.6547.6236.1428.0222.7648.8648.3335.3927.1223.6888.2886.95/ 0.0388.3686.99/ 0.01 0.445.8044.9733.6525.7020.6445.8746.1333.1425.2522.1694.6094.25/ 0.0094.1293.73/0.01 0.5 42.2441.7629.9922.2317.5341.6442.7529.5121.7318.9294.7794.43/ 0.0094.6394.17/0.01 Table 14: Effect of proposed FPG-CTC for SLT. Besides, we also show the CSLR performance based on pseudo gloss sequence obtained by FPG-CTC in the right part. DatasetModel DevTestDevTest ROUGE BLEU1 BLEU2 BLEU3 BLEU4ROUGE BLEU1 BLEU2 BLEU3 BLEU4WERDel/InsWERDel/Ins Phoenix14T w/o FPG-CTC20.1117.228.216.185.2320.0918.1258.566.465.46--/---/- w/ FPG-CTC52.5352.1338.8830.6125.7352.8152.5440.9632.8726.7478.62 72.57/00.2278.32 72.23/00.24 CSL-Daily w/o FPG-CTC36.5134.5423.7214.6559.1135.3133.6519.3412.678.27--/---/- w/ FPG-CTC51.6252.1241.2333.2324.8351.4352.3839.7833.9124.7669.75 56.79/1.6069.77 55.89/1.79 How2Sign w/o FPG-CTC19.4524.1513.567.905.1519.1524.4712.987.815.14--/---/- w/ FPG-CTC37.3437.9622.3217.2414.3139.2640.5126.9518.9214.4770.62 69.57/00.2270.32 68.23/00.24 OpenASL w/o FPG-CTC13.1312.694.572.391.9612.6811.274.872.451.98--/---/- w/ FPG-CTC35.2835.1126.4419.0216.5937.7437.3725.9717.7015.1682.57 69.21/0.6582.85 69.19/0.69 SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMsMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Table 16: SLT Qualitative results on Phoenix-2014T, CSL-Daily, How2Sign and OpenASL. example(a)Phoenix14T dataset Groundtruthund nun die wettervorhersage für morgen sonntag den zwölften juli (And now the weather forecast for tomorrow, Sunday, July 11th) End-To-Endund nun die wettervorhersage für morgen mittwoch den fünfundzwanzigsten august (and now the weather forecast for tomorrow, Wednesday the twenty-fifth of August) + FPG-CTCund nun die wettervorhersage für morgen sonntag den elften juli (And now the weather forecast for tomorrow, Sunday, July 12th) + VPDund nun die wettervorhersage für morgen sonntag den zwölften juli (And now the weather forecast for tomorrow, Sunday, July 11th) example(b)CSL-daily dataset Groundtruth是，他这几天很忙没有时间休息。 (Yes, he has been very busy these days and has no time to rest.) End-To-End他最近很忙 He’s been very busy lately. + FPG-CTC他这几天很忙没有时间休息。 (He has been very busy these days and has no time to rest.) + VPD 是，他这几天很忙没有时间休息。 (Yes, he has been very busy these days and has no time to rest.) example(c)How2Sign dataset GroundtruthSo, we’ve got to find a way to get to the exit. End-To-End There is a way to the exit. + FPG-CTCwe have found a way to get to the exit. + VPDwe’ve got to find a way to get to the exit. example(d)OpenASL dataset GroundtruthWe are thankful to be here, and also to the Saudi club for their help. End-To-EndWe thanks for their help. + FPG-CTC We are thankful to be here, and thanks for their help. + VPDWe are thankful to be here, and also thanks for the club’s help. (a) head=0(b) head=2(c) head=4(d) head=6 (e) head=7(f) head=8(g) head=15(h) head=18 (i) head=20(j) head=24(k) head=28(l) head=30 Figure 6: Visualization of attention maps in the last layer of LlamawithVPD, showing that the attention is mainly focused on visual tokens. 17 Figure 5: SLT Qualitative results on Phoenix-2014T, CSL-Daily, How2Sign and OpenASL. M ’26, November 10–14, 2026, Rio de Janeiro, BrazilShiwei Gan et al. (a) head=0(b) head=2(c) head=4(d) head=6 (e) head=7(f) head=8(g) head=15(h) head=18 (i) head=20(j) head=24(k) head=28(l) head=30 Figure 6: Visualization of attention maps in the last layer of Llama with VPD, showing that the attention is mainly focused on visual tokens. SignLlama: Enhancing Gloss-free Sign Language Translation by Prioritizing Visual Features for LLMsMM ’26, November 10–14, 2026, Rio de Janeiro, Brazil (a) head=0(b) head=2(c) head=4(d) head=6 (e) head=7(f) head=8(g) head=15(h) head=18 (i) head=20(j) head=24(k) head=28(l) head=31 Figure 7: Visualization of attention weight in the last layer of Llama without VPD, showing that the attention is mainly focused on text tokens.