Paper deep dive
ViCLSR: A Supervised Contrastive Learning Framework with Natural Language Inference for Natural Language Understanding Tasks
Tin Van Huynh, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/26/2026, 2:23:24 AM
Summary
ViCLSR is a supervised contrastive learning framework designed to improve sentence representations for the Vietnamese language. By leveraging existing Natural Language Inference (NLI) datasets to create positive and negative sentence pairs, the framework optimizes sentence embeddings and outperforms existing monolingual (PhoBERT) and multilingual (XLM-R) models across five Vietnamese NLU benchmark datasets.
Entities (5)
Relation Signals (3)
ViCLSR → outperforms → PhoBERT
confidence 95% · Our experiments demonstrate that ViCLSR significantly outperforms the powerful monolingual pre-trained model PhoBERT
ViCLSR → improves → Vietnamese
confidence 90% · ViCLSR shows that supervised contrastive learning can effectively address resource limitations in Vietnamese NLU tasks
ViCLSR → usesarchitecture → XLM-R
confidence 90% · By integrating contrastive learning with the XLM-R architecture, ViCLSR leverages existing NLI datasets
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:High-quality text representations are crucial for natural language understanding (NLU), but low-resource languages like Vietnamese face challenges due to limited annotated data. While pre-trained models like PhoBERT and CafeBERT perform well, their effectiveness is constrained by data scarcity. Contrastive learning (CL) has recently emerged as a promising approach for improving sentence representations, enabling models to effectively distinguish between semantically similar and dissimilar sentences. We propose ViCLSR (Vietnamese Contrastive Learning for Sentence Representations), a novel supervised contrastive learning framework specifically designed to optimize sentence embeddings for Vietnamese, leveraging existing natural language inference (NLI) datasets. Additionally, we propose a process to adapt existing Vietnamese datasets for supervised learning, ensuring compatibility with CL methods. Our experiments demonstrate that ViCLSR significantly outperforms the powerful monolingual pre-trained model PhoBERT on five benchmark NLU datasets such as ViNLI (+6.97% F1), ViWikiFC (+4.97% F1), ViFactCheck (+9.02% F1), UIT-ViCTSD (+5.36% F1), and ViMMRC2.0 (+4.33% Accuracy). ViCLSR shows that supervised contrastive learning can effectively address resource limitations in Vietnamese NLU tasks and improve sentence representation learning for low-resource languages. Furthermore, we conduct an in-depth analysis of the experimental results to uncover the factors contributing to the superior performance of contrastive learning models. ViCLSR is released for research purposes in advancing natural language processing tasks.
Tags
Links
- Source: https://arxiv.org/abs/2603.21084v1
- Canonical: https://arxiv.org/abs/2603.21084v1
Trouble viewing inline? Open PDF directly →
Full Text
120,436 characters extracted from source content.
Expand or collapse full text
Springer Nature 2021 L A T E X template ViCLSR: A Supervised Contrastive Learning Framework with Natural Language Inference for Natural Language Understanding Tasks Tin Van Huynh 1,2 , Kiet Van Nguyen 1,2* and Ngan Luu-Thuy Nguyen 1,2 1 University of Information Technology, Ho Chi Minh City, Vietnam. 2 Vietnam National University, Ho Chi Minh City, Vietnam. *Corresponding author(s). E-mail(s): kietnv@uit.edu.vn; Contributing authors: tinhv@uit.edu.vn; ngannlt@uit.edu.vn; Abstract High-quality text representations are crucial for natural language un- derstanding (NLU), but low-resource languages like Vietnamese face challenges due to limited annotated data. While pre-trained models like PhoBERT and CafeBERT perform well, their effectiveness is con- strained by data scarcity. Contrastive learning (CL) has recently emerged as a promising approach for improving sentence representations, en- abling models to effectively distinguish between semantically similar and dissimilar sentences. We propose ViCLSR (Vietnamese Contrastive Learning for Sentence Representations), a novel supervised contrastive learning framework specifically designed to optimize sentence embed- dings for Vietnamese, leveraging existing natural language inference (NLI) datasets. Additionally, we propose a process to adapt existing Vietnamese datasets for supervised learning, ensuring compatibility with CL methods. Our experiments demonstrate that ViCLSR significantly outperforms the powerful monolingual pre-trained model PhoBERT on five benchmark NLU datasets such as ViNLI (+6.97% F1), ViWikiFC (+4.97% F1), ViFactCheck (+9.02% F1), UIT-ViCTSD (+5.36% F1), and ViMMRC2.0 (+4.33% Accuracy). ViCLSR shows that supervised contrastive learning can effectively address resource limitations in Viet- namese NLU tasks and improve sentence representation learning for low-resource languages. Furthermore, we conduct an in-depth analysis 1 arXiv:2603.21084v1 [cs.CL] 22 Mar 2026 Springer Nature 2021 L A T E X template 2 Tin et al. of the experimental results to uncover the factors contributing to the su- perior performance of contrastive learning models. ViCLSR is released for research purposes ∗ in advancing natural language processing tasks. Keywords: Contrastive Learning, Text Representation, Natural Language Understanding, Pre-trained Models 1 Introduction Natural Language Understanding (NLU) has become a critical component in a wide range of natural language processing (NLP) applications, as it enables machines to interpret and reason about human language in a way that aligns with human cognitive processes. Recently, contrastive learning has proven ef- fective for improving the quality of sentence embeddings, which are crucial for NLU tasks [1]. By contrasting positive and negative sentence pairs, contrastive learning optimizes the ability of model to understand subtle semantic relation- ships between sentences, which is crucial for tasks such as natural language inference and fact checking [2, 3]. This technique has been proven to signifi- cantly improve the performance of models in tasks that require deep semantic reasoning and contextual understanding [4, 5]. Moreover, the adaptability of contrastive learning allows it to be effectively utilized in various NLU tasks, further enhancing its versatility in addressing diverse challenges. Text representation is fundamental to NLU tasks, transforming raw text into dense vector embeddings that capture semantic meaning. These embed- dings serve as the foundation for a wide array of NLU tasks, including natural language inference [6], semantic similarity detection [7], fact checking [8], ma- chine reading comprehension (MRC) [9], and text classification [10]. While substantial progress has been made in high-resource languages like English, NLU for low-resource languages such as Vietnamese remains a significant chal- lenge. This is due to the unique linguistic properties of Vietnamese, such as its complex syntax, morphology, and semantics, which are often inadequately captured by existing models. In recent years, transformer-based pre-trained language models have rev- olutionized NLU, with multilingual models like mBERT [11] and XLM-R [12] showcasing exceptional performance across diverse languages. However, these models are trained on textual data spanning hundreds of languages, leading to diluted focus on Vietnamese-specific characteristics. Studies [2, 3, 13] have shown that such multilingual models often struggle to capture subtle linguistic nuances, particularly in complex tasks like NLI and fact checking. Besides, monolingual models like PhoBERT [14], tailored specifically for Vietnamese, leverage the RoBERTa [15] architecture and a large corpus of Vietnamese text. These models have demonstrated notable improvements over multilingual counterparts in specific tasks, as shown in research by Nguyen et ∗ We will provide an access link to it as soon as the article is accepted. Springer Nature 2021 L A T E X template Tin et al.3 al. (2020) [14], and Do et al. (2024) [16]. However, their ability to represent complex semantic relationships remains limited. This limitation is particu- larly evident in tasks requiring profound contextual understanding, such as natural language inference [2, 17], machine reading comprehension [18], and fact checking [13], where PhoBERT still falls short compared to cutting-edge pre-trained models. Addressing these challenges requires more advanced ap- proaches that can fully leverage the unique linguistic characteristics of the Vietnamese language, especially in resource-constrained settings. In this study, we propose a novel supervised contrastive learning frame- work to enhance NLU for Vietnamese. By integrating contrastive learning with the XLM-R architecture, we aim to improve the ability of the model to capture complex semantic relationships. We utilize existing Vietnamese NLI datasets and adapt them into contrastive-like datasets including positive sen- tence pairs (x, x + ) and negative sentence pairs (x, x − ) to be suitable for training contrastive learning models, allowing the model to optimize embed- dings for improved semantic reasoning. Our proposed model demonstrates significant improvements over both multilingual and monolingual pre-trained models, achieving state-of-the-art results across a variety of Vietnamese NLU tasks, particularly those requiring profound contextual understanding, such as NLI, fact checking, machine reading comprehension, and constructive speech detection. The four primary contributions of our research in this paper are summa- rized as follows. • ViCLSR Framework: We propose ViCLSR, a novel supervised contrastive learning framework designed to improve Vietnamese sentence embeddings. By integrating contrastive learning with the XLM-R architecture, ViCLSR leverages existing NLI datasets (ViNLI and XNLI) and introduces a data adaptation process to create positive and negative sentence pairs, enhancing semantic representation for NLU tasks. • Empirical Evaluation: We conduct a comprehensive evaluation of the effectiveness of ViCLSR through extensive experiments on five benchmark datasets: ViNLI, ViWikiFC, ViFactCheck, UIT-ViCTSD, and ViMMRC2.0—covering tasks like natural language inference, fact check- ing, constructive speech detection, and machine reading comprehension. The experimental results demonstrate its superiority over existing multilin- gual (mBERT, XLM-R) and monolingual (PhoBERT) pre-trained models, highlighting its effectiveness in Vietnamese NLU. • In-Depth Analysis: We provide a detailed analysis of the factors in- cluding hyperparameter studies (e.g., temperature τ), ablation tests on auxiliary MLM objectives, pooling strategies, semantic representation capa- bilities, sentence embedding distributions, and attention mechanisms that contribute to the success of ViCLSR, enhancing the understanding of how contrastive learning can improve semantic representations for low-resource languages like Vietnamese. Springer Nature 2021 L A T E X template 4 Tin et al. • Public Release: We publicly release our pre-trained model (ViCLSR) to support reproducibility and future research. This open-access resource is especially valuable for low-resource language communities, enabling re- searchers to build upon or compare with ViCLSR in advancing NLU applications. The structure of our paper is organized as follows. Section 1 introduces text representation and highlights the challenges they pose for low-resource languages like Vietnamese. Section 2 discusses related works, including text representation, contrastive learning, training data resource for contrastive learning. Section 3 presents our ViCLSR approach, covering data prepara- tion, training with contrastive learning, and fine-tuning for downstream tasks. Section 4 presents the experiments and results. Section 5 provides an analysis and discussion of the findings. Finally, Section 6 concludes the paper by sum- marizing the key research contributions and proposing directions for future work. 20222020201420152013 Word2Vec (Mikolov et al., 2013) GloVe (Pennington et al., 2014) Skip-Thought (Kiros et al., 2015) InferSent (Conneau et al., 2017) 2017 SentEval (Conneau and Kiela, 2018) ELMO (Peters et al., 2018) 20182019 BERT (Devlin et al., 2019) SBERT (Reimers and Gurevych, 2019) RoBERTa (Liu et al., 2019) DistilBERT (Sanh et al., 2019) mBERT (Devlin et al., 2019) XLM-R (Conneau et al., 2020) T5 (Raffel et al., 2020) MPNet (Song et al., 2020) mT5 (Xue et al., 2020) SimCSE (Gao et al., 2021) 2021 LaBSE (Feng et al., 2022) Ada-002 (OpenAI) 2023 LLaMA (Meta AI) GPT-4 (OpenAI) Gemini (Google DeepMind) GPT-3 (OpenAI) Claude (Anthropic et al., 2023) Monolingual Multilingual FastSent (Hill et al., 2016) 2016 DiffCSE (Chuang et al., 2022) PaLM (Google Research) Word Embeddings Sentence Embedding Contrastive Learning Contextualized Word Embeddings LLMs DeepSeek-V3 (DeepSeek-AI) 2024 Figure 1: Overview of Typical Text Representation Models from 2013 to 2024. 2 Related Works We provide a review of related work, outlining the evolution of text represen- tation from basic methods to advanced transformer-based models, and discuss how contrastive learning improves sentence representations, especially for NLU tasks requiring semantic understanding. 2.1 Text Representation Text representation plays a crucial role in natural language processing, as it transforms textual data into numerical representations that capture semantic Springer Nature 2021 L A T E X template Tin et al.5 meaning. As illustrated in Figure 1, the evolution of text representation meth- ods has progressed from early statistical approaches to modern transformer- based architectures. Initial techniques such as bag-of-words (BoW) and TF-IDF treated words as independent units, lacking contextual awareness and semantic relationships. The introduction of word embeddings, such as Word2Vec [19] and GloVe [20], marked a significant shift by encoding words into continuous vector spaces based on co-occurrence patterns. These meth- ods improved word-level representations but remained context-independent, limiting their ability to capture sentence-level semantics effectively. To overcome these limitations, researchers developed Sentence Embed- dings, where models like Skip-Thought [21], InferSent [22], and FastSent [23] focused on generating sentence-level representations. These models improved upon word embeddings by considering larger linguistic contexts but were of- ten constrained by their reliance on fixed-length vector representations and lack of task-specific fine-tuning. A paradigm shift occurred with the advent of Contextualized Word Em- beddings, pioneered by models such as ELMo [24], BERT [11], RoBERTa [15], SBERT [25], DistilBERT [26], and T5 [27]. These transformer-based architec- tures introduced self-attention mechanisms, enabling dynamic representations that capture word dependencies across different sentence positions. Unlike traditional embeddings, these models consider both left and right contexts, sig- nificantly improving performance in NLU tasks such as machine translation, text classification, and natural language inference. Recent advancements have explored contrastive learning as a method for improving sentence representations. Models like SimCSE [1] and Dif- fCSE [5]) have leveraged contrastive objectives to align semantically similar sentences while pushing apart dissimilar ones. These approaches have demon- strated superior performance in semantic similarity, sentence clustering, and retrieval-based NLU tasks compared to traditional methods. The rapid evolution of Large Language Models (LLMs) has further trans- formed text representation. Beginning with GPT-3 [28], followed by GPT-4 [29], LLaMA [30] , and Gemini [31] in 2023 or DeepSeek-V3 [32] in 2024, these models have moved beyond static text representations to context- aware, generative models capable of handling multi-turn dialogues and diverse NLP applications. The integration of pre-trained transformer architectures has made it possible to fine-tune these models for specific domains, opti- mizing their performance across various natural language understanding and generation tasks. In the context of multilingual NLU, XLM-R [12] and mBERT [11] have significantly advanced cross-lingual transfer learning by enabling shared rep- resentations across multiple languages. Among these, XLM-R stands out due to its robust training on a massive multilingual corpus, which allows it to capture both syntactic and semantic nuances across diverse linguistic struc- tures. This makes XLM-R particularly effective for low-resource languages like Vietnamese, as it benefits from rich cross-lingual knowledge transfer while Springer Nature 2021 L A T E X template 6 Tin et al. maintaining strong performance on monolingual tasks. In Vietnamese NLU, specialized models such as PhoBERT [14] and CafeBERT [16] have further re- fined language representations. Recent efforts have specifically tailored these architectures to improve Vietnamese natural language inference (NLI). For in- stance, integrating contextualized language models with deep neural network architectures has been shown to optimize classification performance on NLI benchmarks [33]. Furthermore, enhancing these pre-trained models with exter- nal contextual knowledge retrieval has proven effective in addressing complex passage-level reasoning capabilities [34]. However, despite these task-specific adaptations, existing approaches still face challenges in fine-grained seman- tic understanding, particularly in natural language inference and sentence similarity evaluation, where effective sentence embeddings are crucial. To address this, supervised contrastive learning has emerged as a promis- ing approach for enhancing sentence representations by leveraging labeled data to better structure the semantic space. In this research, we propose a supervised contrastive learning framework built upon XLM-R [12], aiming to optimize sentence embeddings for Vietnamese and extend them to other low-resource languages. By leveraging the strong multilingual encoding capa- bilities of XLM-R and refining its representations with contrastive learning objectives, this approach aims to enhance sentence-level semantic alignment and task-specific generalization. This framework has the potential to bridge performance gaps between high-resource and low-resource languages, making sentence embeddings more robust, adaptable, and semantically enriched for diverse NLP applications. 2.2 Contrastive Learning Contrastive learning has emerged as a transformative approach in NLU by focusing on learning meaningful sentence representations through the compar- ison of similar and dissimilar data points. This paradigm shift started with the introduction of models like SimCSE [1], which utilized contrastive learning to enhance sentence embeddings by leveraging both unsupervised and supervised training approaches. In SimCSE, the unsupervised approach generates posi- tive pairs by applying dropout noise, while the supervised method uses natural language inference datasets as labeled pairs, showing that contrastive learning can bring significant improvements over traditional models like BERT. The overall goal of contrastive learning is to minimize the distance between se- mantically similar sentences while maximizing the distance between dissimilar ones in the vector space, which makes it highly effective for tasks like semantic similarity, paraphrase detection, and sentence clustering. Based on the effectiveness of contrastive learning in capturing semantic re- lationships within text, this approach has been widely applied across various NLU tasks, resulting in remarkable improvements. It has led to significant ad- vancements in areas such as text classification [35, 36], sentence embeddings and phrase embeddings [1, 37, 38], information extraction [39, 40], machine Springer Nature 2021 L A T E X template Tin et al.7 translation [41, 42], machine reading comprehension [43, 44], and summariza- tion [45, 46]. Additionally, contrastive learning has been heavily explored in computer vision [47], where it is used to detect similarities in visual data, highlighting its adaptability across both text and image processing domains. Contrastive Learning has become a vital method for improving represen- tation learning in low-resource languages, which often lack large annotated datasets. In the broader context of low-resource languages, various models have been developed using CL to enhance semantic representations for languages like Khmer and Pashto [48], Bengali [49], and Mongolian [50]. Additionally, there has been a notable amount of research on contrastive learning in mul- tilingual settings, which has made valuable strides in supporting semantic representations for low-resource languages [51, 52]. For Vietnamese, the num- ber of studies related to contrastive learning remains limited. Among the existing research, some focus on specific tasks, such as Vietnamese abstractive summarization [53], but these studies lack generalizability and are often not easily applicable to a wide range of NLU tasks. Another example includes re- search on cross-lingual sentence embeddings between Chinese and Vietnamese [52], which, while showing promising results, lacks focus on fully leveraging the unique linguistic features of Vietnamese. This highlights the need for more dedicated research on contrastive learning specifically tailored to Vietnamese, aiming to enhance its applicability across multiple NLU tasks. Such efforts would help better exploit the characteristics of the Vietnamese language and improve its performance in diverse NLU applications. 2.3 Training Data Resource for Contrastive Learning Contrastive-like datasets are essential for training contrastive learning models, containing both positive (x, x+) and negative (x, x-) sentence pairs. Positive pairs represent semantically similar sentences, while negative pairs highlight dissimilarity, helping models learn to distinguish subtle semantic differences. These datasets are fundamental for tasks like natural language inference, where the goal is to determine the relationship between sentence pairs, such as entail- ment and contradiction. In English, datasets such as SNLI [6], MultiNLI [54], and ANLI [55] have been widely used for training and evaluating contrastive learning models [1, 35]. Additionally, Chinese features the OCNLI dataset [56], which has been successfully applied to train the CLOWER model [57]. For multilingual language NLI datasets, XNLI [58] and VietX-NLI [59] provide a crucial resource for building multilingual contrastive learning models [60, 61], further extending the applicability of NLI. For Vietnamese, recent years have witnessed the emergence of notable NLI datasets such as ViNLI [2], ViANLI [62], VnNewsNLI [63], and the dataset from the VLSP 2021 - vnNLI Challenge [64]. These datasets are considered valuable contrastive-like resources, offering significant potential for develop- ing contrastive learning models and enhancing semantic representation in Vietnamese. Springer Nature 2021 L A T E X template 8 Tin et al. 3 Our Proposed Framework 3.1 Overview Positive pairs (x, x⁺) Existed Vietnamese sources Transform Contrastive Data Natural Language Inference Fact Checking Constructive Speech Detection ViCLSR Model Evaluation on Downstream Tasks Machine Reading Comprehension Data Preparation Training with Contrastive Learning Fine-tuning for Downstream Tasks ViNLI XNLI (Vi) Entailment pairs (x, x⁺) Contradiction pairs (x, x⁻) Pre-trained Language Model Negative pairs (x, x⁻) far far close close Figure 2: Overview of our supervised contrastive learning framework (Vi- CLSR) for Vietnamese NLU tasks, including data preparation, contrastive training, and fine-tuning for downstream tasks. An overview of our supervised contrastive learning framework, designed to improve sentence embeddings for Vietnamese text understanding, is presented in Figure 2. This approach tackles the challenge of limited annotated data in low-resource languages like Vietnamese by leveraging contrastive learning techniques and adapting existing resources. By doing so, it enables the develop- ment of robust models that effectively capture semantic relationships between sentences and enhance performance across various Vietnamese NLU tasks. The proposed framework comprises three main components: Data Prepa- ration (Section 3.2), Training the Model with Contrastive Learning (Section 3.3), and Fine-tuning for Downstream Tasks (Section 3.4). • Data Preparation: In this phase, available Vietnamese datasets are adapted for supervised contrastive learning. This involves utilizing labeled datasets, such as those from Vietnamese Natural Language Inference (NLI) tasks, to construct supervised contrastive data. Entailment pairs (x,x + ) are treated as positive examples, while contradiction pairs (x,x − ) serve as neg- ative examples. This process ensures the availability of a substantial and high-quality dataset for contrastive learning. • Training with Contrastive Learning: In this phase, a state-of-the-art pre-trained language model is selected and trained on the constructed con- trastive dataset. The training process employs a supervised contrastive loss function, which minimizes the embedding distance between positive pairs Springer Nature 2021 L A T E X template Tin et al.9 and maximizes the distance between negative pairs. This ensures that the resulting embeddings capture the semantic nuances of Vietnamese text effectively. • Fine-tuning for Downstream Tasks: The final stage involves adapting the trained contrastive model to specific downstream NLU tasks, such as natural language inference, fact checking, text classification, machine read- ing comprehension, and other Vietnamese language processing tasks. The fine-tuning process further optimizes the model for task-specific objectives, ensuring high performance and applicability. This framework demonstrates the potential of supervised contrastive learning to overcome resource limitations in Vietnamese NLU, while also con- tributing to advancements in sentence representation learning for low-resource languages. 3.2 Data Preparation Algorithm 1 Data Preparation for Contrastive Learning 1: Input: 2: NLI Dataset: Premises P, Hypotheses H, Labels L 3: Output: 4: Dataset: Contrastive learning dataset based on NLI 5: Initialize empty datasets D NLI 6: Process NLI Dataset: 7: for each (p,h,l)∈ (P,H,L) do 8: if l = entailment then 9:Assign p→ sentence1, h→ sentence2 10: else if l = contradiction then 11:Assign h→ hard_neg 12: end if 13: end for 14: Construct D NLI =sentence1 (x), sentence2 (x + ), hard_neg (x − ) 15: Return D NLI Table 1: Statistics of Data Generated for Contrastive Learning from NLI Datasets. Dataset premise (x) Hypothesis (x, x+, x-) Entailment (x+) Contradiction (x-) ViNLI6,0946,0946,0946,094 XNLI2,5002,5002,5002,500 Total8,5948,5948,5948,594 Springer Nature 2021 L A T E X template 10 Tin et al. Inspired by SimCSE [1], we construct the contrastive training dataset di- rectly from labeled Vietnamese NLI corpora (ViNLI [2] and XNLI (Vi) [58]) and approach this task in a supervised manner by transforming NLI labels into contrastive pairs. Specifically, we leverage the semantic relations already en- coded in the entailment and contradiction labels to build positive and negative pairs, while neutral cases, which do not provide unambiguous similarity or dis- similarity cues, are excluded from the contrastive dataset to avoid introducing noise. • Positive pairs are derived from entailment instances, where the premise and hypothesis express semantically consistent or semantically aligned mean- ings. This ensures that the encoder learns to place semantically similar sentences closer in the embedding space. • Negative pairs are derived from contradiction instances, where the hy- pothesis conflicts with the meaning of the premise. These pairs explicitly encourage the encoder to push apart semantically dissimilar sentences in the embedding space. This label-based construction strategy provides high-quality contrastive pairs tailored to Vietnamese NLU. Importantly, to avoid any data leakage, we only use the training splits of ViNLI and XNLI (Vi) to construct contrastive pairs. The official dev and test splits of ViNLI are never included in contrastive pre-training and are reserved exclusively for downstream evaluation. Algorithm 1 outlines the process of converting input data from NLI datasets into data appropriate for contrastive learning. Specifically, when using natural language inference datasets, we select the premise (x) and hypothesis (x + ) pairs with the entailment label as positive pairs (x, x + ), and then select the hypothesis (x − ) with the contradiction label as the negative pair (x, x − ) corresponding to the initial premise. The final output of the algorithm is a datasets for contrastive learning training: nli-based dataset. The statistics of the data we have for training contrastive learning are presented in Table 1. 3.3 Training with Contrastive Learning Contrastive learning is a promising approach for training models to distinguish between semantically similar and dissimilar sentence pairs, which is cru- cial for generating high-quality sentence embeddings. However, low-resource languages like Vietnamese face significant challenges due to the limited avail- ability of annotated data. Inspired by the supervised contrastive learning approach in SimCSE [1], which significantly improves sentence embeddings by leveraging natural language inference datasets as labeled pairs, we pro- pose ViCLSR — a model that employs supervised CL to improve sentence embeddings specifically for Vietnamese. Figure 3 illustrates the architecture of the ViCLSR model, which takes three sentences as input: one positive pair (Sentence X and Sentence Y: x, x + ) and one negative pair (Sentence X and Sentence Z: x, x − ). This architecture leverages the XLM-R encoder [12], a multilingual pre-trained model that has Springer Nature 2021 L A T E X template Tin et al.11 Sentence Y y1...yn + Shared XLM-R Encoder E y1 E... E yn Sentence X x1...xm ++ E x1 E... E xm z1...zk ++ E z1 E... E zk Sentence Z T y1 T... T yn Tx1 T... TxmTz1 T... Tzk S Y S X S Z Cosine (S x , S y )Cosine (S x , S z ) + ++ Original Sentences Contextualized Embedding Positional Encoding Transformer Encoder Pooling Layer Sentence Similarity Input Embedding Sentences Positional Embedding + Multi-head Attention Add & Norm Feed Forward Add & Norm Features Supervised Contrastive Loss Contrastive Loss PN XLM-R Encoder Positive PairsNegative Pairs + Figure 3: Our Contrastive Learning Architecture with a shared XLM-R en- coder. demonstrated strong performance across a wide range of NLU tasks and has supported 100 languages, making it easier to develop similar frameworks for other low-resource languages, and includes the following components. • Input Representation: All three sentences (x,x + ,x − ) are first tokenized using the default XLM-RobertaTokenizer provided by HuggingFace. Each tokenized sentence is then transformed into input embeddings through the following steps: – Token and Contextualized Embeddings: Words are mapped to sub- word token embeddings by the XLM-R tokenizer, which are subsequently contextualized by the shared XLM-R encoder to capture their semantic meaning in context. – Positional Embeddings: Learned positional embeddings are added to encode word order, which is crucial for representing sentence structure. • Transformer Encoder: The input embeddings are passed through a shared XLM-R encoder which consists of multiple stacked transformer blocks. Each block follows the standard architecture with the following components: – Multi-head Attention: This mechanism enables the model to attend to different parts of the input sequence simultaneously, capturing complex relationships between words. Springer Nature 2021 L A T E X template 12 Tin et al. – Add & Norm: A residual connection and normalization step are applied to stabilize training. – Feed Forward Network: A fully connected layer refines the output further. – Add & Norm: Another residual connection and normalization complete the transformer block. • Pooling Layer: The output features from the encoder are aggregated into fixed-size vector representations for each sentence (S x ,S y ,S z ) using a pooling operation. • Similarity Computation and Loss Function: – Measures similarity between the positive pair (S x ,S y ) and the negative pair (S x ,S z ). – The model is trained using a supervised contrastive loss function, which minimizes the distance between embeddings of positive pairs while max- imizing the distance between negative pairs. This ensures that the resulting sentence embeddings are well-aligned semantically. Similar to SimCSE [1] and DiffCSE [5], in ViCLSR, we use a cosine similarity-based, temperature-scaled supervised contrastive loss to optimize the model. For a batch of training B =(x i ,x + i ,x − i ) N i=1 , where x i is the premise sentence, x + i is the entailment hypotheses sentence, and x − i is the contra- diction hypotheses sentence. The supervised contrastive loss for the i-th training instance is formulated as Equation (1). L i =− log exp sim(h i ,h + i )/τ P N j=1 exp sim(h i ,h + j )/τ + exp sim(h i ,h − j )/τ ! . (1) Here, N represents the batch size, whileh i ,h + i , andh − i denote the em- beddings of the premise (x), the entailment hypothesis (x + ), and the contradiction hypothesis (x − ), respectively. These embeddings are gen- erated by the XLM-R encoder, whereh i = f(x i ),h + i = f(x + i ), and h − i = f(x − i ), with f(·) representing the encoder function. The similarity function sim() is cosine similarity, which emphasizes the directional align- ment between vectors. The temperature hyperparameter τ controls the sharpness of the similarity distribution, facilitating more effective differ- entiation between positive and negative pairs by scaling their similarity scores. The similarity between two embeddings,h 1 andh 2 , is measured using cosine similarity, defined as Equation (2). sim(h 1 ,h 2 ) = h 1 ·h 2 ∥h 1 ∥h 2 ∥ .(2) Springer Nature 2021 L A T E X template Tin et al.13 This loss function encourages the model to minimize the distance between embeddings of positive pairs while maximizing the separation from nega- tive pairs, enabling the model to learn robust and semantically meaningful sentence representations. It is particularly well-suited for tasks requiring nuanced understanding of semantic similarity, such as Vietnamese natural language inference. 3.4 Fine-tuning for Downstream NLU Tasks Premise X (Context X) Multi-head Attention Hypothesis Y (Claim Y) Original Text Input Embedding XLM-RoBerta Classification Layer [CLS][SEP]x1[SEP]...xny1...ym E x1 E [CLS] E ... E xn E y1 E [SEP] E ... E ym E [SEP] Z x1 Z [CLS] Z ... Z xn Z y1 Z [SEP] Z ... Z ym Z [SEP] Feed Forward Z x1 Z [CLS] Z ... Z xn Z y1 Z [SEP] Z ... Z ym Z [SEP] Softmax P (label = C | X,Y) Sentence X Multi-head Attention [CLS]...x1[SEP]......xn E x1 E [CLS] E ... E ... E ... E xn E [SEP] Z x1 Z [CLS] Z ... Z ... Z ... Z xn Z [SEP] Feed Forward Z x1 Z [CLS] Z ... Z ... Z ... Z xn Z [SEP] Softmax P (label = C | X) Question + Answer Multi-head Attention Context Y [CLS][SEP]x1[SEP]...xny1...ym E x1 E [CLS] E ... E xn E y1 E [SEP] E ... E ym E [SEP] Z x1 Z [CLS] Z ... Z xn Z y1 Z [SEP] Z ... Z ym Z [SEP] Feed Forward Z x1 Z [CLS] Z ... Z xn Z y1 Z [SEP] Z ... Z ym Z [SEP] Softmax P (label = C | X,Y) (a)(b)(c) Statement X Figure 4: Architecture for Fine-tuning the ViCLSR Model on Various Down- stream NLU Tasks. Figure 4(a) Represents Fine-tuning for Natural Language Inference and Fact Checking Tasks, Where the Model Predicts the Semantic Relationship Between a Premise (Context) and Hypothesis (Claim). Figure 4(b) Illustrates Fine-tuning for Constructive Speech Detection Tasks, Pro- cessing Single Input Sentences. Figure 4(c) Demonstrates Fine-tuning for Multiple-Choice Machine Reading Comprehension Tasks, Where the Model Predicts the Correct Answer Choice Based on the Given Context and Ques- tion. Natural Language Understanding is a fundamental component that allows machines to interpret and process human language in a way that mirrors hu- man cognitive abilities. It plays a crucial role in enhancing the performance of language models in NLU tasks, which require a profound understanding of context and meaning. The final stage of our approach involves adapting the contrastively trained ViCLSR model to various Vietnamese downstream NLU tasks, such as natural language inference, fact checking, constructive speech detection, and machine reading comprehension. This fine-tuning process opti- mizes the representations of the model to align with task-specific objectives, ensuring enhanced performance and applicability across Vietnamese language Springer Nature 2021 L A T E X template 14 Tin et al. understanding tasks. Figure 4 illustrates the architecture for fine-tuning ViCLSR on these four downstream tasks described as follows. • Natural language inference and fact checking - Figure 4(a): The model predicts the semantic relationship in NLI tasks (entailment, contradiction, or neutral) between two inputs: a Premise and a Hypoth- esis. Meanwhile, in fact checking, the model evaluates the veracity of a Claim against a Context, classifying the relationship as support, refute, or not-enough-information. The process involves the following steps. – Input Transformation: The Premise (Context) - X and Hypothesis (Claim) - Y are encoded with special tokens ([CLS] at the start and [SEP] between segments). – Embedding: The text is transformed into input embeddings. – Encoding: The embeddings are passed through the XLM-R model, which leverages multi-head attention and a feed-forward network for contextual representation. – Classification: A classification layer with a softmax function pre- dicts the probability of a label given the premise and hypothesis: P(label = C∥X,Y ). • Constructive speech detection - Figure 4(b): For this tasks, where only a single input sentence (X) is used, the process is streamlined as follows. – Input Transformation: The sentence is tokenized and encoded with [CLS] at the start and [SEP] at the end. – Embedding and Encoding: The input is passed through the same pipeline as in Figure 3(a). – Classification: The representation of the [CLS] token is fed into a clas- sification layer with a softmax function to predict the probability of a label: P(label = C∥X) • Machine reading comprehension - Figure 4(c): The model is designed to predict the correct answer to a question based on a given context (Y). The process is as follows. – Input Transformation: The Question and each answer choice are com- bined to form multiple Statements (X), where each Statement represents a potential answer. These Statements are then paired with the Context (Y) for input into the model. – Embedding and Encoding: Each Statement (X) and Context (Y) pair is passed through the XLM-R model to produce embeddings, similar to the NLI process. – Answer Prediction: The model computes the probability distribution over the entailment label for each Statement (X) and Context (Y) pair. The label entailment corresponds to the correct answer, while contradic- tion corresponds to the incorrect answers. The Statement with the highest entailment probability is selected as the final answer. Springer Nature 2021 L A T E X template Tin et al.15 4 Experiments and Results 4.1 Benchmark Datasets To comprehensively evaluate the effectiveness of ViCLSR, we selected a diverse range of benchmark datasets that span multiple natural language processing tasks, including natural language inference (ViNLI [2]), fact checking (ViWik- iFC [13] and ViFactCheck [3]), constructive speech detection (UIT-ViCTSD [65]), and machine reading comprehension (ViMMRC2.0 [18]). Each dataset is aligned with the fine-tuning approach shown in Figure 4, ensuring compat- ibility and allowing for a comprehensive evaluation of the core capabilities of ViCLSR. The information of the datasets is summarized in Table 2. Table 2: Summary of the Benchmark Datasets and NLU Tasks Used to Evaluate the Performance of the ViCLSR Model. DatasetTrain Dev testTaskSourceDomain Metrics Classes ViNLI24,376 3,009 2,991 NLIOnline newsOpen domain Acc/F13 ViWikiFC16,738 2,090 2,091 Fact checkingWikipediaOpen domain Acc/F13 ViFactCheck5,062 723 1,447 Fact checkingOnline newsOpen domain F13 UIT-ViCTSD 7,000 1,000 2,000 Constructive speech detection Social mediaOpen domain Acc/F12 ViMMRC2.03,600 564 1,109 Multiple-choice MRCStudent textbooks LiteratureAcc- ViNLI [2] is a benchmark dataset for open-domain NLI research in Vietnam, focused on determining the logical relationship (entailment, contra- diction, or neutral) between a premise and a hypothesis. It comprises over 30,000 human-annotated premise-hypothesis pairs, carefully collected from more than 800 online news articles spanning 13 distinct topics. To ensure balance and prevent topic bias, the dataset is evenly distributed across devel- opment and test sets. ViNLI serves as a rigorous benchmark for evaluating models’ ability to discern nuanced semantic relationships in open-domain con- texts. The dataset consists of 24,376 training instances, 3,009 development instances, and 2,991 test instances. ViWikiFC [13] is the first Vietnamese fact checking corpus derived from Wikipedia. It involves assessing the veracity of claims by analyzing their se- mantic relationship with context and classifying them as support, refute, or not enough information based on evidence from reliable sources. This dataset includes over 20,000 manually annotated claims, supported by evidence from diverse topics such as history, geography, philosophy, and science. Annotators followed strict guidelines to ensure quality and consistency, which makes Vi- WikiFC an essential resource for evaluating the ability of models to verify factual information. The dataset comprises 16,738 training instances, 2,090 development instances, and 2,091 test instances. ViFactCheck [3] is a publicly available benchmark dataset designed for multi-domain Vietnamese news fact checking. This dataset comprises 7,232 human-annotated claim-evidence pairs categorized into three labels: support, refute, or not enough information, similar to ViWikiFC. It is sourced from Springer Nature 2021 L A T E X template 16 Tin et al. reputable Vietnamese online news outlets and covers 12 diverse topics. A no- table feature of ViFactCheck is its focus on real-world fact checking scenarios, where annotators are required to generate intricate claims that combine multi- ple pieces of evidence. ViFactCheck serves as a valuable resource for evaluating the performance of state-of-the-art pre-trained language models and large lan- guage models in fact checking tasks. It is designed to test these models’ ability to process complex information, including identifying semantic ambiguities and navigating intricate inferential chains. UIT-ViCTSD [65] focuses on detecting constructive and toxic speech in Vietnamese social media comments. The dataset comprises 10,000 annotated comments categorized into labels such as “constructive,” “non-constructive,” “toxic,” and “non-toxic.” In our experiments with ViCLSR, we specifically targeted the task of determining whether a comment is constructive or not. This dataset serves as a valuable benchmark for evaluating the ability of models to promote healthier online discussions by effectively filtering non- constructive, toxic content. UIT-ViCTSD includes 7,000 training instances, 1,000 development instances, and 2,000 test instances. ViMMRC2.0 [18] is a multiple-choice machine reading comprehension dataset designed to evaluate the ability of models to understand text by an- swering questions related to the content. This dataset is specifically created for educational purposes, containing 699 reading passages sourced from Viet- namese textbooks for grades 1 to 12. These passages, including both prose and poetry, are paired with 5,273 multiple-choice questions, with difficulty lev- els progressively increasing across grades. ViMMRC 2.0 serves as a valuable benchmark for evaluating language understanding in educational contexts. The dataset includes 3,600 training instances, 564 development instances, and 1,109 test instances. 4.2 Comparative Models Table 3: Summarizing the Detailed Information of Previous Pre-Trained Models Compared to Our Model (ViCLSR). Model#Params #Layers #Heads #Hidden #Vocab Language Data Source CL Strategy PhoBERT Base 135M121276864K Monolingual Wikipedia, NewsNo PhoBERT Large 370M2416102464K Monolingual Wikipedia, NewsNo mBERT179M1212768119K Multilingual WikipediaNo XLM-R Base 270M128768250K Multilingual CommonCrawlNo XLM-R Large 550M24161024250K Multilingual CommonCrawlNo CafeBERT550M24161024250K Multilingual Wikipedia, NewsNo DiffCSE125M121276850K Monolingual WikipediaYes ViCLSR550M24161024250K Multilingual CommonCrawl, NLIYes To evaluate the effectiveness of the ViCLSR model, we conducted a comprehensive experimental comparison with several state-of-the-art models, encompassing both monolingual and multilingual pre-trained models. These benchmarks, designed for a variety of NLU research tasks as presented in Springer Nature 2021 L A T E X template Tin et al.17 Section 4.1, help us determine the advantages of our proposed contrastive learning approach for Vietnamese text understanding. Table 3 provides de- tailed parameter specifications for each model compared to our proposed model. PhoBERT [14]: This model represents the first large-scale pre-trained language model specifically tailored for Vietnamese, based on the RoBERTa architecture. It was trained on a 20GB corpus consisting of Vietnamese Wikipedia and curated news datasets. With its focus on the linguistic characteristics unique to Vietnamese, PhoBERT has demonstrated strong per- formance across various Vietnamese NLU tasks, establishing itself as a critical model for evaluating Vietnamese language models. The model is available in two versions, PhoBERT Base and PhoBERT Large , both of which were evaluated in our experiments. CafeBERT [16]: This model is a Vietnamese-adapted language model introduced alongside the VLUE benchmark for comprehensive Vietnamese NLU evaluation. It builds upon the multilingual XLM-R Large architecture through continued pretraining on an 18GB corpus of Vietnamese text, includ- ing approximately 1GB of Wikipedia and 17GB of curated news articles. The model retains the configuration of XLM-R Large , consisting of 24 transformer layers, 16 attention heads, a hidden size of 1024, and roughly 550 million pa- rameters. This adaptation enhances the linguistic proficiency of the model in Vietnamese, enabling it to outperform both monolingual and multilingual baselines across diverse NLU tasks. By combining cross-lingual generaliza- tion from XLM-R with domain-specific Vietnamese knowledge, CafeBERT achieves state-of-the-art performance on multiple datasets within the VLUE benchmark. Multilingual BERT [11]: Pre-trained on a massive corpus spanning 104 languages, including Vietnamese, mBERT provides strong cross-lingual trans- fer capabilities. With 12 layers, 12 attention heads, 768 hidden units, and approximately 179 million parameters, it is a versatile model for multilin- gual NLU tasks. Trained using a masked language modeling (MLM) objective, mBERT leverages a WordPiece vocabulary of approximately 119,000 tokens. However, due to its generalized nature, it may not fully capture the nuances of Vietnamese as effectively as monolingual models. XLM-R [12]: This model demonstrates exceptional capacity for language comprehension, surpassing both mBERT and PhoBERT in many tasks. Its su- periority stems from its foundation in the robust RoBERTa architecture and its training on a massive multilingual dataset, CommonCrawl, encompassing 100 languages, including Vietnamese. XLM-R Base consists of 12 layers, 8 atten- tion heads, and 270 million parameters, while XLM-R Large is more powerful, featuring 24 layers, 16 attention heads, and 550 million parameters. XLM-R has achieved outstanding results across a wide array of NLU tasks, making it an ideal benchmark for evaluating multilingual performance. We evaluate both XLM-R Base and XLM-R Large versions in our experiments. Springer Nature 2021 L A T E X template 18 Tin et al. DiffCSE [5]: This model is an unsupervised contrastive learning frame- work that improves sentence embeddings through a difference-based ob- jective. It introduces a difference prediction loss inspired by equivariant contrastive learning to better capture semantic variations. The model em- ploys RoBERTa Base (12 layers, 12 attention heads, 768 hidden units, 125M parameters) as the encoder. DiffCSE achieves state-of-the-art performance on semantic textual similarity and transfer tasks, establishing a strong unsupervised baseline for sentence representation learning. 4.3 Experimental Settings For training ViCLSR, the model was fine-tuned using a supervised con- trastive learning approach based on the XLM-R Large architecture. The training configuration included a learning rate of 1e-5, a train batch size of 32, and a total of 10 training epochs. The pooler type was set to CLS, ensuring that the contextualized embedding of the [CLS] token was utilized for gener- ating sentence embeddings. These settings were carefully selected to maximize the performance of the model while ensuring stability during the fine-tuning process. For the fine-tuning of downstream tasks, all models including mBERT, PhoBERT, XLM-R, CafeBERT, DiffCSE, and the proposed Vi- CLSR—were trained under the same experimental settings to ensure fairness and comparability. A learning rate of 3e-5 was applied to base models, while large models were fine-tuned with a learning rate of 1e-5. Other hyperparame- ters, such as a batch size of 16 and 7 training epochs, were consistently applied across models, with minor adjustments to accommodate task-specific charac- teristics. This standardized fine-tuning procedure enabled an effective and fair evaluation of model capabilities across various Vietnamese NLU tasks. To ensure a robust and comprehensive evaluation of all models on down- stream tasks, a standardized experimental setup was employed. All experi- ments were conducted within a Google Colaboratory 1 environment, utilizing the computational power of NVIDIA L4 GPUs. The implementation leveraged the Transformers library provided by Huggingface 2 , which offers pre-trained models and fine-tuning utilities, streamlining the experimentation process. This consistent environment ensured comparability and reproducibility in all model assessments. 4.4 Evaluation Metrics To evaluate the performance of ViCLSR and other models on the down- stream tasks. For the tasks of natural language inference, fact checking, and constructive speech detection, the evaluation metrics of Accuracy and macro-averaged F1-score were applied following the evaluation methodolo- gies established in the corresponding datasets: ViNLI [2] for NLI, ViWikiFC 1 https://colab.research.google.com/ 2 https://huggingface.co/ Springer Nature 2021 L A T E X template Tin et al.19 [13] and ViFactCheck [3] for fact checking, and UIT-ViCTSD [65] for con- structive speech detection. By adopting these standard practices, a consistent and reliable comparison of model performance across tasks was ensured. • Accuracy: Accuracy measures the proportion of correctly classified in- stances among the total number of instances and is calculated as Equation (3). Accuracy = Number of Correct Predictions Total Number of Instances .(3) • Macro-averaged F1-Score: Macro-averaged F1-score provides a balanced measure of the model performance by combining precision and recall, considering all classes equally. The formula is presented as Equation (4). F1 = 2× Precision× Recall Precision + Recall .(4) Where Precision and Recall are defined as Equation (5) and Equation (6) respectively. Precision = True Positives True Positives + False Positives .(5) Recall = True Positives True Positives + False Negatives .(6) For the Multiple-Choice MRC task, we evaluated the models using the Ac- curacy (Acc) metric, following the approach used in the ViMMRC2.0 dataset [18]. This metric calculates the proportion of questions for which the model selects the correct answer from the answer options. The formula is presented as Equation (7). Acc = Number of Correct Answers Total Number of Questions .(7) 4.5 Experimental Results The experimental results of ViCLSR and several competitive baselines are summarized in Table 4. Overall, ViCLSR achieves superior performance across four representative Vietnamese NLU tasks — natural language inference, fact checking, constructive speech detection, and multiple-choice machine reading comprehension. The model consistently surpasses both monolingual models such as PhoBERT and CafeBERT, and multilingual counterparts including mBERT and XLM-R, as well as a recent contrastive baseline, DiffCSE. These results demonstrate that our supervised contrastive learning framework ef- fectively enhances Vietnamese sentence representations, leading to stronger generalization across diverse downstream understanding tasks. Springer Nature 2021 L A T E X template 20 Tin et al. Table 4: The Result of Model Performance with Our ViCLSR Across NLU Tasks. Models NLIFact Checking Constructive Speech Detection MRC ViNLIViWikiFCViFactCheckUIT-ViCTSDViMMRC2.0 AccF1AccF1F1AccF1Acc PhoBERT Base 72.87 72.7980.67 80.7077.7679.4078.5953.92 PhoBERT Large 75.93 75.8781.63 81.6279.7680.7076.8654.73 mBERT64.84 64.8375.93 76.0169.9478.9076.7053.38 XLM-R Base 71.59 71.5178.53 78.5681.1080.4078.1029.39 XLM-R Large 81.36 81.3185.11 85.1588.0281.7079.4457.52 CafeBERT82.06 82.0585.22 85.2487.4583.0081.5357.98 DiffCSE58.35 58.3466.14 66.1261.8481.8080.3641.11 ViCLSR82.86 82.8486.56 86.5788.7883.4082.2259.06 Compared to XLM-R↑1.50 ↑1.53↑1.45 ↑1.42↑0.76↑1.70 ↑2.78↑1.54 Compared to PhoBERT↑6.93 ↑6.97↑4.93 ↑4.97↑9.02↑2.70 ↑5.36↑4.33 Compared to CafeBERT↑0.80 ↑0.79↑1.34 ↑1.33↑1.33↑0.40 ↑0.69↑1.08 Compared to DiffCSE↑24.51 ↑24.50↑20.42 ↑20.45↑26.94↑2.35 ↑1.86↑17.95 For natural language inference, ViCLSR achieved an accuracy of 82.86% and a macro-averaged F1-score of 82.84% on the ViNLI dataset, establish- ing the highest performance among all evaluated models. Compared with the strongest multilingual baseline, XLM-R Large , ViCLSR achieved improve- ments of about 1.50% in accuracy and 1.50% in F1-score. The gap became considerably larger when compared with the monolingual PhoBERT Large , where ViCLSR performed roughly 7% better in accuracy, confirming the effectiveness of contrastive supervision over conventional encoder-only ar- chitectures. The model also slightly outperformed the Vietnamese-adapted CafeBERT by approximately 0.8% accuracy and showed a substantial mar- gin over the contrastive baseline DiffCSE, exceeding it by more than 24% in both accuracy and F1-score. These consistent improvements demonstrate that the proposed supervised contrastive learning framework captures richer and more discriminative semantic representations, enabling ViCLSR to model fine-grained sentence relationships and generalize effectively across diverse inference patterns. For fact checking, ViCLSR consistently outperformed both multilingual and monolingual baselines across the ViWikiFC and ViFactCheck datasets. The model achieved notable gains over XLM-R Large and CafeBERT, while maintaining a clear performance margin compared to PhoBERT Large and the contrastive DiffCSE model. Specifically, on ViWikiFC, ViCLSR surpassed PhoBERT Large by more than 4% F1-score and outperformed DiffCSE by over 20% F1-score; on ViFactCheck, the improvements were even larger over 9% F1-score against PhoBERT Large and more than 26% F1-score compared to DiffCSE. These substantial gains underscore the superior ability of ViCLSR to capture factual consistency and detect misleading or contradictory claims. Overall, the results confirm that the proposed supervised contrastive frame- work not only enhances semantic discrimination but also strengthens factual reasoning in Vietnamese text understanding. For constructive speech detection, ViCLSR achieved strong results on the UIT-ViCTSD dataset, reaching 83.40% accuracy and 82.22% F1-score. The Springer Nature 2021 L A T E X template Tin et al.21 model achieved a slight improvement over CafeBERT, but demonstrated a much larger advantage compared to PhoBERT Large , with nearly 3% higher accuracy, and maintained a clear lead of more than 2% accuracy over the contrastive DiffCSE model. These findings confirm that ViCLSR effectively captures the pragmatic and semantic nuances that distinguish constructive from non-constructive speech. The results further underscore the model’s ability to represent discourse-level meaning, which is essential for promoting healthier and more responsible online communication. For machine reading comprehension, ViCLSR achieved 59.06% accu- racy on the ViMMRC 2.0 dataset, showing a moderate improvement over all other models. Our model exhibited a more notable gain compared to PhoBERT Large , with approximately 4% higher accuracy, and maintained a substantial advantage of over 17% accuracy against the contrastive DiffCSE baseline. Although the overall accuracy is lower than in other tasks due to the inherent complexity of multi-sentence reasoning, these results demonstrate that ViCLSR possesses strong comprehension and reasoning capabilities, effectively generalizing across challenging question-answering contexts. The ViCLSR model effectively demonstrates the advantages of incor- porating supervised contrastive learning to enhance Vietnamese sentence representations, achieving state-of-the-art performance across a wide range of NLU tasks. The model consistently surpasses both monolingual encoders, such as PhoBERT and CafeBERT, and multilingual baselines, including mBERT and XLM-R, while also outperforming the contrastive DiffCSE model by a sub- stantial margin. These consistent improvements highlight the robustness and generalization capability of ViCLSR, reinforcing its potential as an effective and scalable framework for low-resource languages like Vietnamese. Furthermore, to provide empirical support regarding model convergence stability and explicitly address potential overfitting risks on smaller datasets like ViMMRC2.0, we present the training and validation loss curves for all evaluated tasks in Appendix A. These loss curves confirm that selecting the best model based on validation scores effectively prevents overfitting, ensuring stable performance across all evaluated tasks. 5 Results Analysis and Discussion In this section, we conduct a comprehensive analysis of the ViCLSR model to better understand its language understanding capabilities. We perform hyper- parameter and ablation studies to optimize the configuration of the model (see Section 5.1), followed by an evaluation of its semantic representation capability and the distribution of sentence embeddings (see Section 5.2 and Section 5.3). Next, we present a sensitivity analysis that examines the effect of different training data sizes on the performance of ViCLSR (see Section 5.4). In addi- tion, we analyze the attention mechanism to explore how the model handles sentence relationships (see Section 5.5). Finally, we provide recommendations for adapting the model to other low-resource languages (see Section 5.6). These Springer Nature 2021 L A T E X template 22 Tin et al. analyses aim to offer valuable insights into the strengths and limitations of ViCLSR across various aspects of natural language understanding. 5.1 Hyperparameter and Ablation Studies Temperature Hyperparameter τ: To evaluate the impact of the tempera- ture hyperparameter on the performance of the proposed ViCLSR model, we investigate its effect in the contrastive loss function. The temperature controls the sharpness of the probability distribution in the contrastive learning objec- tive, influencing how the model differentiates between similar and dissimilar sentence pairs. Following the experimental protocol of SimCSE, we treat τ as a fixed global scaling parameter and determine its value through grid search on the development set, keeping it constant during pretraining to enable con- trolled analysis of its effect. In this experiment, ViCLSR was trained with multiple τ values ranging from 0.001 to 1 and subsequently fine-tuned on five downstream NLU tasks: ViNLI, ViWikiFC, ViFactCheck, UIT-ViCTSD, and ViMMRC2.0. τ0.001 0.010.050.10.51 ViNLI82.66 82.48 83.28 82.04 82.75 81.46 ViWikiFC87.70 86.65 88.56 86.79 86.75 87.03 ViFactCheck 88.38 88.11 87.97 87.00 87.23 85.20 UIT-ViCTSD 81.59 82.30 82.75 82.35 80.55 82.70 ViMMRC2.055.67 55.62 56.38 55.85 55.11 55.02 Table 5: Impact of Temperature Hyperparameter τ in Contrastive Loss on Fine-Tuning Accuracy for Downstream NLU Tasks. The results in Table 5 present the development set accuracy of ViCLSR across downstream tasks when fine-tuned with different values of τ. The findings indicate that the choice of the temperature hyperparameter has a significant effect on model accuracy across various NLU tasks. The best over- all performance is observed at τ = 0.05, achieving the highest accuracy on most datasets, such as 83.28% on ViNLI, 88.56% on ViWikiFC, 82.75% on UIT-ViCTSD, and 56.38% on ViMMRC2.0. These results suggest that τ = 0.05 provides the best balance in the contrastive loss function, producing well-calibrated similarity scores. Deviating from this optimal value leads to performance degradation. When τ is too small (e.g., 0.001), the model accuracy drops due to overly sharp probability distributions, as seen in ViNLI (82.66%) and ViWikiFC (87.70%). Conversely, when τ is too large (e.g., 1.0), the accuracy also declines (81.46% on ViNLI and 55.02% on ViMMRC2.0), likely because a flatter probability distribution reduces contrastive learning effectiveness. Interestingly, ViCLSR achieves its highest accuracy on the ViFactCheck dataset when trained with τ = 0.001, reaching 88.38%. This observation suggests that a sharper contrastive distribution can occasionally benefit datasets that require finer-grained seman- tic discrimination. Overall, these analyses reinforce the importance of properly Springer Nature 2021 L A T E X template Tin et al.23 tuning the temperature hyperparameter to maximize sentence representation quality and overall model performance. Recent studies have further explored dynamic or adaptive temperature scheduling strategies to enhance contrastive learning optimization [66], where the temperature is adjusted during training to account for evolving embed- ding distributions or varying negative sample hardness. Such approaches have shown potential in improving convergence stability and boundary refinement, particularly in unsupervised or large-scale heterogeneous settings. In contrast, our framework adopts a fixed temperature following SimCSE [1] and DiffCSE [5], as our supervised NLI-based setup already provides structured and seman- tically grounded contradiction pairs that serve as informative hard negatives. Nonetheless, our sensitivity analysis across a wide range of τ values offers insight into how temperature scaling affects representation geometry in this su- pervised context. Incorporating adaptive temperature scheduling into ViCLSR remains a promising extension that could further refine training dynamics without altering the core supervised contrastive formulation. Ablation Tests on Auxiliary MLM Objective: To investigate whether incorporating the auxiliary Masked Language Modeling (MLM) objective im- proves the performance of ViCLSR on Vietnamese NLU tasks, we conducted an ablation study to assess its effect during training. The weighting parame- ter λ controls the influence of the MLM objective on the overall loss, allowing us to balance the contributions of the contrastive and MLM objectives, simi- lar to the approach of Gao et al. (2021) [1], where the overall loss in Eq. 1 is formulated as L = L CL + λL MLM . In this experiment, ViCLSR was trained with multiple λ values (0.001, 0.01, 0.05, 0.1, 0.5, and 1) and compared with a baseline model trained without the MLM objective (w/o MLM). The models obtained from different λ settings were then fine-tuned and evaluated on down- stream NLU tasks, including ViNLI, ViWikiFC, ViFactCheck, UIT-ViCTSD, and ViMMRC2.0. The results in Table 6 present the development set accuracy of ViCLSR across these downstream tasks under different λ configurations. The findings show that excluding the MLM objective (w/o MLM) produces the highest ac- curacy on most tasks, such as 83.28% on ViNLI and 88.56% on ViWikiFC. When the MLM objective is included, model performance generally declines across datasets, except for ViFactCheck, where ViCLSR achieves its best re- sult at λ = 0.05. This observation suggests that while the MLM objective can enhance token-level representations, it may also interfere with the contrastive objective optimized for sentence-level semantic alignment. Consequently, ex- cluding the MLM objective is more effective for tasks focusing on sentence-level understanding, whereas its inclusion should be applied selectively for datasets that benefit from richer token-level contextualization. Masking Rate: To further explore the impact of the auxiliary MLM objective and address the concern regarding masking configurations, we con- ducted an additional analysis focusing on masking-rate variations under a Springer Nature 2021 L A T E X template 24 Tin et al. λw/o 0.001 0.010.050.10.51 ViNLI83.28 82.08 83.10 82.04 82.57 82.35 82.44 ViWikiFC88.56 86.41 86.99 88.18 86.89 87.99 87.13 ViFactCheck 87.97 87.11 87.28 88.11 86.86 87.55 86.58 UIT-ViCTSD 82.75 81.69 82.30 82.49 82.23 82.15 80.40 ViMMRC2.0 56.38 54.28 55.27 55.73 54.78 54.78 54.28 Table 6: Effect of Auxiliary MLM Objective on Downstream NLU Tasks Performance Under Different λ Settings. fixed masking paradigm. In the previous experiments, the weighting parame- ter λ controlled the contribution of the MLM objective to the overall loss. As discussed earlier, the configuration without the MLM objective (w/o MLM) achieved the highest overall performance, while a small contribution of the MLM loss (λ = 0.05) produced relatively competitive results on several tasks such as ViWikiFC, ViFactCheck, and UIT-ViCTSD. Importantly, our backbone model, XLM-R, already employs dynamic subword-level masking, where mask positions are re-sampled across epochs. This dynamic mechanism introduces stochastic regularization and exposes the model to diverse contextual patterns without requiring architectural modifica- tion. While alternative masking paradigms (e.g., whole-word or span masking) alter the structural granularity of token corruption, our objective is to isolate the interaction between contrastive supervision and contextual reconstruction under a controlled setting. Therefore, rather than modifying the masking paradigm itself, we system- atically vary its strength. Specifically, under the configuration that includes the MLM objective (λ = 0.05), we trained ViCLSR with masking rates of 10%, 15%, 20%, 30%, 40%, and 50%, and compared them against the com- monly used default rate of 15% [11]. After pretraining under each masking configuration, the resulting ViCLSR models were fine-tuned and evaluated on five downstream NLU tasks: ViNLI, ViWikiFC, ViFactCheck, UIT-ViCTSD, and ViMMRC2.0. The development set accuracies of these configurations are summarized in Figure 5. Overall, the results in Figure 5 show that the performance of ViCLSR remains relatively stable across moderate masking rates, with the optimal con- figuration varying slightly across tasks. The commonly used masking rate of 15% yields the highest accuracy for ViNLI, ViFactCheck, and UIT-ViCTSD, whereas a higher masking rate of 30% performs best for ViWikiFC and ViMMRC2.0. Very low masking rates (e.g., 10%) provide limited contextual perturbation, which may reduce the diversity of reconstruction signals and limit abstraction. Conversely, excessively high masking rates (e.g., 50%) re- move too much contextual information, which may hinder stable sentence-level abstraction. These findings indicate that the masking rate strongly influences the balance between token-level reconstruction and sentence-level abstraction, and that maintaining a moderate range of 15%–30% allows ViCLSR to achieve stable and well-balanced performance across Vietnamese NLU tasks. Springer Nature 2021 L A T E X template Tin et al.25 ViNLIViWikiFCViFactCheckUIT-ViCTSDViMMRC2.0 0 20 40 60 80 100 Tasks Accuracy (%) 10% 15% 20% Masking Rate Accuracy of ViCLSR by Tasks with Different Masking Rate 30% 40% 50% Figure 5: Impact of masking rate during contrastive learning on the down- stream performance of ViCLSR across five Vietnamese NLU tasks. Pooling Strategies: Besides, we evaluated the impact of different pool- ing strategies on the effectiveness of sentence representations in ViCLSR. Pooling methods aggregate token-level embeddings into a single sentence- level representation, which significantly influences the performance of sentence embeddings. We tested four pooling methods: [CLS] Pooling (using embed- dings of [CLS] token), Mean Pooling (averaging embeddings of all tokens), First-Last Pooling (averaging embeddings from the first and last layers), and Top-2 Pooling (averaging embeddings from the top two layers). The model was fine-tuned on the ViNLI, ViWikiFC, ViFactCheck, UIT-ViCTSD, and ViMMRC2.0 datasets for each strategy, and the fine-tuning accuracy on dev set was recorded in Table 7. Pooling Type [CLS] Mean First-Last Top-2 ViNLI83.28 82.9382.6682.50 ViWikiFC88.56 87.3787.5187.56 ViFactCheck87.97 87.0087.1487.00 UIT-ViCTSD 82.75 82.6482.2482.02 ViMMRC2.056.38 55.4955.1655.92 Table 7: Impact of Different Pooling Strategies on Downstream NLU Tasks. The [CLS] Pooling method consistently achieves the highest accuracy among all evaluated strategies, highlighting its effectiveness in capturing global sentence-level semantics. Specifically, it outperforms all other methods across datasets, achieving 83.28% on ViNLI, 88.56% on ViWikiFC, and 87.97% on ViFactCheck, demonstrating its robustness across different tasks. The re- sults show that using the [CLS] token is more effective than averaging across multiple layers. In contrast, Mean Pooling, which averages all token embeddings, results in a slight performance drop, with scores of 82.93% on ViNLI and 87.37% on Vi- WikiFC, likely due to feature dilution. First-Last Pooling and Top-2 Pooling Springer Nature 2021 L A T E X template 26 Tin et al. produce competitive but suboptimal results, showing that deeper represen- tations do not always lead to better accuracy. These findings emphasize the critical role of pooling strategies in optimizing sentence embeddings, as an in- effective method can limit model performance even with strong pre-trained representations. [CLS] Pooling remains the most effective, reinforcing the im- portance of selecting a strategy that preserves key semantic structures for downstream NLU tasks. 5.2 Semantic Representation Capability To evaluate the semantic representation capability of the ViCLSR model, which is trained using contrastive learning to optimize for capturing sentence similarity and dissimilarity, we compared our model with other baseline mod- els on the information retrieval (IR) task. The evaluation was conducted on the fact checking dataset (ViWikiFC), where each claim sentence is paired with one most relevant evidence sentence in the context to verify its factuality. For this experiment, we filtered the dataset to include only samples with the support label and computed accuracy across four scenarios: Top 1, Top 3, Top 5, and Top 10 most similar sentences predicted by the model. We evaluate according to Accuracy@TopK as shown in Equation (8). Accuracy@TopK = 1 n n X i=1 I (y i ∈ TopK(x i ,C)).(8) where: • n: The total number of claim sentences in the dataset. • x i : The i-th claim sentence. • y i : The corresponding evidence sentence for the claim x i . • C: The set of context sentences. • TopK(x i ,C): The TopK set of sentences that the model predicts as the most similar to x i in C, based on cosine similarity. • I(·): The indicator function, which returns: – 1, if y i ∈ TopK(x i ,C); – 0, otherwise. To obtain sentence representations, we employed two pooling strategies: [CLS] pooling and Mean pooling. Table 8 compares the retrieval accuracy of our model, ViCLSR, against several strong baselines, including XLM- R Large , PhoBERT Large , mBERT, CafeBERT, and the contrastive learning model DiffCSE, across different Accuracy@TopK settings. The results show that our proposed model, ViCLSR, achieves the best retrieval performance, significantly surpassing both multilingual models (mBERT and XLM-R Large ), monolingual Vietnamese models (PhoBERT Large and CafeBERT), and the contrastive learning baseline DiffCSE across all four TopK scenarios and both pooling strategies. Notably, ViCLSR demonstrates exceptional accuracy at the top ranks (Top 1 and Top 3), highlighting its strength in prioritizing the most Springer Nature 2021 L A T E X template Tin et al.27 relevant evidence, which is crucial for IR tasks. Furthermore, while broader ranges like Top 10 indicate that there is potential to further improve the ranking quality, ViCLSR consistently retrieves relevant sentences effectively, indicating its robustness and practical applicability in retrieval scenarios where comprehensive evidence coverage is required. Table 8: Comparison of IR System Accuracy Between ViCLSR and Other Models on ViWikiFC Using [CLS] Pooling and Mean Pooling. Pooling Type ModelsTop 1 Top 3 Top 5 Top 10 [CLS] Pooling ViCLSR82.62 90.96 93.50 96.04 XLM-R Large 16.52 19.06 19.3521.18 PhoBERT Large 24.43 28.38 29.5131.92 mBERT48.72 59.32 63.5567.65 CafeBERT42.37 50.00 54.9459.60 DiffCSE40.68 49.01 52.5456.50 Mean Pooling ViCLSR83.05 92.51 94.35 97.32 XLM-R Large 54.24 61.02 64.6967.09 PhoBERT Large 76.69 86.02 87.9990.82 mBERT78.25 86.86 89.4192.23 CafeBERT72.60 78.95 81.3684.89 DiffCSE44.07 52.82 56.9262.43 The differing performance of [CLS] pooling and Mean pooling in ViNLI (Table 7) and IR (Table 8) tasks reflects the distinct requirements of these tasks. For ViNLI, which focuses on sentence-level semantic relationships, the [CLS] token captures global semantics, making it more effective. In contrast, IR requires retrieving evidence from a large pool of context sentences, where Mean pooling aggregates information from all tokens, ensuring that relevant tokens contribute to the representation. This broader coverage makes Mean pooling better suited for IR tasks. 5.3 Distribution of Sentence Embeddings We analyze the distribution of sentence embeddings generated by the Vi- CLSR model (ViCLSR) on the ViNLI dataset. The analysis focuses on two primary metrics: Alignment refers to how closely the embeddings of semanti- cally similar pairs are grouped together and how effectively the embeddings of semantically dissimilar pairs are pushed apart, while Uniformity evaluates the extent to which embeddings are evenly distributed across the hypersphere, ensuring they are not overly clustered or concentrated in specific regions. The goal of this analysis is to evaluate the quality of embeddings produced by the model and compare it against baseline models, including XLM-R Large , Springer Nature 2021 L A T E X template 28 Tin et al. ViCLSR XLM-RPhoBERT mBERT CafeBERTDiffCSE 0.00 0.07 0.14 0.21 0.29 0.2858 0.0074 0.0719 0.1673 0.1324 0.0991 (a) Alignment - E ViCLSR XLM-RPhoBERT mBERT CafeBERTDiffCSE 0.00 0.15 0.31 0.46 0.61 0.6102 0.0078 0.0767 0.1755 0.1387 0.1086 (b) Alignment - C ViCLSR XLM-RPhoBERT mBERT CafeBERTDiffCSE -2.50 -1.88 -1.26 -0.63 -0.01 -2.5012 -0.0090 -0.1808 -0.4573 -0.4603 -0.3621 (c) Uniformity Figure 6: Analysis of Alignment and Uniformity on ViNLI Dataset Compar- ing ViCLSR with Other Models Using Sentence Embeddings Derived from CLS Pooling (E: Alignment for Positive Pairs, C: Alignment for Negative Pairs). PhoBERT Large , mBERT, CafeBERT, and DiffCSE. These metrics provide in- sights into the ability of the models to encode semantic relationships effectively and maintain generalizable embedding distributions. The Alignment metric was calculated as the squared distance between the embeddings of semantically similar pairs (E) and dissimilar pairs (C). Specifically, positive pairs correspond to sentence pairs with an entailment relationship, while negative pairs correspond to sentence pairs with a contra- diction relationship in the ViNLI dataset. Alignment is calculated as Equation (9). ℓ align ≜ E (x,x + )∼p pos ∥f(x)− f(x + )∥ 2 .(9) The Uniformity metric was calculated to evaluate how well the embed- dings are distributed uniformly across the hypersphere. This computation was Springer Nature 2021 L A T E X template Tin et al.29 performed on all sentence pairs present in the ViNLI dataset. Uniformity is calculated as Equation (10). ℓ uniform ≜ log E x,y∼p data e −2∥f(x)−f(y)∥ 2 .(10) Figure 6b shows that ViCLSR achieves an Alignment - C score of 0.6102, significantly outperforming all other models. This demonstrates the excep- tional ability of the model to separate semantically dissimilar sentence pairs (e.g., contradiction pairs). This high score indicates that ViCLSR is highly effective in pushing embeddings of contradictory sentences apart, which is crucial to accurately capture semantic differences in sentence relationships. With an Uniformity score of -2.5012 of the ViCLSR model as shown in Figure 6c shows that our model outperforms the baseline models by a wide margin. This indicates that the embeddings produced by the model are evenly distributed across the hypersphere, avoiding over-clustering or collapsing into dense regions. This distribution ensures the generalizability of the model and prevents it from being biased towards specific regions of the embedding space. In contrast, other models such as XLM-R Large (-0.0090), PhoBERT Large (- 0.1808), DiffCSE (-0.3621), mBERT (-0.4573), and CafeBERT (-0.4603) show much poorer Uniformity scores, suggesting that their embeddings are less evenly distributed and may suffer from over-concentration, which hinders their ability to generalize across diverse semantic contexts. Although ViCLSR performs as expected on Alignment E, with a score of 0.2858 (indicating good grouping of similar sentence pairs) as shown in Figure 6a, it is noticeable that other models, such as XLM-R Large , PhoBERT Large , and DiffCSE exhibit unusually low scores. This suggests a phenomenon of “false low Alignment - E”, where embeddings of semantically similar pairs are not adequately grouped. This behavior, combined with similar Alignment - E and Alignment - C values in these models, points to a possible issue: over-concentration of embeddings in the hypersphere. Such over-concentration is supported by the extremely low Uniformity scores of these models, in- dicating that their embeddings collapse into dense regions, leading to poor differentiation between similar and dissimilar sentences. Table 9: Comparison of Retrieved Top-3 Examples by ViCLSR Against XLM- R and PhoBERT on the ViWikiFC Dataset. Retrieved sentences highlighted in blue-green text represent correct retrievals, whereas those in bittersweet text denote incorrect retrievals. ViCLSRXLM-R Large PhoBERT Large Query 1: "Tô Định là kẻ vô cùng xấu xa và tàn độc." (Tô Định was an extremely wicked and cruel man.) Gold evidence: "Năm 39, thái thú quận Giao Chỉ là Tô Định tàn ác, giết chồng của Trưng Trắc là Thi Sách." (In the year 39, Tô Định, the cruel governor of Giao Chỉ district, killed Thi Sách, the husband of Trưng Trắc.) Springer Nature 2021 L A T E X template 30 Tin et al. ViCLSRXLM-R Large PhoBERT Large #1. Năm 39, thái thú quận Giao Chỉ là Tô Định tàn ác, giết chồng của Trưng Trắc là Thi Sách. (In the year 39, Tô Định, the cruel gov- ernor of Giao Chỉ dis- trict, killed Thi Sách, the husband of Trưng Trắc.) #1. Nhà Thương thường phái quân đội đi chiến đấu chống lại những bộ tộc lân cận. (The Shang dynasty of- ten sent its armies to fight against neighboring tribes.) #1. Điều này hiển nhiên rất khó khiến người ta tin tưởng. (This is obviously very hard for people to be- lieve.) #2. Năm Cam, Khánh Trắng là một số ví dụ về băng nhóm tội phạm có tổ chức. (Năm Cam and Khánh Trắng are examples of orga- nized criminal gangs.) #2. Các trang phục của người dân tộc thiểu số cũng có thể sử dụng. (The traditional costumes of ethnic minorities can also be used.) #2. Theo cư dân địa phương, bà rất linh thiêng. (According to local residents, she is considered very sacred.) #3. Năm 931, Dương Đình Nghệ là tướng cũ của Khúc Hạo đem quân đánh phủ thành Đại La. (In 931, Dương Đình Nghệ, a former general of Khúc Hạo, led his army to attack the Đại La citadel.) #3. Chính quyền này kiểm soát chặt chẽ mọi mặt của đời sống. (This govern- ment tightly con- trols all aspects of life.) #3. Chính quyền này kiểm soát chặt chẽ mọi mặt của đời sống. (This government tightly con- trols all aspects of life.) Query 2: "Singapore sở hữu 12 khu vực công nghiệp." (Singapore has 12 industrial zones.) Gold_evidence: "Singapore có 12 khu vực công nghiệp lớn, trong đó lớn nhất là Khu công nghiệp Jurong." (Singapore has 12 major industrial zones, the largest of which is the Jurong Industrial Estate.) #1. Singapore có 12 khu vực công nghiệp lớn, trong đó lớn nhất là Khu công nghiệp Jurong. (Singapore has 12 major indus- trial zones, the largest of which is the Jurong Industrial Estate.) #1. Tại đó, người dân bầu cho mọi việc. (At that place, the people vote on everything.) #1. Kinh tế của Palawan chủ yếu phụ thuộc vào nông nghiệp. (Palawan’s economy mainly depends on agriculture.) Springer Nature 2021 L A T E X template Tin et al.31 ViCLSRXLM-R Large PhoBERT Large #2. Singapore có cơ sở hạ tầng và một số ngành công nghiệp phát triển cao hàng đầu châu Á. (Singa- pore has world-class infrastructureand some of the most ad- vanced industries in Asia.) #2. Các phương tiện có động cơ không được tham gia lưu thông. (Mo- tor vehicles are not allowed to operate on the roads.) #2. Hệ thống giao thông công chánh ở Singapore rất phát triển. (Singa- pore’s public transporta- tion system is highly de- veloped.) #3. Quảng Nam có Khu kinh tế mở Chu Lai nổi tiếng với nhà máy của THACO. (Quang Nam has the Chu Lai Open Eco- nomic Zone, which is famous for the THACO automobile plant.) #3. Và thực tế đúng là như vậy. (And that is indeed true.) #3. Các công ty có liên kết với chính phủ kiểm soát hầu hết truyền thông nội địa tại Singapore. (Government-linked com- panies control most of the domestic media in Singa- pore.) Query 3: "Vùng duyên hải có mức thủy triều trung bình. " (The coastal zone has an average tidal range.) Gold_evidence: "Vùng duyên hải bao phủ khu vực nằm giữa các mức thủy triều cao và thấp nhất, nó là khu vực chuyển tiếp giữa các điều kiện đại dương và đất liền." (The coastal zone covers the area between the highest and lowest tide levels; it is a transitional region between oceanic and terrestrial conditions.) #1. Các dòng biển phần nào đó điều hòa khí hậu. (Ocean cur- rents help regulate the climate to some ex- tent.) #1. 1942: Hiến chương Đại Tây Dương được ký kết. (1942: The At- lantic Charter was signed.) #1. Các khu vực nhỏ hơn của đại dương được gọi là các biển, vịnh hay một số các tên gọi khác. (Smaller areas of the ocean are called seas, bays, or by other names.) Springer Nature 2021 L A T E X template 32 Tin et al. ViCLSRXLM-R Large PhoBERT Large #2. Các khu vực nhỏ hơn của đại dương được gọi là các biển, vịnh hay một số các tên gọi khác. (Smaller areas of the ocean are called seas, bays, or by other names.) #2. Các dòng biển phần nào đó điều hòa khí hậu. (Ocean currents help regulate the climate to some extent.) #2. Nhờ nằm giữa biển Đông nên quần đảo Hoàng Sa có khí hậu điều hòa, không quá lạnh về mùa đông, không quá nóng về mùa hè nếu so với những vùng đất cùng vĩ độ trong lục địa. (Thanks to its lo- cation in the middle of the East Sea, the Paracel Islands enjoy a moderate climate — not too cold in winter and not too hot in summer compared to in- land areas at the same lat- itude.) #3. Nhờ nằm giữa biển Đông nên quần đảo Hoàng Sa có khí hậu điều hòa, không quá lạnh về mùa đông, không quá nóng về mùa hè nếu so với những vùng đất cùng vĩ độ trong lục địa. (Thanks to its location in the middle of the East Sea, the Paracel Islands enjoy a mod- erate climate — not too cold in winter and not too hot in summer compared to inland ar- eas at the same lati- tude.) #3. Các khu vực nhỏ hơn của đại dương được gọi là các biển, vịnh hay một số các tên gọi khác. (Smaller ar- eas of the ocean are called seas, bays, or by other names.) #3. 1942: Hiến chương Đại Tây Dương được ký kết. (1942: The Atlantic Charter was signed.) Query 4: "Nó có độ nóng chảy ở mức gần 30 độ C." (It has a melting point of around 30°C.) Gold_evidence: "Nó là một kim loại kiềm mềm, màu bạc, và với điểm nóng chảy là 28°C (83°F) khiến cho nó trở thành một trong các kim loại ở dạng lỏng tại hay gần nhiệt độ phòng." (It is a soft, silvery alkali metal, and with a melting point of 28°C (83°F), it is one of the few metals that are liquid at or near room temperature.) Springer Nature 2021 L A T E X template Tin et al.33 ViCLSRXLM-R Large PhoBERT Large #1. Ở trung tâm của Trái Đất, nhiệt độ có thể đạt tới 7000K và áp suất có thể lên tới 360 Gpa. (At the Earth’s core, temperatures can reach up to 7000 K and pressures can rise to 360 GPa.) #1. Đường phố được dành cho người đi bộ thưởng lãm. (The streets are reserved for pedestrians to stroll and enjoy the scenery.) #1. Nếu tình trạng tiến triển thành say nắng, thì da nóng, khô là điển hình khi các mạch máu giãn ra trong nỗ lực tăng mất nhiệt. (If the condition progresses to heatstroke, hot and dry skin is typical as blood vessels dilate in an effort to increase heat loss.) #2. Nếu tình trạng tiến triển thành say nắng, thì da nóng, khô là điển hình khi các mạch máu giãn ra trong nỗ lực tăng mất nhiệt. (If the condi- tion progresses to heat- stroke, hot and dry skin is typical as blood vessels dilate in an ef- fort to increase heat loss.) #2. Có một mức độ tiêm chủng cao. (There is a high level of vaccination coverage.) #2. Ở trung tâm của Trái Đất, nhiệt độ có thể đạt tới 7000K và áp suất có thể lên tới 360 Gpa. (At the Earth’s core, tempera- tures can reach up to 7000 K and pressures can rise to 360 GPa.) #3. Caesi có điểm nóng chảy ở 28,4°C (83,1°F), là một trong ít các kim loại nguyên tố ở dạng lỏng trong điều kiện gần nhiệt độ phòng. (Caesium has a melting point of 28.4°C (83.1°F) and is one of the few elemental met- als that exist in liq- uid form under condi- tions near room tem- perature.) #3. Nếu tình trạng tiến triển thành say nắng, thì da nóng, khô là điển hình khi các mạch máu giãn ra trong nỗ lực tăng mất nhiệt. (If the con- dition progresses to heatstroke, hot and dry skin is typical as blood vessels di- late in an effort to increase heat loss.) #3. Caesi có điểm nóng chảy ở 28,4°C (83,1°F), là một trong ít các kim loại nguyên tố ở dạng lỏng trong điều kiện gần nhiệt độ phòng. (Caesium has a melting point of 28.4°C (83.1°F) and is one of the few elemental metals that exist in liquid form un- der conditions near room temperature.) To complement the quantitative alignment–uniformity results, Table 9 presents qualitative retrieval examples that illustrate how the learned embed- ding geometry affects semantic evidence selection. For clarity, we compare ViCLSR with XLM-R Large (the multilingual backbone of our model) and Springer Nature 2021 L A T E X template 34 Tin et al. PhoBERT Large (a strong Vietnamese-specific baseline), representing multi- lingual and monolingual encoder architectures, respectively. For XLM-R Large and PhoBERT Large , the top-ranked sentences often achieve high cosine sim- ilarity scores but fail to capture the exact information required to verify the claim. In contrast, ViCLSR consistently retrieves semantically precise evi- dence, particularly in cases with explicit factual cues such as Query 1 and Query 2. However, more challenging scenarios arise in implicit semantic cases, as illustrated in Query 3 and Query 4, where the model must go beyond surface-level matching to perform deeper reasoning — such as numerical ap- proximation (e.g., “28°C” implying “around 30°C”) or conceptual inference (e.g., “average tidal range” referring to “between high and low tide levels”). These examples highlight the limitations of current models in handling implicit semantics and reasoning-intensive retrieval tasks. 5.4 Sensitivity to Training Data Size To further examine the robustness of ViCLSR in low-resource scenarios, we conducted a sensitivity analysis to evaluate how the model performs when the amount of available training data is limited. This experiment aims to assess the stability and scalability of the supervised contrastive learning framework when trained with varying data sizes, which is a common challenge in low-resource language settings. As described in Section 3.2, the contrastive dataset used for training con- sists of 8,594 instances derived from Vietnamese NLI corpora. To simulate different levels of data availability, we randomly sampled three subsets corre- sponding to 25%, 50%, and 75% of the original dataset. The ViCLSR model was trained independently on each subset while keeping all other hyperparam- eters fixed. After contrastive pretraining, the resulting models were fine-tuned on five downstream NLU tasks, including ViNLI, ViWikiFC, ViFactCheck, UIT-ViCTSD, and ViMMRC2.0. Figure 7 presents the accuracy obtained on the development sets for each task among ViCLSR models trained with dif- ferent portions of the original dataset and the baseline model trained with the full data. The results show a consistent upward trend as the amount of training data increases, indicating that ViCLSR effectively leverages additional contrastive supervision to refine semantic representations. More importantly, even with only 25% of the original data, the model maintains competitive performance across all tasks, suggesting strong generalization capability and robustness to data scarcity. These findings confirm that the proposed framework can function reliably in extreme low-resource conditions and is well-suited for adaptation to other languages with limited annotated resources. Springer Nature 2021 L A T E X template Tin et al.35 ViNLIViWikiFCViFactCheckUIT-ViCTSDViMMRC2.0 0 20 40 60 80 100 81.95 82.39 82.44 83.28 86.94 87.08 87.32 88.56 84.37 86.99 87.13 87.97 82.50 82.70 82.75 82.74 54.68 55.06 55.97 56.38 Tasks Accuracy (%) 25% 50% 75% 100% Training Data Percentage for ViCLSR Accuracy of ViCLSR by Tasks with Different Training Data Sizes Figure 7: Impact of training data size during contrastive learning on the downstream performance of ViCLSR across five Vietnamese NLU tasks. 5.5 Attention Analysis In this analysis experiment, we visualize the attention mechanisms of ViCLSR and the XLM-R model to compare how they handle semantic attention be- tween premise and hypothesis sentences in the NLI task. The goal of this analysis is to evaluate how the models focus on important keywords in the sentences and to highlight the effectiveness of ViCLSR in understanding the semantic relationships between sentences, in comparison to the XLM-R model. The attention visualizations from both models, extracted from the last layer of each as shown in Figure 8, reveal a clear difference in how they pro- cess information. In pairs of sentences such as premise = “Người phụ nữ đang chơi đàn guitar trong công viên” (A woman is playing guitar in the park) and hypothesis = “Một nhạc sĩ đang biểu diễn với nhạc cụ” (A musician is per- forming with a musical instrument) in Figures 8a and 8b, the ViCLSR model focuses attention on crucial keywords like “guitar” and “nhạc cụ (instrument)” or “phụ nữ (woman)” and “nhạc sĩ (musician)”. Additionally, in the pair of sen- tences premise = “Không khí lạnh tràn về thành phố tối nay” (Cold air flows into the city tonight) and hypothesis = “Đêm nay nhiệt độ giảm” (Tonight the temperature drops), as seen in Figures 8c and 8d, the attention of the ViCLSR model is predominantly on the words “lạnh (cold)” and “nhiệt độ giảm (tem- perature drops)”. In the pair premise = “Ăn nhiều muối tăng nguy cơ bệnh cao huyết áp” (Eating a lot of salt increases the risk of high blood pressure disease) and hypothesis = “Thức ăn quá mặn không tốt cho sức khỏe” (Too salty food is not good for health), as shown in Figures 8e and 8f, the words “muối (salt)” and “mặn (salty)” as well as “bệnh (disease)” and “không tốt cho sức khỏe (not good for health)” receive more attention from the ViCLSR model. In con- trast, the attention from the XLM-R model is less clearly demonstrated in the visualizations, indicating that ViCLSR has a stronger ability to understand semantics compared to XLM-R. Springer Nature 2021 L A T E X template 36 Tin et al. [CLS] Ng i ph n ang ch i àn guitar trong công viên [SEP] M t nh c s ang bi u di n v i nh c c [SEP] Tokens [CLS] Ngi ph n ang chi àn guitar trong công viên [SEP] Mt nhc s ang biu din v i nhc c [SEP] Tokens 0.02 0.04 0.06 0.08 0.10 0.12 0.14 Attention Weights (a) ViCLSR [CLS] Ng i ph n ang ch i àn guitar trong công viên [SEP] M t nh c s ang bi u di n v i nh c c [SEP] Tokens [CLS] Ngi ph n ang chi àn guitar trong công viên [SEP] Mt nhc s ang biu din v i nhc c [SEP] Tokens 0.05 0.10 0.15 0.20 0.25 0.30 Attention Weights (b) XLM-R [CLS] Không khí l nh tràn v thành ph t i nay [SEP] êm nay nhi t gi m [SEP] Tokens [CLS] Không khí l nh tràn v thành ph t i nay [SEP] êm nay nhit gim [SEP] Tokens 0.04 0.06 0.08 0.10 0.12 Attention Weights (c) ViCLSR [CLS] Không khí l nh tràn v thành ph t i nay [SEP] êm nay nhi t gi m [SEP] Tokens [CLS] Không khí l nh tràn v thành ph t i nay [SEP] êm nay nhit gim [SEP] Tokens 0.05 0.10 0.15 0.20 0.25 0.30 Attention Weights (d) XLM-R [CLS] n nhi u mu i t ng nguy c b nh cao huy t áp [SEP] Th c n quá m n không t t cho s c kh e [SEP] Tokens [CLS] n nhiu mu i t ng nguy c b nh cao huyt áp [SEP] Th c n quá m n không t t cho s c khe [SEP] Tokens 0.02 0.04 0.06 0.08 0.10 0.12 Attention Weights (e) ViCLSR [CLS] n nhi u mu i t ng nguy c b nh cao huy t áp [SEP] Th c n quá m n không t t cho s c kh e [SEP] Tokens [CLS] n nhiu mu i t ng nguy c b nh cao huyt áp [SEP] Th c n quá m n không t t cho s c khe [SEP] Tokens 0.05 0.10 0.15 0.20 0.25 0.30 Attention Weights (f) XLM-R Figure 8: Visualization of the Attention Map Between a Premise and Hy- pothesis Sentence Using the Pre-Trained XLM-R Model and Our ViCLSR Model. Springer Nature 2021 L A T E X template Tin et al.37 ViCLSR places more emphasis on words with strong semantic connections between the premise and hypothesis, enabling the model to more accurately identify the similarities and differences between the sentences. In contrast, the XLM-R model distributes attention more broadly and does not focus on specific words, instead highlighting the overall context of the sentence. This suggests that while XLM-R can understand the global meaning of a sentence, it lacks the nuance required to attend to specific semantic relationships between the sentences. Therefore, ViCLSR proves superior in tasks requiring semantic relationship understanding, thanks to its ability to learn and recognize clearer semantic connections. 5.6 Recommendation for Other Low Resource Languages For low-resource languages, the key challenge lies in the availability of anno- tated datasets, which are crucial for training supervised contrastive learning models. In the case of Vietnamese, we utilized existing NLI datasets to create positive and negative sentence pairs, which are essential for learning effec- tive sentence embeddings. The ViCLSR framework can be applied to other low-resource languages in the same way: by identifying or creating NLI-style datasets that include sentence pairs labeled with entailment and contradic- tion. This flexibility of ViCLSR allows it to handle a variety of low-resource languages, such as Thai, Indonesian, Burmese, and Lao, as long as the key re- quirement that positive and negative sentence pairs are met. If NLI datasets are not available, alternative datasets such as paraphrase or fact checking datasets can be repurposed to fit this task. Additionally, augmenting the dataset with synthetic data generated through machine translation, sentence paraphrasing, or other augmentation techniques ensures a more diverse and rich training set, further enhancing performance in low-resource settings. In addition to dataset preparation, the choice of pre-trained multilingual models plays a critical role in transferring knowledge from high-resource lan- guages to low-resource ones. By leveraging models like XLM-R, which have been pre-trained on a vast multilingual corpus, the ViCLSR framework can be fine-tuned on the NLI dataset of the target language. This fine-tuning process significantly improves the ability of the model to understand the unique se- mantic relationships of the target language, even when there is limited labeled data. The adaptability of ViCLSR makes it a powerful tool for low-resource languages, as it capitalizes on the strength of multilingual pre-trained models and applies contrastive learning to improve sentence embeddings. For lan- guages with particularly complex syntax or semantic nuances, adjusting the model architecture or the data preparation process is necessary. Ultimately, the success of applying the ViCLSR framework to other low-resource languages depends on the careful preparation of data, the use of suitable pre-trained models like XLM-R, and ongoing research to refine the framework for specific linguistic characteristics. Springer Nature 2021 L A T E X template 38 Tin et al. 6 Conclusion and Future Works In this article, we introduced ViCLSR, a supervised contrastive learning model specifically designed to optimize Vietnamese sentence embeddings. The core contribution of this work lies in adapting existing Vietnamese NLI datasets for contrastive learning, enabling the creation of high-quality sentence embed- dings tailored to the linguistic characteristics of Vietnamese. Our experiments demonstrated that ViCLSR consistently outperforms both monolingual and multilingual pre-trained models, including PhoBERT, mBERT, XLM-R, and CafeBERT, as well as a contrastive learning baseline DiffCSE, across vari- ous Vietnamese NLU tasks such as natural language inference, fact checking, constructive speech detection, and machine reading comprehension. In particular, ViCLSR surpassed the strong multilingual XLM-R Large model by 1.5% accuracy on ViNLI, exceeded the monolingual PhoBERT Large by over 6.9% accuracy, and outperformed the contrastive DiffCSE model by more than 24% accuracy on the same benchmark. These substantial improve- ments highlight the effectiveness of our supervised contrastive framework in capturing fine-grained semantic relationships and complex reasoning patterns. Furthermore, the analysis of sentence embedding distributions reveals that ViCLSR produces embeddings that are both well-aligned and uniformly dis- tributed, demonstrating the robustness of contrastive learning in generating high-quality and semantically coherent representations for Vietnamese. In future, further adaptation of contrastive learning techniques will be explored to fully leverage the unique linguistic features of Vietnamese. In particular, extending ViCLSR with advanced mechanisms such as adap- tive negative sampling and dynamic temperature adjustment represents a promising direction that could further enhance performance. Additionally, a primary focus of future research will be transitioning from encoder-only ar- chitectures to sequence-to-sequence frameworks to address generative tasks such as abstractive summarization, machine translation, and dialogue sys- tems. Our supervised contrastive learning strategy will be integrated with Vietnamese-specific generative models like BARTpho [67] and ViT5 [68]. ViCLSR’s high-quality embeddings can enhance Retrieval-Augmented Gener- ation (RAG) for Vietnamese-centric LLMs such as GPT [29] and DeepSeek [32]. Finally, image captioning datasets (e.g., UIT-ViIC [69]) can be leveraged to explore multimodal contrastive learning enhanced with adversarial data from ViANLI [62], aligning Vietnamese text with visual semantics. Acknowledgement This research is funded by Vietnam National University HoChiMinh City (VNU-HCM) under grant number DS.C2025-26-10. Springer Nature 2021 L A T E X template Tin et al.39 Declarations Conflict of interest The authors declare that they have no conflict of interest. Data Availability No datasets were generated or analyzed during the current study. Author Contribution Tin Van Huynh: Conceptualization; Formal analysis; Investigation; Method- ology; Validation; Visualization; Writing - original draft. Kiet Van Nguyen: Conceptualization; Formal analysis; Investigation; Val- idation; Visualization; Supervision; Writing - review&editing. Ngan Luu-Thuy Nguyen: Conceptualization; Formal analysis; Investiga- tion; Methodology; Validation; Supervision; Writing - review&editing. References [1] Gao, T., Yao, X., Chen, D.: Simcse: Simple contrastive learning of sen- tence embeddings. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 6894–6910 (2021) [2] Van Huynh, T., Van Nguyen, K., Nguyen, N.L.-T.: Vinli: A viet- namese corpus for studies on open-domain natural language inference. In: Proceedings of the 29th International Conference on Computational Linguistics, p. 3858–3872 (2022) [3] Hoa, T.T., Duy, T.Q., Tran, K.Q., Van Nguyen, K.: Vifactcheck: A new benchmark dataset and methods for multi-domain news fact-checking in vietnamese. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, p. 308–316 (2025) [4] Wang, Y., Wu, A., Neubig, G.: English contrastive learning can learn universal cross-lingual sentence embeddings. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 9122–9133 (2022) [5] Chuang, Y.-S., Dangovski, R., Luo, H., Zhang, Y., Chang, S., Soljaˇci ́c, M., Li, S.-W., Yih, S., Kim, Y., Glass, J.: Diffcse: Difference-based contrastive learning for sentence embeddings. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 4207–4218 (2022) [6] Bowman, S., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. In: Proceedings of the 2015 Springer Nature 2021 L A T E X template 40 Tin et al. Conference on Empirical Methods in Natural Language Processing, p. 632–642 (2015) [7] Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., Specia, L.: Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In: Proceedings of the 11th International Workshop on Seman- tic Evaluation (SemEval-2017) (2017). Association for Computational Linguistics [8] Thorne, J., Vlachos, A., Christodoulopoulos, C., Mittal, A.: Fever: a large-scale dataset for fact extraction and verification. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), p. 809–819 (2018) [9] Lai, G., Xie, Q., Liu, H., Yang, Y., Hovy, E.: Race: Large-scale reading comprehension dataset from examinations. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, p. 785–794 (2017) [10] Howard, J., Ruder, S.: Universal language model fine-tuning for text clas- sification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 328–339 (2018) [11] Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceed- ings of the 2019 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 4171–4186 (2019) [12] Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., Stoyanov, V.: Unsupervised cross-lingual representation learning at scale. In: Pro- ceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, p. 8440–8451. Association for Computational Lin- guistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.747. https://aclanthology.org/2020.acl-main.747 [13] Le, H.T., To, L.T., Nguyen, M.T., Van Nguyen, K.: Viwikifc: Fact- checking for vietnamese wikipedia-based textual knowledge source. arXiv preprint arXiv:2405.07615 (2024) [14] Nguyen, D.Q., Nguyen, A.T.: Phobert: Pre-trained language models for vietnamese. In: Findings of the Association for Computational Linguis- tics: EMNLP 2020, p. 1037–1042 (2020) Springer Nature 2021 L A T E X template Tin et al.41 [15] Liu, Y.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 364 (2019) [16] Do, P., Tran, S., Hoang, P., Nguyen, K., Nguyen, N.: Vlue: A new benchmark and multi-task knowledge transfer learning for vietnamese natural language understanding. In: Findings of the Association for Computational Linguistics: NAACL 2024, p. 211–222 (2024) [17] Van Huynh, T., To, H.Q., Van Nguyen, K., Nguyen, N.L.-T.: Error inves- tigation of pre-trained bertology models on vietnamese natural language inference. In: Asian Conference on Intelligent Information and Database Systems, p. 176–188 (2022). Springer [18] Luu, S.T., Hoang, K.T., Pham, T.Q., Van Nguyen, K., Nguyen, N.L.-T.: A multiple choices reading comprehension corpus for vietnamese language education. arXiv preprint arXiv:2303.18162 (2023) [19] Mikolov, T.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 3781 (2013) [20] Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empiri- cal Methods in Natural Language Processing (EMNLP), p. 1532–1543 (2014) [21] Kiros, R., Zhu, Y., Salakhutdinov, R.R., Zemel, R., Urtasun, R., Tor- ralba, A., Fidler, S.: Skip-thought vectors. Advances in neural information processing systems 28 (2015) [22] Conneau, A., Kiela, D., Schwenk, H., Barrault, L., Bordes, A.: Supervised learning of universal sentence representations from natural language infer- ence data. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, p. 670–680 (2017) [23] Hill, F., Cho, K., Korhonen, A.: Learning distributed representations of sentences from unlabelled data. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 1367–1377 (2016) [24] Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. In: Walker, M., Ji, H., Stent, A. (eds.) Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, Volume 1 (Long Papers), p. 2227–2237. Association for Computational Linguistics, New Orleans, Louisiana (2018). https://doi.org/10.18653/v1/N18-1202. https://aclanthology.org/N18-1202/ Springer Nature 2021 L A T E X template 42 Tin et al. [25] Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings us- ing Siamese BERT-networks. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 3982–3992. Asso- ciation for Computational Linguistics, Hong Kong, China (2019). https: //doi.org/10.18653/v1/D19-1410. https://aclanthology.org/D19-1410/ [26] Sanh, V.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019) [27] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21(140), 1–67 (2020) [28] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020) [29] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023) [30] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023) [31] Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., Millican, K., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023) [32] Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al.: Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024) [33] Nguyen, D.V.-T., Van Huynh, T., Van Nguyen, K., Nguyen, N.L.-T.: Transformer-based contextualized language models joint with neural networks for natural language inference in vietnamese. arXiv preprint arXiv:2411.13407 (2024) [34] Nguyen, N.L.-T., Phan, K.T.-K., Huynh, T.V., Nguyen, K.V.: Lmck: pre- trained language models enhanced with contextual knowledge for viet- namese natural language inference. Multimedia Tools and Applications Springer Nature 2021 L A T E X template Tin et al.43 84(15), 15575–15595 (2025) [35] Wang, D., Ding, N., Li, P., Zheng, H.: Cline: Contrastive learning with semantic negative examples for natural language understanding. In: Pro- ceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 2332–2342 (2021) [36] Wang, Z., Wang, P., Huang, L., Sun, X., Wang, H.: Incorporating hier- archy into text encoder: a contrastive learning approach for hierarchical text classification. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 7109–7119 (2022) [37] Kim, T., Yoo, K.M., Lee, S.-g.: Self-guided contrastive learning for bert sentence representations. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 2528–2540 (2021) [38] Li, J., Shang, J., McAuley, J.: Uctopic: Unsupervised contrastive learn- ing for phrase representations and topic mining. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6159–6169 (2022) [39] Das, S.S.S., Katiyar, A., Passonneau, R.J., Zhang, R.: Container: Few- shot named entity recognition via contrastive learning. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6338–6353 (2022) [40] Liu, S., Hu, X., Zhang, C., Wen, L., Philip, S.Y., et al.: Hiure: Hierar- chical exemplar contrastive learning for unsupervised relation extraction. In: Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, p. 5970–5980 (2022) [41] Pan, X., Wang, M., Wu, L., Li, L.: Contrastive learning for many-to- many multilingual neural machine translation. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers) (2021). Association for Computational Linguistics [42] Li, Y., Liu, F., Collier, N., Korhonen, A., Vuli ́c, I.: Improving word translation via two-stage contrastive learning. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 4353–4374 (2022) Springer Nature 2021 L A T E X template 44 Tin et al. [43] You, C., Chen, N., Zou, Y.: Self-supervised contrastive cross-modality representation learning for spoken question answering. In: Findings of the Association for Computational Linguistics: EMNLP 2021, p. 28–39 (2021) [44] Ji, Y., Chen, L.-Y., Dou, C., Ma, B., Li, X.: To answer or not to an- swer? improving machine reading comprehension model with span-based contrastive learning. In: Findings of the Association for Computational Linguistics: NAACL 2022, p. 1292–1300 (2022) [45] Cao, S., Wang, L.: Cliff: Contrastive learning for improving faithfulness and factuality in abstractive summarization. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, p. 6633–6649 (2021) [46] Wu, H., Ma, T., Wu, L., Manyumwa, T., Ji, S.: Unsupervised reference- free summary quality evaluation via contrastive learning. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 3612–3621 (2020) [47] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple frame- work for contrastive learning of visual representations. In: International Conference on Machine Learning, p. 1597–1607 (2020). PMLR [48] Tan, W., Koehn, P.: Bitext mining for low-resource languages via contrastive learning. arXiv e-prints, 2208 (2022) [49] Maisha, J., Swarnali, F.H., Iftikar, M.S.I., Ahammed, S.B., Shah, F.M.: A study of contrastive learning methods for bengali social analysis. In: 2024 6th International Conference on Electrical Engineering and Information & Communication Technology (ICEEICT), p. 272–277 (2024). IEEE [50] Liu, X., Su, Y., Nier, W., Ji, Y., Qing Dao Er Ji, R., Lu, M.: An ap- proach to mongolian neural machine translation based on rwkv language model and contrastive learning. In: International Conference on Neural Information Processing, p. 327–340 (2023). Springer [51] Hu, X., Chen, X., Qi, P., Kong, D., Liu, K., Wang, W.Y., Huang, Z.: Language agnostic multilingual information retrieval with contrastive learning. In: Findings of the Association for Computational Linguistics: ACL 2023, p. 9133–9146 (2023) [52] Huang, Y., Liang, Y., Wu, Z., Zhu, E., Yu, Z.: Cross-lingual sen- tence embedding for low-resource chinese-vietnamese based on contrastive learning. ACM Transactions on Asian and Low-Resource Language Information Processing 22(6), 1–18 (2023) Springer Nature 2021 L A T E X template Tin et al.45 [53] Dang, N.M., Nguyen, T.S.: Contrastive learning for vietnamese abstrac- tive summarization. In: 2023 15th International Conference on Knowledge and Systems Engineering (KSE), p. 1–6 (2023). IEEE [54] Williams, A., Nangia, N., Bowman, S.: A broad-coverage challenge corpus for sentence understanding through inference. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long Papers), p. 1112–1122 (2018) [55] Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., Kiela, D.: Adversarial nli: A new benchmark for natural language understand- ing. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 4885–4901 (2020) [56] Hu, H., Richardson, K., Xu, L., Li, L., K ̈ubler, S., Moss, L.S.: Ocnli: Orig- inal chinese natural language inference. In: Findings of the Association for Computational Linguistics: EMNLP 2020, p. 3512–3526 (2020) [57] Chen, B., Tang, H., Bu, J., Zhang, K., Wang, J., Wang, Q., Zheng, H.-T., Wu, W., Yu, L.: Clower: A pre-trained language model with contrastive learning over word and character representations. In: Proceedings of the 29th International Conference on Computational Linguistics, p. 3098– 3108 (2022) [58] Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S., Schwenk, H., Stoyanov, V.: Xnli: Evaluating cross-lingual sentence representations. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 2475–2485 (2018) [59] Bui, T.B., Nguyen, L.T.-T., Van Huynh, T.: Vietx-nli: A cross-lingual nat- ural language inference dataset with vietnamese as the source language. In: 2025 International Conference on Multimedia Analysis and Pattern Recognition (MAPR), p. 1–6 (2025). IEEE [60] Zhao, K., Wu, Q., Cai, X.-Q., Tsuruoka, Y.: Leveraging multi-lingual positive instances in contrastive learning to improve sentence embedding. In: Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 976–991 (2024) [61] Shuaibo, W., Hui, D., Hui, H., Siyu, L., Kazushige, O., Yufeng, C., Jinan, X.: Supervised contrastive learning for cross-lingual transfer learning. In: Proceedings of the 21st Chinese National Conference on Computational Linguistics, p. 884–895 (2022) Springer Nature 2021 L A T E X template 46 Tin et al. [62] Van Huynh, T., Van Nguyen, K., Luu-Thuy Nguyen, N.: A new bench- mark dataset and mixture-of-experts language models for adversarial nat- ural language inference in vietnamese. Expert Systems with Applications 306, 130109 (2026). https://doi.org/10.1016/j.eswa.2025.130109 [63] Nguyen, C.T., Nguyen, D.T.: Building a vietnamese dataset for natural language inference models. SN Computer Science 3(5), 395 (2022) [64] Anh, H.T., Huyen, N.T.M., Lien, N., et al.: Vlsp 2021-vnnli challenge: Vietnamese and english-vietnamese textual entailment. VNU Journal of Science: Computer Science and Communication Engineering 38(2) (2022) [65] Nguyen, L.T., Van Nguyen, K., Nguyen, N.L.-T.: Constructive and toxic speech detection for open-domain social media comments in vietnamese. In: Advances and Trends in Artificial Intelligence. Artificial Intelligence Practices: 34th International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2021, Kuala Lumpur, Malaysia, July 26–29, 2021, Proceedings, Part I 34, p. 572–583 (2021). Springer [66] Manna, S., Chattopadhyay, S., Dey, R., Pal, U., Bhattacharya, S.: Dynam- ically scaled temperature in self-supervised contrastive learning. IEEE Transactions on Artificial Intelligence 6(6), 1502–1512 (2025) [67] Tran, N.L., Le, D.M., Nguyen, D.Q.: BARTpho: Pre-trained Sequence- to-Sequence Models for Vietnamese. In: Proceedings of the 23rd Annual Conference of the International Speech Communication Association (2022) [68] Phan, L., Tran, H., Nguyen, H., Trinh, T.H.: Vit5: Pretrained text-to- text transformer for vietnamese language generation. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, p. 136–142 (2022) [69] Lam, Q.H., Le, Q.D., Nguyen, V.K., Nguyen, N.L.-T.: Uit-viic: A dataset for the first evaluation on vietnamese image captioning. In: International Conference on Computational Collective Intelligence, p. 730–742 (2020). Springer A Analysis of Model Stability and Convergence To explicitly address potential overfitting risks and provide empirical support for our experimental rigor, Figure 9, presents the training and validation loss curves across all evaluated tasks. While the training loss steadily decreases Springer Nature 2021 L A T E X template Tin et al.47 across all datasets, the validation trajectories reveal distinct convergence patterns based on task complexity and data scale. For natural language inference and fact-checking (ViNLI, ViWikiFC, and ViFactCheck; Figure 9a, 9b, and 9c), the validation loss exhibits a standard and highly stable convergence trend. It reaches an optimal minimum within the early epochs before plateauing or showing a very slight upward drift. This indicates a robust learning process where the model captures seman- tic relationships and generalizes well to unseen data before any significant memorization occurs. For constructive speech detection (UIT-ViCTSD; Figure 9d), the train- ing loss demonstrates a smooth and consistent downward trend throughout the training process, indicating effective optimization of the supervised con- trastive objective. In contrast, the validation loss initially remains stable but begins to increase gradually in later epochs. This divergence suggests a ten- dency toward overfitting when training continues excessively. However, the best-performing model checkpoint is selected based on validation performance prior to this divergence, ensuring that the final reported results correspond to the most generalizable model state. This behavior reflects a common pattern in moderately sized classification datasets, where semantic boundary learning stabilizes early while prolonged training may lead to marginal memorization effects. For the machine reading comprehension task (ViMMRC2.0; Figure 9e), which has a relatively smaller training size and higher reasoning complex- ity, the validation loss remains remarkably stable across epochs with only minor fluctuations. Notably, no sharp divergence between training and val- idation curves is observed. Although the training loss continues to decrease steadily, the validation loss maintains a nearly flat trajectory, indicating that the model does not excessively overfit to the training distribution. This em- pirical evidence contradicts the concern of severe overfitting on small datasets and demonstrates that the supervised contrastive framework maintains stable generalization behavior even under limited data conditions. Springer Nature 2021 L A T E X template 48 Tin et al. 01234567 Epochs 1 2 3 4 Loss Training and Validation Loss Curves Training Loss Validation Loss (a) ViNLI 01234567 Epochs 0 1 2 3 4 Loss Training and Validation Loss Curves Training Loss Validation Loss (b) ViWikiFC 1234567 Epochs 1 2 3 4 Loss Training and Validation Loss Curves Training Loss Validation Loss (c) ViFactCheck 01234567 Epochs 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Loss Training and Validation Loss Curves Training Loss Validation Loss (d) UIT-ViCTSD 01234567 Epochs 0.5 1.0 1.5 2.0 2.5 Loss Training and Validation Loss Curves Training Loss Validation Loss (e) ViMMRC2.0 Figure 9: Training-validation loss curves of the ViCLSR model on down- stream tasks.