Paper deep dive
Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment
Barah Fazili, Koustava Goswami
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/20/2026, 12:39:08 PM
Summary
This paper proposes a method to enhance multilingual embeddings by training standard pretrained models (XLM-Roberta, mBERT, mE5) with a multi-way parallel corpus using contrastive learning. By treating translations in multiple languages as positive pairs for every language anchor, the approach achieves significant improvements in cross-lingual alignment, outperforming English-centric bilingual alignment on bitext mining, semantic textual similarity, and classification tasks across the MTEB benchmark.
Entities (17)
Relation Signals (16)
Multi-way Parallel Corpus → containslanguages → English
confidence 98% · translations of English text
Contrastive Learning → achieves → Cross-lingual Alignment
confidence 95% · achieve strong cross-lingual alignment through contrastive learning
Multi-way Parallel Corpus → containslanguages → Japanese
confidence 95% · pool of six target languages... Japanese
Multi-way Parallel Corpus → containslanguages → Hindi
confidence 95% · pool of six target languages... Hindi
Multi-way Parallel Corpus → containslanguages → French
confidence 95% · pool of six target languages... French
Multi-way Parallel Corpus → containslanguages → German
confidence 95% · pool of six target languages... German
Multi-way Parallel Corpus → containslanguages → Spanish
confidence 95% · pool of six target languages... Spanish
Multi-way Parallel Corpus → containslanguages → Chinese
confidence 95% · pool of six target languages... Chinese
Multi-way Parallel Corpus → improvesperformanceon →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multilingual pretraining typically lacks explicit alignment signals, leading to suboptimal cross-lingual alignment in the representation space. In this work, we show that training standard pretrained models for cross-lingual alignment with a multi-way parallel corpus in a diverse pool of languages can substantially improve multilingual and cross-lingual representations for NLU tasks. We construct a multi-way parallel dataset using translations of English text from an off-the-shelf NMT model for a pool of six target languages and achieve strong cross-lingual alignment through contrastive learning. This leads to substantial performance gains across both seen and unseen languages for multiple tasks from the MTEB benchmark evaluated for XLM-Roberta and multilingual BERT base models. Using a multi-way parallel corpus for contrastive training yields substantial gains on bitext mining (21.3%), semantic similarity (5.3%), and classification (28.4%) compared to English-centric (En-X) bilingually parallel data, where X is sampled from a pool of multiple target languages. Furthermore, finetuning mE5 model on a small dataset with multi-way parallelism significantly improves bitext mining compared to one without, underscoring the importance of multi-way cross-lingual supervision even for models already pretrained for high-quality sentence embeddings.
Tags
Links
- Source: https://arxiv.org/abs/2602.21543v1
- Canonical: https://arxiv.org/abs/2602.21543v1
Trouble viewing inline? Open PDF directly →
Full Text
37,925 characters extracted from source content.
Expand or collapse full text
Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment Barah Fazili 1* * , Koustava Goswami 2 , 1 IIT Bombay, Mumbai, India 2 Adobe Research, Bangalore, India Abstract Multilingual pretraining typically lacks explicit alignment signals, leading to suboptimal cross- lingual alignment in the representation space. In this work, we show that training standard pre- trained models for cross-lingual alignment with a multi-way parallel corpus in a diverse pool of languages can substantially improve multilin- gual and cross-lingual representations for NLU tasks. We construct a multi-way parallel dataset using translations of English text from an off- the-shelf NMT model for a pool of six tar- get languages and achieve strong cross-lingual alignment through contrastive learning. This leads to substantial performance gains across both seen and unseen languages for multiple tasks from the MTEB benchmark–evaluated for XLM-Roberta and multilingual BERT base models. Using a multi-way parallel corpus for contrastive training yields substantial gains on bitext mining (+21.3%), semantic similarity (+5.3%), and classification (+28.4%) compared to English-centric (En–X) bilingually parallel data, where X is sampled from a pool of multi- ple target languages. Furthermore, finetuning mE5 model on a small dataset with multi-way parallelism significantly improves bitext min- ing compared to one without, underscoring the importance of multi-way cross-lingual super- vision even for models already pretrained for high-quality sentence embeddings. 1 Introduction Pretrained multilingual language models (Conneau et al., 2020) have demonstrated strong transfer ca- pabilities both across tasks in zero-shot settings (e.g bitext mining) and tasks involving finetuning on source data (primarily in English) (e.g. classi- fication). Among the various factors studied be- hind this emergent behavior—such as shared to- kens across some languages—cross-lingual align- ment has been shown to be strongly correlated * Research conducted during an internship at Adobe Re- search in 2024. with downstream task performance in cross-lingual tasks (Deshpande et al., 2022; Tang et al., 2022). Cross-lingual alignment implies that "similar meanings across languages have more similar repre- sentations than dissimilar meanings do" (Hämmerl et al., 2024). In the context of natural language understanding tasks, this principle suggests that if a model can effectively map semantically sim- ilar expressions from different languages closer in the representation space, then supervision in one language can generalize to others, enabling cross-lingual transfer. There have been numer- ous attempts at achieving better alignment, (see Section 5) by leveraging parallel text or bilingual dictionaries, but mostly the focus has been on bilin- gually parallel text, even when more languages are included in the corpus. We aim to harness the richer signal of multi-way parallelism by intro- ducing multi-way cross-lingual alignment, using translations generated by an off-the-shelf machine translation model. We summarize our contributions below: 1.We show that using a multi-way parallel cor- pus spanning a pool of target languages can be more effective for cross-lingual alignment training than relying solely on bilingually par- allel text. 2.We treat each language as an anchor during contrastive learning, ensuring that every lan- guage actively pulls its translations closer in the embedding space. This approach is a key factor in the success of our multi-way cross- lingual alignment training. 3.We show that training incorporating more lan- guages beside a target language for alignment (wrt English) can lead to better downstream performance across multiple tasks with Hindi evaluated as the target language. arXiv:2602.21543v1 [cs.CL] 25 Feb 2026 4.We also observe that post alignment, our model representations lead to significant gains even for languages unseen during alignment training. This suggests that with our proposed multi-way parallel synthetic corpus, cross- lingual alignment benefits can generalize be- yond the seen languages. 5.The evaluation is done over multiple tasks from MTEB(Muennighoff et al., 2023) benchmark for sentence embeddings. Using XLM-R base as the model, we show that lever- aging multi-way parallelism in the alignment dataset yields substantial gains across diverse tasks: 21.3% improvement in bitext mining, 5.3% in semantic textual similarity, and 28.4% in classification. 2 Methodology In the context of images, Khosla et al. (2021) pro- posed the supervised contrastive loss, which ex- tends the self-supervised approach by incorporating class information. Instead of relying solely on aug- mented versions of an anchor image, this approach also considers other instances from the same class within a batch as positive examples, while treating all remaining instances as negatives. We adapt this idea of allowing multiple posi- tives during contrastive learning to the domain of multilingual text representation learning. Consider a multi-way parallel text corpusDwithkfields, with rowD i =D i,j ∀j ∈1,...,kcontaining source text in English and its translations in multi- ple languages over the followingk− 1fields. For text in each column (in a distinct language across the row) as an anchor, the remainingk−1columns serve as positives. Here, the notion of class is based on semantics, with the translation acting as the augmentation. We created this corpus of multi-way parallel text after sampling English text from OPUS which is then machine translated. The created corpus con- tains four columns and approximately 75K rows (75,822 in train+validation set). The first column consists of original sentences from OPUS in En- glish, while the other three columns contain trans- lations into three of six target languages—Chinese, Japanese, French, German, Hindi, and Span- ish—sampled uniformly at random (see Figure 1 ). Equation 1 shows the formulation of loss with P(i)indicating the positives of a sentenceiwith Chinese : 中文 French : Français German : Deutsch Hindi : िहन्दी Japanese : 日本語 Spanish : Español English:English Figure 1: Each sentence (shown as a black cell) in English is translated into three target languages chosen randomly from the pool. embeddingz i andA(i)≡ I \ (i∪ P(i))the set of remaining sentences in the batch excludingi and the translations ofiandτis the temperature 1 . Instead of using only the English sentence as the anchor in contrastive loss, we allow all languages to serve as the anchor for each row(semantic in- stance) in our alignment dataset. This approach improves convergence across language subspaces and leads to more cohesive multilingual represen- tations L sup = X i∈I L i = X i∈I −1 |P(i)| X p∈P(i) log exp(z i · z p /τ) P a∈A(i) exp(z i · z a /τ) (1) 3 Experiments 3.1 Dataset Creation 3.1.1 Sampling English Sentences To construct a diverse set of English source sen- tences representative of both formal and conversa- tional domains, we combined data from Wikipedia and OpenSubtitles. Specifically, we obtained for- mal text from Wikipedia by downloading the En- glish corpus from the OPUS-Wikipedia dataset 2 , originally containing over 441M sentences. For conversational text, we used OpenSubtitles, ac- quiring the English monolingual data from OPUS- OpenSubtitles v2018 3 4 , which contained over 10M 1 Note that we enforce strong alignment (Hämmerl et al., 2024), ensuring that a sentence’s nearest neighbors in the embedding space are its translations, while (semantically) dissimilar sentences in the same and different languages are spaced farther apart. 2 https://object.pouta.csc.fi/OPUS-Wikipedia/ v1.0/mono/en.txt.gz 3 https://opus.nlpl.eu/OpenSubtitles/de&en/ v2018/OpenSubtitles 4 https://object.pouta.csc.fi/ OPUS-OpenSubtitles/v2018/mono/en.txt.gz sentences. To ensure a balanced representation of both styles, we randomly sampled 38,000 sentences from each dataset (with a minimum length of 10 words), using a fixed random seed (42). The sam- pled subsets were then concatenated into a single corpus. We allocated 90% of the sentences for training and the remaining 10% for validation. 3.1.2 Translation The selected English sentences were then trans- lated into a diverse set of six target lan- guages—French (fra_Latn), German (deu_Latn), Spanish (spa_Latn), Japanese (jpn_Jpan), Chi- nese (zho_Hans), and Hindi (hin_Deva)—using the NLLB-200 3.3B translation model 5 . The text for each language was maintained in its original script. Our setup does not rely on human-curated multi-way corpora, which are rare. Instead, our approach can be easily replicated in any setting where reasonable English-to-target language NMT models exist—a condition increasingly satisfied across many languages due to advances in neural MT. Moreover, while the data is English-rooted, our alignment strategy does not rely on English as a fixed pivot. 3.2 Model Training We train XLM-Roberta and mBERT-base models 6 using a contrastive objective over a multi-way par- allel corpus. A regularization term (see Equation 2) is added to the contrastive loss (Equation 1), which penalizes the squared Euclidean distance between the model embeddings and the corresponding pre- trained embeddings (z orig,i ), normalized over all sentences in the batch (N = P i∈I 1). The strength of this regularization is controlled by a scalar hy- perparameter λ. As shown by Saattrup Nielsen et al. (2025), en- coder models can outperform decoder models on NLU tasks despite having significantly fewer pa- rameters. Our work is situated in this context, offering a lightweight method for improving en- coder models’ cross-lingual transfer performance in real-world applications where inference cost, latency, and model size matter (e.g., edge deploy- ment, multilingual assistants in low-connectivity regions). Additional training details are provided in Appendix A. 5 https://huggingface.co/facebook/nllb-200-3. 3B 6 1.https://huggingface.co/FacebookAI/ xlm-roberta-base2.https://huggingface.co/ google-bert/bert-base-multilingual-cased R = P i ∥z orig,i − z i ∥ 2 2 N (2) L =L sup + λR(3) 3.3 Evaluation Results . To assess the quality of learned embeddings for multilingual tasks, we evaluate over four different tasks from the Massive Text Embedding Bench- mark (MTEB) (Muennighoff et al., 2023). We report results for languages included in the dataset used for alignment training, namely English (en), French (fr), German (de), Spanish (es), Hindi (hi), Chinese (zh), and Japanese (jp). While bitext min- ing and STS are evaluated in a zero-shot manner, classification and clustering involve additional task- specific training using MTEB’s default settings, with scores presented on the respective test sets unless stated otherwise. 7 3.3.1 Bitext mining The input consists of two sets of sentences, each from a different language. The goal is to iden- tify the best match in the second set for each sen- tence in the first set, typically corresponding to translations. A given model is used to generate sentence embeddings, and the closest pairs are de- termined based on cosine similarity. Prior work (Hu et al., 2020; Feng et al., 2022) highlights the weak performance of pretrained multilingual mod- els on bitext retrieval tasks due to the absence of a sentence-level objective. As shown in Table 1, our aligned models (mBERT-aligned and XLM- R-aligned) significantly outperform their standard pretrained counterparts (mBERT-base and XLM-R base) on both BUCC and Tatoeba. With cross- lingual alignment using just 450K sentence pairs, F1 scores improve dramatically (e.g. from 21.6 to 95 for Chinese-English in BUCC for XLMR) across test sets spanning subsets of the pool of tar- get languages. 3.3.2 Semantic textual similarity (STS) The goal is to assess the similarity of a given sentence pair, where labels represent continuous 7 Datasets can be found here:https://huggingface. co/mteband evaluation scripts on the MTEB repo:https: //github.com/embeddings-benchmark/mteb/tree/main. Seehttps://github.com/embeddings-benchmark/mteb/ blob/main/docs/tasks.mdfor language abbreviation details. scores, with higher values indicating greater simi- larity. The provided model generates embeddings for the sentences, and their similarity is measured using cosine values. These computed values are evaluated against ground truth similarity scores using Spearman correlation. Table 2 shows the scores for STS17 and STS22.v2 from the MTEB benchmark. The evaluation sets include both same language(e.g. en-en, es-es) and different languages in the two sets (e.g es-en) in Table 2. We again see dramatic improvements for this task with the Spear- man correlation coefficient going from as low as -1.2 to 52.7 for en-de after alignment of pretrained mBERT-base. Both monolingual and cross-lingual test sets for both datasets get vast improvement across the evaluated languages. 3.3.3 Classification Unlike bitext mining and STS, classification task requires training a classifier head with the sentence embeddings from the model fed as input. A lo- gistic regression classifier, trained on the embed- dings from the training set with a maximum of 100 iterations, is evaluated on the test set. Amazon Counterfactual consists of customer review sen- tences labeled for counterfactual detection (CFD) as a binary classification task. AmazonReviews categorizes reviews into five classes corresponding to the available rating options. MassiveIntent and MassiveScenario contain 60 and 51 labels, respec- tively, while MTOP Domain has 11 classes, and MTOP Intent spans a variable range of 0 to 112 labels. Table 3 shows the accuracy across six dif- ferent datasets for languages in our alignment pool where we also indicate the steep percentage gain of finetuned XLMR-aligned model with respect to its counterpart without alignment (finetuned XLMR- base). 3.3.4 Clustering A mini-batch k-means model with a batch size of 32 and k set to the number of unique labels is trained on the embedded text representations (Pedregosa et al., 2011) . The model’s performance is evalu- ated using V-measure (Rosenberg and Hirschberg, 2007). Table 4 shows the scores for five datasets. With the exception of model initialized to XLMR- aligned performing worse than the pretrained ver- sion for SpanishNewsP2P, both aligned models gen- erally perform significantly better across tasks and languages. 4 Analysis 4.1 Multi-way Parallelism In order to check whether the choice of alignment training over the multi-way parallel corpus is any better than English-centric (En–X) bilingually par- allel data, where X is sampled from a pool of mul- tiple target languages, we train XLMR-base model using the following two alignment corpora: 1. The model (par-model-A) is trained on a sub- set of N/6 of the rows in our original dataset D. 2. Another model (par-model-B) is trained on a bilingually parallel slice ofDwith all N rows but with randomly chosen second column (lan- guage) beside the first column in English for alignment. To ensure a fair comparison, we retain only N/6 rows for the multi-way parallel setup so that the total number of parallel sentence pairs remains the same across both settings. As shown in Tables 1, 2 and 3, the multi-way aligned model (par-model- A) consistently outperforms its bilingually paral- lel counterpart (par-model-B) across all three eval- uated downstream tasks for most languages and language-pairs 8 . 4.2 Bilingual vs multilingual To evaluate whether aligning additional languages beside the target improves performance, we de- signed four training datasets for Hindi evaluation: 1.Bilingual alignment (eh): A corpus containing only English and Hindi (two columns) with N rows. 2.European language augmentation (eh-euro): English and Hindi as fixed columns, with the remaining two columns randomly sam- pled from French, German, and Spanish (four columns, N rows). 3. Asian language augmentation (eh-asian): En- glish and Hindi as fixed columns, with the re- maining two columns sampled from Chinese and Japanese (four columns, N rows). 8 For classification, par-model-A significantly outperforms par-model-B across 31 dataset-language pairs (paired t-test: t = 10.81, p < 1e-10; Wilcoxon: W = 0.0, p < 1e-6), confirming the statistical significance of the improvement. Model BUCCTatoeba fr-ende-enzh-ende-enjp-enes-enfr-enhi-en mBERT-base10.811.34.99.46.911.514.62.3 mBERT-aligned91.49591.791.876.284.583.273.6 XLMR-base18.226.921.63213.523.822.49.9 XLMR-aligned92.995.99595.786.392.589.689.4 XLMR-en-anchor23.237.140.147.829.448.937.828.8 XLMR-en-ablate87.993.493.39072.981.177.968.1 par-model-A94.865.796.89687.892.791.290 par-model-B80.783.486.883.663.872.671.458.1 Table 1: Results for bitext mining Model STS17STS22.v2 en- en fr-enen- de es-eses- en enfrdeeszhde- en de- fr es- en zh- en mBERT-base24.92.9 −1.229.26.83.210 −24.51 −1.7 −4.74.72.53.9 mBERT-aligned65.357.152.769.543.654.967.93760.265.151.343.568.662.3 XLMR-base19.92.794.820.54.450.862.719.750.954.942.838.940.345.2 XLMR-aligned63.755.251.970.647.661.775.241.766.868.456.949.671.366.6 XLMR-en-anchor35.29.421.632.53158.560.930.260.667.257.348.863.460.7 XLMR-en-ablate70.155.25774.858.757.86733.764.668.351.151.969.160.9 par-model-A66.958.458.773.952.558.27343.665.767.752.854.27063.4 par-model-B65.647.848.768.845.458.468.641.46668.554.75270.964.9 Table 2: Results for STS (semantic textual similarity) 4.Full multilingual alignment (eh-all): English and Hindi as fixed columns, with the other two columns sampled from all remaining lan- guages (four columns, N rows). As shown in Tables 5 and 6, aligning across mul- tiple languages not only doesn’t hurt but can also enhance performance on a specific target language. For bitext mining, adding only European (eh-euro) or only Asian (eh-asian) languages does not im- prove over bilingual alignment (eh), but combining both groups (eh-all) leads to a performance boost. For other tasks, including classification, cluster- ing, and STS, all multilingual variants (eh-euro, eh- asian, and eh-all) outperform the bilingual baseline, with eh-all consistently achieving the best results. 4.3 Multiple languages vs English only as anchor A key design choice in our alignment training was allowing all languages in the dataset to serve as anchors, with their respective translations acting as positive pairs. To assess the impact of this decision, we trained an alternative model where only English was used as the anchor. The results show a sig- nificant drop in performance when restricting the anchor to English alone. As illustrated in Tables 1, 2, 3, and 4, the scores for XLMR-en-anchor decline considerably compared to XLMR-aligned, where all languages (present in the corresponding row) were included as anchors. This shows the advan- tage of leveraging multilingual anchors for better alignment and transferability across languages. 4.4 Effect of removing mandatory English To evaluate whether it is essential to retain English in every semantic instance, we created an alterna- tive dataset where all four languages per instance were randomly sampled from a pool of seven lan- guages (including English), rather than ensuring the presence of English in each row. We trained XLMR-base on this modified dataset, naming the model XLMR-en-ablate. While XLMR-en-ablate performs worse than XLMR-align (our default set- ting), it still significantly outperforms XLMR-en- anchor, where only English was used as the anchor. This suggests that allowing all languages to act as anchors during contrastive training plays a far more critical role in alignment than strictly maintaining English in every instance. 4.5 Effect of alignment on other languages Beyond achieving substantial gains on test sets for languages included in our alignment training, we also assess its impact on languages outside our se- DatasetLangmBERT- base mBERT- aligned XLMR- base XLMR- aligned (% gain) XLMR- en- anchor XLMR- en- ablate par- model- A par- model- B AmazonCounterfactual en60.666.862.572.0 (15.2)6061.672.558.4 de59.465.255.470.5 (27.3)52.360.969.761.2 jp57.1595972.6(23.05)5658.872.956.2 AmazonReviews en27.532.326.136.8 (40.99)2628.636.928.3 de24.732.328.439.5 (39.08)27.531.639.630.9 es25.832.425.138.2 (52.19)25.731.638.430.3 fr25.431.625.638.5 (50.39)25.231.838.330 jp26.729.826.936.9 (37.17)27.732.836.729.8 zh25.830.729.436.5 (24.15)30.33336.829.9 MassiveIntent de28.341.319.353.9 (179)21.237.554.638.7 es29.946.722.856.4 (147)22.841.156.442.2 en37.851.729.560.7 (105)29.445.461.546.2 fr31.74720.255.9 (176)21.540.256.342 jp32.949.524.260 (147)19.744.960.246.4 hi24.538.623.453.8 (129)17.741.154.540.9 MassiveScenario en40.157.439.866.8 (67.84)39.959.867.662 hi26.842.13158.9 (90)26.350.759.450.7 de31.245.928.760.9 (112)31.952.962.352.3 fr35.650.927.660.6 (119)30.654.361.656.1 jp34.853.133.565.3 (94.9)27.961.366.260.2 es32.45232.362.5 (93.5)32.554.66357.3 MTOPDomain en5572.139.680.3 (102)35.379.481.374.7 de4969.840.280.5 (100)28.877.781.572.8 es5170.837.278.1 (109)33.376.179.471.5 fr49.466.538.774.9 (93.5)34.569.974.865.4 hi44.859.634.476.4 (122)31.772.376.668.4 MTOPIntent en40.151.719.566 (238)8.837.46538 de35.841.320.565.3 (218)8.135.564.438 es35.146.621.562 (188)8.539.261.440 fr32.14715.558 (274)9.637.256.934 hi28.438.613.462.1 (363)6.637.86139.3 Table 3: Results for classification. Model IndicReviewsP2PMasakhaNewsP2PMasakhaNewsS2SMewsC16JaSpanishNewsP2P hienfrenfrjpes mBERT-base23.20.625.21.5226.84.5 mBERT-aligned35.247.156.732.642.244.926.2 XLMR-base321622.71.22122.838.7 XLMR-aligned37.852.257.234.727.143.426 XLMR-en-anchor35.229.848120.934.947.5 XLMR-en-ablate42.252.663.838.6534842.4 Table 4: Results for clustering ModelTatoeba (hi-en)IndicReviewsP2P (hi) multi-eh69.241.3 multi-euro67.943.1 multi-eh-asian67.143 multi-eh-all71.443.8 Table 5: Results for bitext mining and clustering after bilingual vs multilingual alignment lected pool of seven. We evaluate the STS task using XLM-R base and aligned models across all available test languages. Notably, we observe sig- DatasetLang multi- eh multi- eh- asian multi- eh- euro multi- eh- all MassiveIntenthi37.33939.139.9 MassiveScenariohi46.850.448.850.7 MTOPDomainhi70.871.871.674.3 MTOPIntenthi33.437.73839.5 Table 6: Results for classification after bilingual vs multilingual alignment nificant and consistent improvements even in lan- guages like Arabic, Russian, and Turkish (see Ta- ble 7). We also evaluate on three different clus- tering tasks (see Table 8) over twelve Indian lan- guages in IndicReviewsClusteringP2P and fourteen African languages in each of MasakhaNewsCluster- ingP2P and MasakhaNewsClusteringS2S. With the exception of two cases (bd and lin in the first and third task respectively), we again see substantial im- provements across the board. This highlights how multi-way cross-lingual alignment training over a small subset of diverse languages can potentially generalize to unseen languages. 4.6 On retrained sentence embedding models:bitext mining We also experiment with mE5 (Wang et al., 2024) as the model. Despite mE5’s strong prior pretrain- ing (through large-scale weakly supervised con- trastive pretraining on 1B multilingual pairs, fol- lowed by supervised fine-tuning with hard nega- tives and knowledge distillation), we find that fine- tuning it with our method over a small multi-way parallel dataset improves performance on bitext mining tasks. Our multi-way aligned mE5 vari- ant (mE5-par-model-A) outperforms both the orig- inal model (mE5-base) on this task and alignment over English-centric bilingually parallel dataset (mE5-par-model-B) (see Table 9), demonstrating that even for well-aligned models, multi-way cross- lingual supervision adds value 9 . Given that our fine-tuning uses only∼ 10Krows from our dataset with supervised contrastive loss, this result under- scores the strength of the multi-way signal and its potential as a lightweight, complementary strategy to large-scale training. These gains demonstrate the strength of our alignment strategy, and further improvements are likely with larger corpora and tailored training methods. 4.7 Visualization To visualize the impact of alignment on cross- lingual similarity, we plotted histograms of cosine similarity scores for sentence pairs from the align- ment development set before and after alignment. Specifically, for a given language pair (A, B), we randomly sampled 100 sentences in language A and paired them with their corresponding transla- tions in language B to form 100 matched pairs. To create a contrastive set, we then shuffled the trans- 9 A paired t-test for mE5-par-model-A vs mE5-par-model- B scores yields a t-statistic of 4.44 with a p-value of 0.003, and Wilcoxon signed-rank test gives W = 0.0 with p-value = 0.0078 indicating a statistically significant difference. lations, forming 100 random pairs where sentences were no longer aligned. We computed cosine similarity scores for both sets using XLMR-base before and after alignment and plotted histograms for different language pairs from our pool, as shown in Figure 2, and more in appendix B . The results clearly show that post- alignment, cosine similarity scores for matched pairs become higher and well separated from those of random pairs, demonstrating improved cross- lingual alignment. Figure 2: Histograms of cosine similarity scores before and after alignment for German and Japanese sentences 5 Related Work Early work on explicit cross-lingual alignment training was introduced by Cao et al. (2020), which proposed the task of "contextual word retrieval" to optimize word correspondences across a bilingual parallel corpus using contextual word embeddings. Wu and Dredze (2020) extended this to multi-way parallel texts, but found limited gains when evalu- ated across diverse models and datasets. Several approaches improve cross-lingual align- ment at the word level.DICT-MLM (Chaud- hary et al., 2020) applies MLM while translating masked tokens using a bilingual dictionary. Align- MLM (Tang et al., 2022) enhances word embed- ding alignment by encouraging similarity between dictionary pairs via an auxiliary loss. Other meth- ods include dual-encoder masking (Li et al., 2023) and pre-alignment via code-switching (Li et al., 2024). Unlike these, our method retrofits align- ment into a pretrained model using a small curated corpus, avoiding changes to the original (and ex- pensive) pretraining process. Sentence-level cross-lingual alignment has also been widely studied. Artetxe and Schwenk (2019) trained a single BiLSTM encoder with a shared BPE vocabulary and decoder. Feng et al. (2022) trained language-agnostic embeddings using a mix of MLM, TLM, and translation ranking with mar- gin loss.Sentence-BERT (SBERT) (Reimers Model STS17STS22.v2 en- ar ar- ar en- tr it-ennl- en ko- ko ares-itruittrfr-plplde- pl pl- en XLMR-base8.83.93.22.2432.248.342.146.153.830.416.927.98.327.6 XLMR-aligned48.34652.653.659.156.960.569.657.769.862.339.44150.867.3 Table 7: Semantic textual similarity scores for languages unseen (during alignment training) DatasetLangXLMR- base XLMR- aligned IndicReviewsClusteringp2p as21.830.4 bd21.120.4 bn29.936.9 gu27.236.3 kn28.337.3 ml2537.2 mr24.136.6 or23.333.2 pa27.134.6 ta26.938.1 te28.739.3 ur30.336.2 MasakhaNewsClusteringp2p amh43.160.4 hau16.961.7 ibo23.340.8 lin46.264.1 lug42.250.9 orm21.530.8 pcm38.566.1 run46.453.4 sna43.445.3 som29.132.2 swa21.831.8 tir42.160.8 xho20.635.9 yor21.133.1 MasakhaNewsClusterings2s amh40.550.3 hau5024.2 ibo22.430.8 lin45.943.1 lug40.542.3 orm20.420.7 pcm24.152 run41.949.5 sna40.441.7 som22.127 swa4.69 tir41.447.8 xho23.324.9 yor21.232.8 Table 8: Clustering scores for out-of-pool languages lang pairmE5- base mE5- par-A mE5- par-B BUCC fr-en96.797.497 de-en9999.399 zh-en97.898.498 Tatoeba de-en97.998.597.8 jp-en9190.989.8 es-en97.197.497 fr-en92.793.192.4 hi-en95.795.594 Table 9: Results for bitext mining for multilingual E5 model and Gurevych, 2019) and its multilingual ex- tension (Reimers and Gurevych, 2020) leverage NLI data and knowledge distillation. LASER and LaBSE are strong multilingual baselines but require large-scale parallel corpora—223M for LASER and 6B for LaBSE—and often struggle with semantically similar but non-identical sen- tence pairs. 6 Conclusion Cross-lingual alignment ensures that semantically similar meanings across languages have closer rep- resentations. While previous work has focused on aligning bilingual text pairs, we show that training on multi-way parallel text—where all translations of a semantic instance are aligned together—is more effective. By leveraging translations from an MT model and allowing all languages to serve as anchors, our approach enhances cross-lingual alignment and improves downstream performance across multiple tasks in the MTEB benchmark. We also find notable improvements even for languages not included in the alignment training. This demon- strates the broader generalization impact of cross- lingual alignment with the proposed multi-way par- allelism in the alignment dataset. 7 Limitations 1.Our approach relies on a multi-way parallel corpus, synthesized by translating English source text into six target languages using the NLLB model (Team et al., 2022). How- ever, we do not analyze the sensitivity of our training to translation quality. With consis- tently strong improvements across languages, we do not observe clear trends for lower- resource languages like Chinese, where trans- lation quality may lag behind that of lan- guages closer to English, such as German or French. 2.This study focuses on a limited set of six lan- guages (in addition to English) as a prelimi- nary investigation. Extending our approach to a massively multilingual setting remains an open direction for future work. 3. Our alignment experiments are conducted on a relatively small dataset of∼ 76Krows of four-way parallel sentences in multiple lan- guages. Scaling up the alignment corpus to larger sizes and adapting the training tech- niques to different kinds of models is left for future work. 4.The alignment technique presented here is de- signed for multilingual understanding tasks and may not directly benefit generative tasks, as suggested by (Li and Murray, 2023). Fu- ture research could explore evaluating this alignment on generative tasks and adapting the training strategy to generative models. References Mikel Artetxe and Holger Schwenk. 2019.Mas- sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610. Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multi- lingual alignment of contextual word representations. Preprint, arXiv:2002.03518. Aditi Chaudhary, Karthik Raman, Krishna Srinivasan, and Jiecao Chen. 2020. Dict-mlm: Improved mul- tilingual pre-training using bilingual dictionaries. Preprint, arXiv:2010.12566. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Lin- guistics. Ameet Deshpande, Partha Talukdar, and Karthik Narasimhan. 2022. When is BERT multilingual? isolating crucial ingredients for cross-lingual transfer. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 3610–3623, Seattle, United States. Association for Computational Linguistics. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022.Language- agnostic bert sentence embedding. Preprint, arXiv:2007.01852. Katharina Hämmerl, Jind ˇ rich Libovický, and Alexan- der Fraser. 2024.Understanding cross-lingual Alignment—A survey. In Findings of the Associa- tion for Computational Linguistics: ACL 2024, pages 10922–10943, Bangkok, Thailand. Association for Computational Linguistics. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generaliza- tion. Preprint, arXiv:2003.11080. Prannay Khosla,Piotr Teterwak,Chen Wang, Aaron Sarna,Yonglong Tian,Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2021. Supervised contrastive learning. Preprint, arXiv:2004.11362. Jiahuan Li, Shujian Huang, Aarron Ching, Xinyu Dai, and Jiajun Chen. 2024. PreAlign: Boosting cross- lingual transfer by early establishment of multilingual alignment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10246–10257, Miami, Florida, USA. Associa- tion for Computational Linguistics. Tianjian Li and Kenton Murray. 2023. Why does zero- shot cross-lingual generation fail? an explanation and a solution. Preprint, arXiv:2305.17325. Ziheng Li, Shaohan Huang, Zihan Zhang, Zhi-Hong Deng, Qiang Lou, Haizhen Huang, Jian Jiao, Furu Wei, Weiwei Deng, and Qi Zhang. 2023. Dual- alignment pre-training for cross-lingual sentence em- bedding. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3466–3478, Toronto, Canada. Association for Computational Linguistics. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. Mteb: Massive text embedding benchmark. Preprint, arXiv:2210.07316. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gram- fort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vin- cent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. Journal of machine learning re- search, 12(Oct):2825–2830. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. Preprint, arXiv:1908.10084. Nils Reimers and Iryna Gurevych. 2020.Making monolingual sentence embeddings multilingual using knowledge distillation. Preprint, arXiv:2004.09813. Andrew Rosenberg and Julia Hirschberg. 2007. V- measure: A conditional entropy-based external clus- ter evaluation measure. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 410– 420, Prague, Czech Republic. Association for Com- putational Linguistics. Dan Saattrup Nielsen, Kenneth Enevoldsen, and Peter Schneider-Kamp. 2025. Encoder vs decoder: Com- parative analysis of encoder and decoder language models on multilingual NLU tasks. In Proceedings of the Joint 25th Nordic Conference on Computa- tional Linguistics and 11th Baltic Conference on Hu- man Language Technologies (NoDaLiDa/Baltic-HLT 2025), pages 561–572, Tallinn, Estonia. University of Tartu Library. Henry Tang,Ameet Deshpande,and Karthik Narasimhan. 2022. Align-mlm: Word embedding alignment is crucial for multilingual pre-training. Preprint, arXiv:2211.08547. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Rop- ers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022.No language left behind: Scal- ing human-centered machine translation. Preprint, arXiv:2207.04672. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. Preprint, arXiv:2402.05672. Shijie Wu and Mark Dredze. 2020. Do explicit align- ments robustly improve multilingual encoders? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4471–4482, Online. Association for Computa- tional Linguistics. A Model Training details For all models, no auxiliary layers or parameters were added while finetuning for alignment end-to- end. The loss was applied over the pooler output with 768 dimensions. Early stopping with patience of 10 over 20 epochs with batch size of 32 was im- plemented for XLMR-base and mBERT-base. For mE5, batch size of 128 was used over 5 epochs. We additionally boosted positive pairs with lower co- sine similarity during contrastive learning by scal- ing respective logits withexp(−z i .z p ). Tempera- ture and regularisation scalar were carefully tuned for each of the models to select the best performing aligned version. Note that we maintain anchors in the contrastive loss from each language in both models described in Section 4.1. We conducted experiments using a NVIDIA A100-SXM4-80GB GPU with CUDA 12.4 and using PyTorch 2.6. B More plots Figures 4 shows how the range of similarity scores for matched Chinese and French sentences get well separated and higher post alignment from random pairs of sentences from the two languages. Simi- larly for English-Hindi in 3,English-French in Fig- ure 5 and for Chinese-Hindi in Figure 6. Figure 3: Histograms of cosine similarity scores before and after alignment for English and Hindi sentences Figure 4: Histograms of cosine similarity scores before and after alignment for Chinese and French sentences Figure 5: Histograms of cosine similarity scores before and after alignment for English and French sentences Figure 6: Histograms of cosine similarity scores before and after alignment for Chinese and Hindi sentences