Paper deep dive

Improving Cross-Lingual Token Representations by Adding a Pinch of SALT

Guillem Ramírez

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 85

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 9/10/2026, 3:35:37 AM

Summary

The paper introduces SALT (Span-Aligned Learning for cross-lingual Tokens), a lightweight post-training method that improves token representations in cross-lingual sentence encoders by injecting span-level supervision. SALT augments standard sentence-level training with span-aligned contrastive and translation objectives while preserving the original sentence embedding space via an interpolation loss. Evaluated on five multilingual benchmarks, SALT achieves state-of-the-art results on four token-level tasks (word alignment, sequence tagging, etc.) and maintains or improves sentence-level performance.

Entities (13)

SALT · method · 98%Sequence Tagging · task · 95%word alignment · task · 95%SONAR · model · 92%MASSIVE · dataset · 90%MEXMA · model · 90%PAN-X · dataset · 90%UDPOS · dataset · 90%WiC · dataset · 90%CASE · method · 88%LaBSE · model · 85%Llama-70B · model · 85%XLM-R · model · 85%

Relation Signals (10)

SALT → improves → token representations

confidence 95% · SALT, a lightweight post-training method that improves token representations by injecting span-level supervision into existing sentence encoders.

SALT → uses → span-level supervision

confidence 95% · We propose SALT... that improves token representations by injecting span-level supervision into existing sentence encoders.

SALT → outperforms → alternative fine-tuning strategies

confidence 92% · Across five multilingual token-level benchmarks, SALT achieves the best overall results on four of them, outperforming alternative fine-tuning strategies and competitive encoders.

SALT → appliedto → SONAR

confidence 90% · We initialiseE,Dwith the respective pre-trained SONAR weights... We fine-tune SONAR on a subset of the data...

SALT → improves → PAN-X

confidence 90% · Among the fine-tuning objectives, SALT outperforms in ... PAN-X...

SALT → improves → word alignment

confidence 90% · Among the fine-tuning objectives, SALT outperforms in word alignment (AER)...

SALT → improves → Massive

confidence 90% · Among the fine-tuning objectives, SALT outperforms in ... Massive...

SALT → improves → WiC

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Cross-lingual sentence encoders enable scalable transfer across hundreds of languages, powering applications such as translation mining and zero-shot learning in low-resource settings. Although trained for sentence-level alignment, they are increasingly also applied to token-level tasks such as hallucination detection and sequence tagging, exposing a mismatch between training and usage. We propose SALT, a lightweight post-training method that improves token representations by injecting span-level supervision into existing sentence encoders. Across five multilingual token-level benchmarks, SALT achieves the best overall results on four of them, outperforming alternative fine-tuning strategies and competitive encoders. It also improves sentence-level performance on cross-lingual retrieval and classification tasks. These results demonstrate that span-level supervision is an effective signal for improving both token and sentence representations.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

83,167 characters extracted from source content.

Expand or collapse full text

Improving Cross-Lingual Token Representations by Adding a Pinch of SALT Guillem Ramírez Santos ILCC, University of Edinburgh gramirez@ed.ac.uk Abstract Cross-lingual sentence encoders enable scalable transfer across hundreds of languages, powering applications such as translation mining and zero-shot learning in low-resource settings. Although trained for sentence-level alignment, they are increasingly also applied to token-level tasks such as hallucination detection and sequence tagging, exposing a mismatch between training and usage. We propose SALT, a lightweight post-training method that improves token representations by injecting span-level supervision into existing sentence encoders. Across five multilingual token-level benchmarks, SALT achieves the best overall results on four of them, outper- forming alternative fine-tuning strategies and competitive encoders. It also improves sentence-level performance on cross-lingual retrieval and classification tasks. These results demonstrate that span-level supervision is an effective signal for improving both token and sentence representations. 1 Introduction Cross-lingual sentence encoders have become a core building block of multilingual NLP. Mod- els such as SONAR (Duquenne et al.,2023) or MEXMA ( Janeiro et al.,2025) learn a shared em- bedding space in which semantically equivalent sentences across languages are mapped to nearby representations, leading to strong performance on sentence-level tasks such as cross-lingual retrieval, sentence mining, and zero-shot classification. Despite being trained for cross-lingual sentence alignment, these models are increasingly used as token-level representationsin downstream appli- cations, including word alignment ( Miao et al., 2024), sequence tagging (Mehta and Varma,2023), hallucination detection (Huang et al.,2024), sen- tence segmentation (Omnilingual MT Team et al., 2026) and label projection (Parekh et al.,2024). This practice exposes a structural mismatch: sen- tence encoders are optimised to preserve global semantic similarity, yet are repurposed to support fine-grained token- and phrase-level alignment. As a result, they may yield well-structured sentence embeddings but an inconsistent or noisy token rep- resentation. A promising way to address this mismatch is to introduce training signals at a finer granularity by identifying aligned words across languages and incorporating token-level losses (Alqahtani et al., 2021;Miao et al.,2024). However, word-level alignment is often too restrictive, as meaning is fre- quently distributed across multiple tokens and only fully preserved at the level of phrases rather than individual words. To this end, we propose using phrases orspans— contiguous sequences of tokens that express equiv- alent meaning across languages. While this align- ment signal is implicitly present in parallel data, previous sentence- or token-level objectives do not exploit it. We introduce SALT(Span- AlignedLearning for cross-lingualTokens), a novel lightweight post-training method for multi- lingual sentence encoders (see Figure 1). SALT augments standard sentence-level training with span-aligned contrastive and translation objec- tives that explicitly shape token- and phrase-level representations, while a sentence-level interpola- tion objective preserves the original embedding space. Importantly, SALT requires no architectural modifications and can be applied directly to exist- ing encoders. We evaluate SALT on a suite of multilingual token-level benchmarks, including word align- ment, cross-lingual sequence tagging, and word sense disambiguation. Across tasks, SALT con- sistently improves token-level performance while maintaining or improving sentence-level quality. In particular, it achieves the best overall perfor- mance among several fine-tuning strategies on four arXiv:2609.09953v1 [cs.CL] 9 Sep 2026 Pre-processing Span alignment extraction with your age, all chest pain should be treated this way a su edad el dolor de pecho debe tratarse de esta manera with your age , all chest pain should be treated this way a su edad el dolor de pecho debe tratarsede esta manera Encoder Sentence-level representation + = Teacher embedding Training a su edad el dolor de pecho debe tratarse de esta manera with your age, all chest pain should be treated this way EncoderEncoder · asuedadeldolordepechomanera · withyourage,allchestpainway Average pool L INT =MSE( + , ) · · · Span-aggregated embeddings · · · Span-based contrastive loss L CON SPAN Decoder with your age L MT SPAN Figure 1:Pre-processing(top): extraction of aligned spans and teacher embeddings from a pre-trained encoder. Training SALT(bottom): we train the encoder with span-level contrastive and translation losses to improve token- level representations, besides an interpolation loss that preserves sentence embeddings. out of five token-level benchmarks, demonstrating that span-level supervision provides a strong com- plementary signal rather than a competing objec- tive. Our contributions are as follows: •We proposeSALT, a lightweight post- training method that injects span-level alignment into sentence encoders, bridging sentence-level training and token-level applications. •We demonstrate that span-aligned super- vision consistently improves performance across multiple token-level multilingual tasks, while preserving and sometimes improving sentence-level embedding quality. •We analyse the geometry of the word em- bedding space and find that SALT yields a better-structured cross-lingual representation, improving hubness while preserving isotropy and reducing language-specific clustering. 2 Related work Sentence encodersCross-lingual sentence en- coders aim to learn a shared semantic space in which sentences with equivalent meaning across languages are mapped to nearby embeddings. Early approaches based on multilingual word embeddings have evolved into large-scale sen- tence encoders trained on parallel corpora us- ing contrastive, regression, or decoding objec- tives. Models such as SONAR ( Duquenne et al., 2023), MEXMA (Janeiro et al.,2025), Om- niSONAR (Omnilingual SONAR Team et al., 2026) or LaBSE (Feng et al.,2022) exemplify this paradigm. Word and phrase alignmentEarly statistical machine translation relied on word alignment mod- els and derived phrase pairs as discrete transla- tion units for decoding ( Brown et al.,1993;Koehn et al.,2003), establishing a notion of cross-lingual span correspondence. While this is related in spirit to SALT, we use such span correspondences only as a training signal. More recently, word align- ment has been revisited in neural encoders, with methods extracting alignments from contextual representations ( Jalili Sabet et al.,2020). Related work improves cross-lingual alignment through ob- jectives that encourage finer-grained token-level correspondence ( Chi et al.,2021;Dou and Neubig, 2021). Bridging sentence- and token-level representationsPrior work improves cross- lingual sentence encoders by incorporating auxiliary word- or token-level objectives during pre-training (Wei et al.,2021;Alqahtani et al., 2021;Li et al.,2021,2023;Miao et al.,2024), typically relying on explicit alignment signals to enhance sentence-level embeddings. In con- trast, we directly target token- and span-level representations, both in training and evaluation, explicitly focusing on improving their quality rather than treating them as a by-product of sentence-level learning, focusing on adapting existing sentence encoders for token-centric tasks. 3SALT: Span-Aligned Learning for Cross-Lingual Tokens To improve cross-lingual token representations produced by sentence encoders, we propose SALT, a training method that aligns semantically equiva- lent spans across languages. SALT is lightweight and can be applied post-training to any pre-trained encoder without modifying its architecture. As illustrated in Figure 1, we first extract semanti- cally equivalent multi-word spans from parallel sentences and precompute their corresponding sen- tence representations. During training, span-level alignment objectives refine token and span embed- dings while preserving the structure of the original sentence-level representation space. 3.1 Method Letxandybe a translation pair. A tokeniserT with vocabularyVmaps them to token sequences t x =T(x)∈V n andt y =T(y)∈V m . A pre- trained encoderEthen produces contextual embed- dingsH x =E(t x )∈R n×d andH y =E(t y )∈ R m×d . These representations are typically trained such that their pooled sentence embeddings are aligned, i.e., pool(H x )≈pool(H y ). Following Duquenne et al.(2023), we adopt average pool- ing, although our approach is compatible with al- ternative strategies, such as max pooling or the use of a designated [CLS] token (as in Janeiro et al. (2025)). We define aspan pairas a pair of contiguous token subsequences from a translation pair with similar contextual meaning. Formally, it is given by(t x [i:j], t y [i ′ :j ′ ]).t x [i:j]denotes tokens i, . . . , j−1with0≤i < j≤n; 0≤i ′ < j ′ < m. The span representations are obtained by pooling the token embeddings within each subsequence, i.e., pool(H x [i:j])and pool(H y [i ′ :j ′ ]). 3.1.1 Span extraction We require aligned spans across parallel sentences to construct training supervision. We consider two approaches for span extraction: (i) an LLM-based method, and (i) a lightweight heuristic, CASE. In most of our experiments, we use LLaMA- 70B (Team,2024) to extract aligned spans (see Ap- pendixB.1). However, LLM-based extraction can be computationally expensive and not always avail- able. To address this, we introduce CASE (Con- stituent Alignment Span Extraction), a heuristic that extracts aligned spans using token-level align- ments derived from encoderErepresentations. CASEGiven a source sentencex, we compute token-level alignmentsM:0, . . . , n−1 → 0, . . . , m−1 ∪ ∅using Argmax (Jalili Sa- bet et al. ,2020), extended to multi-token words. For a target tokent, we define the inverse mapping M −1 ( t ) = s | M ( s ) = t . We consider candidate source spanst x [i:j]. LetA i:j =M(s)|s∈ i, . . . , j−1, M(s)̸= ∅. Then we define the aligned target span as the tightest span covering all aligned tokens:i ′ = minA i:j ,j ′ =maxA i:j + 1. To ensure syntac- tic validity, we restrict source spans to constituents identified using the Benepar parser ( Kitaev et al., 2019). We discard span pairs for which align- ment links are not fully contained within the span pair, i.e., no token in the target span is aligned to a source token outside the source span, and vice versa. Finally, we define the coverage of a span pair as the proportion of aligned tokens in both directions: cover= | s|s∈ i, . . . , j−1, M(s)̸=∅ | 2((j−i) + (j ′ −i ′ )) + | t|t∈ i ′ , . . . , j ′ −1, M −1 (t)̸=∅ | 2((j−i) + (j ′ −i ′ )) (1) We discard any pair of source-target spans whose coverage falls below a thresholdc= 0.7, en- suring that only well-aligned spans are used. Evaluation of span extractionWe evaluate span quality via human evaluation on Chinese (zho_Hans), French, Hindi, Russian, and Span- ish. Spans are extracted from English–target sen- tence pairs from our word alignment dataset (Sec- tion 4.1). We recruit 7 native speakers to annotate a total of 2,361 span pairs for both LLaMA-70B and 1234 0 20 40 60 80 100 Relative distribution Chinese LLaMA-70B SONAR (CASE) 1234 Hindi 1234 0 20 40 60 80 100 Relative distribution Russian 1234 Spanish Figure 2: Human evaluation of extracted spans. Score distributions for LLaMA-70B and SONAR (CASE). CASE (with SONAR), rating span semantic equiv- alence on a 4-point scale: (1) unrelated, (2) some- what related, (3) nearly equivalent, (4) equivalent. See Appendix Gfor guidelines and full results. Figure 2shows the score distribution. Both LLaMA-based extraction and CASE yield high- quality span pairs, with over 90% of annotations in the top two categories. LLaMA performs bet- ter on Russian and French, while CASE matches or exceeds it in other languages. We retain score- 4 spans to construct CrossSpan for the evaluation in Section 5.1. Overall, both methods effectively identify semantically equivalent spans, supporting their use as supervision signals. CASE offers an ef- ficient alternative to LLM-based extraction while preserving alignment quality. 3.1.2 SALT training objective SALT uses a translation (MT) and a contrastive (CON) objective at the span level, as well as an in- terpolation objective (INT) at the sentence level. L SALT =L CON Span +αL MT Span +βL INT (2) whereα, β∈Rare hyperparameters set to one unless otherwise specified. For simplicity, we present the losses for a single translation pair (x, y), though they are computed in batches during training. Contrastive objectiveWe apply the span extrac- tion method (LLM or CASE) to obtain a collection ofpaligned spanst x [i 1 :j 1 ], . . . , t x [i p :j p ]and t y [i ′1 :j ′1 ], . . . , t y [i ′p :j ′p ]. To increase coverage of both sentences, we aug- ment this set with unaligned spans. Specifically, for every maximal contiguous subsequence of to- kens that does not belong to any aligned span, we form an additional span. This yields addi- tional spans(t x [i p+1 :j p+1 ], . . . , t x [i q :j q ])and (t y [i ′p+1 :j ′p+1 ], . . . , t y [i ′v :j ′v ]). We compute the corresponding span embed- dingsH 1 x , . . . , H q x andH 1 y , . . . , H v y by averaging the token embeddings within each span. We then define the contrastive loss as L CON Span =− 1 p p ∑ i=1 [ log exp(⟨H i x , H i y ⟩/τ) ∑ v j=1 exp(⟨H i x , H j y ⟩/τ) +log exp(⟨H i x , H i y ⟩/τ) ∑ q j=1 exp(⟨H j x , H i y ⟩/τ) ] (3) where⟨·,·⟩denotes the dot product,(H i x , H i y ) corresponds to an aligned span pair, andτis a tem- perature hyperparameter. The first term applies a row-wise softmax over target spans for each source span, while the second term applies a column- wise softmax over source spans for each target span. This loss encourages semantically equiva- lent spans to have similar embeddings while push- ing apart spans with different meanings. Translation objectiveGiven a pre-trained de- coderDthat we keep frozen, we define the transla- tion loss as the cross-entropy over the target span: L MT Span =− 1 p p ∑ l=1 j ′l −1 ∑ k=i ′l logP D ( t y [k];t y [i ′l :k], H l x ) (4) where (t x [i l :j l ]andt y [i ′l :j ′l ]) is an extracted span pair,H l x is the pooled source span represen- tation andP D denotes the probability assigned by the decoder to the next target token, conditioned on the previous target tokens and the pooled represen- tation of the source span. Interpolation objectiveFollowingTsiamas et al.(2025), we introduce a sentence-level inter- polation objective that encourages the encoder to preserve the original sentence representation space. We obtain a teacher representationz xy by averaging the source and target sentence embed- dings produced by the frozen encoderE f rozen . The interpolation loss is then defined as L INT =MSE ( z xy , pool(H x ) +pool(H y ) 2 ) (5) where MSE denotes the mean squared error. In- tuitively, this loss constrains the encoder so that token-level training objectives do not distort the sentence-level embedding space, maintaining con- sistency with the initial frozen encoder. Task (#langs) Example AER (18) Thenecessarycorrectionwillbemade. EineentsprechendeAnderungwirdvorgenommenwerden. Massive (52) What is today ’s forecast for Berlin DATEPLACE PAN-X (40) REDIRECCIÓN Algarrobo ( Chile ) B-LOC I-LOC UDPOS (51) NOUN ADP NOUN AUX PART PUNCT WiC (6) Bolivia holds a key playin this process of peace. A musical playon the same subject was also staged. Different meaning Table 1: Summary of the tasks and number of lan- guages used for the evaluation of cross-lingual token representations. Source:Omnilingual SONAR Team et al.(2026) 4 Experiment details We initialiseE,Dwith the respective pre-trained SONAR weights (Duquenne et al.,2023). DataWe fine-tune SONAR on a subset of the data originally used to train it, namely the publicly available NLLB Primary dataset ( Costa-jussà et al., 2022). 1 This dataset comprises high-quality par- allel sentences spanning a diverse set of languages. We begin by excluding all language pairs for which neither language belongs to the set of 57 desig- nated test languages, yielding a total of 129 lan- guage pairs (see Appendix C.1). The 29 test lan- guages not observed during training can be used to evaluate the generalisation of our method (see Appendix D.3). For each retained language pair, we downsam- ple the corpus to 40,000 sentence pairs, observ- ing no degradation in performance as a result of this reduction. In line with the procedure ofOm- nilingual SONAR Team et al.(2026), we further re- fine the dataset by removing sentence pairs whose BLASER 2 score ( Dale and Costa-jussà,2024) de- viates by more than one standard deviation from the mean score computed for the given language pair. Training hyperparametersWe train SONAR for 20,000 steps, with a maximum of 1400 tokens 1 Downloaded using the script provided athttps:// github.com/gordicaleksa/Open-NLLB. per batch (roughly corresponding to 50 pairs of sen- tences per batch) and a learning rate ofμ= 10e−5. We use an AdamW optimiser (Loshchilov and Hut- ter,2019) and warm-up of 1000 steps. We do the training runs on three seeds and report average val- ues in all experiments unless otherwise specified. Standard deviations are reported in AppendixF. BaselinesWe report the performance of the encoders XLM-R (Conneau and Lample,2019), XLM-Align (Chi et al.,2021), LaBSE (Feng et al., 2022), and MEXMA (Janeiro et al.,2025). In addi- tion, we compare SALT against variants obtained by fine-tuning SONAR with alternative training objectives. Specifically, we train the encoder for longer using the original SONAR loss proposed by Duquenne et al.(2023). We also include token-level objectives: the Self-Objective (SO) loss (Dou and Neubig,2021), a contrastive word- alignment objective; the WordOT loss (Alqahtani et al.,2021), using Optimal Transport; the WACSE loss ( Miao et al.,2024), which adds a masked language modeling head; and the OmniSONAR- Token loss (Omnilingual SONAR Team et al., 2026), which extends SO loss with interpolation. See Appendix C.4for implementation details. Sentence evaluationWe evaluate the task of sentence mining using the FLORES-200 de- vtest ( NLLB Team,2024) (80 languages) and re- port xsim ( Feng et al.,2022) and xsim++ (Chen et al.,2023). For classification, we report on tasks from MTEB (Muennighoff et al.,2023) (English only). 4.1 Cross-lingual token evaluation Standard benchmarks often evaluate sentence en- coders through sequence-level tasks, which can mask weaknesses in token representations (Con- neau et al. ,2018;Hu et al.,2020;Enevoldsen et al.,2025). FollowingOmnilingual SONAR Team et al.(2026), we instead focus on tasks that isolate the embeddings of individual tokens, pro- viding a more rigorous assessment of cross-lingual token quality. For word alignment, we test whether tokens across languages align semantically. For se- quence tagging, we feed the classifier head only the embedding of a single word to directly probe the quality of its token-level representations across languages. Word alignmentGiven a translation pairx, y, the task of word alignment identifies which words correspond to each other semantically. More for- mally, given a source–target sentence pairx,ywith word lengthsw x ,w y , respectively, we infer a bi- nary matrixM∈ 0,1 w s ×w y , whereM ij = 1 represents that thei-th word in the source sentence aligns semantically with thej-th word in the target sentence. We present an example in Table1. Many current word alignment methods derive alignments based on the similarity of token embed- dings (Jalili Sabet et al.,2020;Azadi et al.,2023). We define thetoken similarity matrixS∈R n×m ; S i,j :=sim(E(t x [i]),E(t y [j])), wheresimis a similarity measure (we use cosine similarity),t x [i] andt y [j]∈Vare thei-th andj-th tokens of the source and target sentencesxandy. Anextrac- tionmethod is then applied to convert these sim- ilarities into a discrete alignment. We use Iter- max (Jalili Sabet et al.,2020) to extract alignments and report Alignment Error Rate (AER) (Och and Ney,2000). The full list of alignment datasets is in Appendix C.2. Sequence taggingWe evaluate token-level cross-lingual representations using standard sequence labelling tasks, where each tokenx i in an input sequencex= (x 1 , . . . , x n )is as- signed a discrete label1, . . . , N. To isolate the contribution of the encoder, we train a linear classification headC∈R d×N placed on top of the token embeddings while keeping the encoder parameters fixed. The classifier is trained only on English data and subsequently evaluated in other languages (zero-shot); see Appendix C.3for implementation details. We consider a diverse set of benchmarks, includ- ing Massive (Slot Filling) ( FitzGerald et al.,2023), PAN-X (Named Entity Recognition) ( Pan et al., 2017), UDPOS (Part-of-Speech Tagging) (Nivre et al. ,2020), and WiC (Word Sense Disambigua- tion) (Pilehvar and Camacho-Collados,2019). WiC is not technically a sequence tagging task; we adapt it by extracting the contextual embeddings of the target word in each sentence, concatenating the resulting vectors, and feeding them into a classifier C∈R 2d×2 . When a target word is split into mul- tiple subword tokens, we represent it by averaging the corresponding embeddings. Finally, to handle mismatches between dataset annotations and tokeniser segmentation, we map labels to tokens based on maximal character over- lap. When multiple segments overlap a token equally, we resolve ties by selecting the first. 5 Results Table2compares SALT with pre-trained en- coders and alternative fine-tuning objectives for SONAR across five cross-lingual token bench- marks. Among the pre-trained models, MEXMA achieves the lowest AER (0.171), while SONAR remains competitive on the other tasks. Continued training with the SONAR loss improves all token- level results, suggesting that SONAR was primar- ily optimised for sentence mining rather than token representations. Among the fine-tuning objectives, SALT outper- forms in word alignment (AER), PAN-X, Massive, and WiC, showing that span-level objectives pro- vide complementary signals beyond sentence- or token-level training. Gains are particularly pro- nounced on tasks requiring rich cross-lingual se- mantics (PAN-X, Massive) and on word alignment. On UDPOS, SALT achieves a strong score of 0.571 (vs. 0.548 for the pre-trained model), but is slightly outperformed by SO Loss (0.577). This may be due to UDPOS being more language-specific, ex- hibiting limited cross-lingual transfer relative to other tasks (see AppendixD.2) Taken together, the results show that SALT achieves the best overall performance among the considered strategies, and suggest that incorpo- rating span-level supervision is an effective and principled way to enrich multilingual sentence representations beyond sentence-level objectives alone. Does improved token-level representation come at the cost of sentence-level quality? For cross- lingual sentence mining, SALT consistently im- proves over SONAR on xsim and xsim++ (Ta- ble 3). For MTEB classification tasks, SALT matches SONAR on XNLI and MIntent and im- proves on STS17. Overall, SALT does not sacri- fice sentence-level quality and, in several cases, im- proves it. 5.1 Analysis Span embeddings SALT supports span-level representations by averaging token embeddings within each span. We find that these span representations are more semantically informative than their SONAR counterparts. On the CrossSpan test set of high-quality aligned cross-lingual spans, SALT representations of aligned spans consistently exhibit higher cosine similarity across layers (Fig- ure 4, left), indicating stronger cross-lingual align- AER (↓) PAN-X (↑) Massive (↑) UDPOS (↑) WiC (↑) Pre-trained encoders XLM-R0.3070.5510.2200.5270.538 XLM-Align0.2460.5320.1870.5220.556 LaBSE0.2320.5890.4310.5510.538 MEXMA0.1710.5930.3960.5530.565 SONAR0.1830.5820.4020.5480.576 SONAR + additional objectives SONAR Loss(Duquenne et al.,2023)0.1790.5880.4260.5540.581 SO Loss(Dou and Neubig,2021)0.1780.5970.4100.5770.574 WordOT(Alqahtani et al.,2021)0.3200.6090.4170.5610.585 WACSE(Miao et al.,2024)0.2050.6000.4090.5690.573 OmniSONAR-Token( Omnilingual SONAR Team et al.,2026)0.1800.6100.4170.5740.583 SALT(ours)0.171 0.6220.4520.5710.585 Table 2: Cross-lingual token-level evaluation across five benchmarks for frozen pre-trained encoders and SONAR trained with additional objectives. SALT achieves the best or joint-best performance on four of five benchmarks. -10-505 PC1 -10 -5 0 5 10 15 PC2 La Gran Piràmide es va crear en honor del faraó Kheops, i moltes de les piràmides, tombes i temples menors es van erigir en honor de les esposes i dels membres de la família del faraó. Die große Pyramide wurde dem Pharao Cheops zu Ehren erbaut und viele der kleineren Pyramiden, Grabmale und Tempel wurden zu Ehren von Cheops Ehefrauen und Familienmitgliedern errichtet. La grande pyramide a été crée pour honorer le pharaon Khéops, et beaucoup de petites pyramides, tombes et temples ont été construits pour honorer ses femmes ainsi que les membres de sa famille. महान पिरामिड फेरो खुफू के सम्मान में बनाया गया था, और कई छोटे पिरामिड, कब्र और मंदिरों का निर्माण खुफू की पत्नियों और परिवार के सदस्यों के सम्मान में किया गया था। L'importante piramide fu creata in onore del faraone Cheope, e molte delle piramidi, delle tombe e dei templi minori furono edificati per onorare le mogli e i familiari del faraone. ▁大ピラミッドはファラオ・クフ王を称えて建てられた巨大な墓であり、その周囲にある小ピラミッド、墓、神殿の多くはクフ王の妻や家族を称えるものです。 A grande pirâmide foi criada em homenagem ao Faraó Khufu, e muitas das pirâmides, tumbas e templos menores foram construídos para homenagear as esposas e familiares de Khufu. Великая Пирамида была создана в честь фараона Хеопса, а много меньших пирамид, гробниц и храмов были построены в честь жён Хеопаса и его родствеников. La gran pirámide fue construida en honor al faraón Khufu, y muchas otras de este tipo, tumbas y templos más pequeños se levantaron en honor a sus esposas y familiares. ▁大金字塔是為了榮耀古夫法老所建造的 , 而許多較小的金字塔、陵墓和神廟則是為了榮耀古夫的妻妾和家庭成員而建。 The great pyramid was created to honor the Pharaoh Khufu, and many of the smaller pyramids, tombs, and temples were built to honor Khufu's wives and family members. MEXMA -0.50.00.51.01.5 PC1 -1.0 -0.5 0.0 0.5 1.0 La Gran Piràmide es va crear en honor del faraó Kheops, i moltes de les piràmides, tombes i temples menors es van erigir en honor de les esposes i dels membres de la família del faraó. Die große Pyramide wurde dem Pharao Cheops zu Ehren erbaut und viele der kleineren Pyramiden, Grabmale und Tempel wurden zu Ehren von Cheops Ehefrauen und Familienmitgliedern errichtet. La grande pyramide a été crée pour honorer le pharaon Khéops, et beaucoup de petites pyramides, tombes et temples ont été construits pour honorer ses femmes ainsi que les membres de sa famille. महान पिरामिड फेरो खुफू के सम्मान में बनाया गया था, और कई छोटे पिरामिड, कब्र और मंदिरों का निर्माण खुफू की पत्नियों और परिवार के सदस्यों के सम्मान में किया गया था। L'importante piramide fu creata in onore del faraone Cheope, e molte delle piramidi, delle tombe e dei templi minori furono edificati per onorare le mogli e i familiari del faraone. ▁大ピラミッドはファラオ・クフ王を称えて建てられた巨大な墓であり、その周囲にある小ピラミッド、墓、神殿の多くはクフ王の妻や家族を称えるものです。 A grande pirâmide foi criada em homenagem ao Faraó Khufu, e muitas das pirâmides, tumbas e templos menores foram construídos para homenagear as esposas e familiares de Khufu. Великая Пирамида была создана в честь фараона Хеопса, а много меньших пирамид, гробниц и храмов были построены в честь жён Хеопаса и его родствеников. La gran pirámide fue construida en honor al faraón Khufu, y muchas otras de este tipo, tumbas y templos más pequeños se levantaron en honor a sus esposas y familiares. ▁大金字塔是為了榮耀古夫法老所建造的 , 而許多較小的金字塔、陵墓和神廟則是為了榮耀古夫的妻 <unk> 和家庭成員而建。 The great pyramid was created to honor the Pharaoh Khufu, and many of the smaller pyramids, tombs, and temples were built to honor Khufu's wives and family members. SONAR -1.0-0.50.00.51.0 PC1 -0.4 0.0 0.4 0.8 La Gran Piràmide es va crear en honor del faraó Kheops, i moltes de les piràmides, tombes i temples menors es van erigir en honor de les esposes i dels membres de la família del faraó. Die große Pyramide wurde dem Pharao Cheops zu Ehren erbaut und viele der kleineren Pyramiden, Grabmale und Tempel wurden zu Ehren von Cheops Ehefrauen und Familienmitgliedern errichtet. La grande pyramide a été crée pour honorer le pharaon Khéops, et beaucoup de petites pyramides, tombes et temples ont été construits pour honorer ses femmes ainsi que les membres de sa famille. महान पिरामिड फेरो खुफू के सम्मान में बनाया गया था, और कई छोटे पिरामिड, कब्र और मंदिरों का निर्माण खुफू की पत्नियों और परिवार के सदस्यों के सम्मान में किया गया था। L'importante piramide fu creata in onore del faraone Cheope, e molte delle piramidi, delle tombe e dei templi minori furono edificati per onorare le mogli e i familiari del faraone. ▁大ピラミッドはファラオ・クフ王を称えて建てられた巨大な墓であり、その周囲にある小ピラミッド、墓、神殿の多くはクフ王の妻や家族を称えるものです。 A grande pirâmide foi criada em homenagem ao Faraó Khufu, e muitas das pirâmides, tumbas e templos menores foram construídos para homenagear as esposas e familiares de Khufu. Великая Пирамида была создана в честь фараона Хеопса, а много меньших пирамид, гробниц и храмов были построены в честь жён Хеопаса и его родствеников. La gran pirámide fue construida en honor al faraón Khufu, y muchas otras de este tipo, tumbas y templos más pequeños se levantaron en honor a sus esposas y familiares. ▁大金字塔是為了榮耀古夫法老所建造的 , 而許多較小的金字塔、陵墓和神廟則是為了榮耀古夫的妻 <unk> 和家庭成員而建。 The great pyramid was created to honor the Pharaoh Khufu, and many of the smaller pyramids, tombs, and temples were built to honor Khufu's wives and family members. SALT Figure 3: Principal component projections of token embeddings for the same sentence across 80 languages. Each colour corresponds to a language. SONAR produces a more distributed and language-agnostic embedding space. Sentence:The great pyramid was created to honor the Pharaoh Khufu, and many of the smaller pyramids, tombs, and temples were built to honor Khufu’s wives and family members. Mining (↓)Classification (↑) xsim xsim++ XNLI STS17 MIntent SONAR 0.29.90.61 0.650.58 SALT0.1 8.80.610.660.58 Table 3: Cross-lingual sentence mining (FLORES-80) and English-only MTEB classification performance. ment. A by-product of our span-translation objective is that the learned representations can be directly translated with afrozenSONAR decoder (Fig- ure 4, right). SALT span embeddings yield high- quality translations, indicating that they preserve rich semantic information for downstream genera- tion. Intermediate representationsSentence en- coders typically exhibit aparabolicbehaviour, where intermediate layers achieve the strongest cross-lingual alignment ( Jalili Sabet et al., 2020). This arises because cross-lingual token 812162024 Layer 0.8 0.9 Similarity ( ↑ ) SALT SONAR en-X-en ChrF++ ( ↑ ) 11.7 12.4 51.7 56.8 Figure 4: Left: cosine similarity of span pairs across layers. SALT shows a better similarity than SONAR across all layers. Right: translation performance for span embeddings. Both experiments use CrossSpan. representations emerge in mid-layers but are not necessarily preserved in the final layers, which are more specialised for the pretraining objective. The SALT loss shifts this pattern, with deeper layers becoming more cross-lingually aligned for both spans (Figure 4, left) and words (AppendixE.1). Geometry of the embedding spaceWe com- pare the geometry of word embeddings produced by SALT, SONAR, and MEXMA (Table4; see Ap- pendixE.2for more details on experiments). We construct a word retrieval task by merging the datasets from the word alignment task. For each English word, we retrieve the nearest neigh- bours with cosine similarity and evaluate whether the matches correspond to gold-aligned targets. SALT achieves the highest recall. A good cross-lingual representation should be language-agnostic, without inducing language- specific subspaces. We probe language separabil- ity by training a linear classifier to predict language ID from individual word embeddings. SALT em- beddings are significantly harder to classify, in- dicating more language-invariant representations than SONAR and MEXMA. Isotropy captures whether the embedding space is dominated by privileged directions, leading to uneven use of representational capacity. We mea- sure it via mean cosine similarity over random word pairs, and via uniformity ( Wang and Isola, 2020). MEXMA is highly anisotropic. SONAR and SALT produce more isotropic spaces by these measures, though as we show below, SONAR’s ap- parent isotropy masks a severe hubness problem. Hubness measures the tendency of a few embed- dings to dominate nearest-neighbour retrieval re- gardless of semantics ( Radovanovic et al.,2010). We quantify it via the skewness of thek-occurrence distribution, which counts how often an embed- ding appears in another word’s nearest neighbours. Zero skewness indicates a uniform space, while high skewness reflects hub collapse. SONAR exhibits severe hubness (11.5 skewness), while MEXMA and SALT are substantially better be- haved. Taken together, these results reveal contrast- ing geometric failures in the baselines. MEXMA produces an anisotropic, language-stratified space, while SONAR suffers from severe hub collapse. SALT avoids both, yielding a better-distributed cross-lingual token space (Figure 3). This suggests that isotropy, hubness and language inseparability are not independent but mutually reinforcing signa- tures of a well-formed cross-lingual geometry. 5.2 Ablations Impact of the different lossesWe assess the contribution of each loss term of the SALT training objective by removing them and evaluating across two metrics: word alignment (AER) and averaging the four sequence tagging tasks. The results are presented in Table6. We see that Word Ret.↑Lang. ID↓Isotropy↓Hub↓ sim / unif MEXMA63%0.490.44 / -2.2 4.0 SONAR62%0.460.04 /-3.811.5 SALT65%0.35 0.03/ -3.63.5 Table 4: Word retrieval (Recall@5), language identifi- cation (F1), isotropy (mean cosine similarity and unifor- mity) and hubness (skewness) for three encoders. AER PAN-X Massive UDPOS WiC SALT (LLaMA) 0.1710.6220.4520.571 0.585 SALT (CASE) 0.171 0.6130.4590.568 0.584 Table 5: Cross-lingual token-level evaluation for SALT using LLaMA-70B spans or the self-supervised CASE. post-training onL CON Span or onL INT improves on both tasks, being the latest the strongest sin- gle loss;L MT Span worsens the pre-trained model, and it worsens word alignment when used in con- junction withL CON Span or onL INT ; however, it adds complementary information not fully cap- tured by the other two terms — its inclusion in SALT pushes SeqTag to the best score (0.557), even asL CON Span +L INT yields a marginally bet- ter AER. The full SALT objective achieves the best se- quence tagging score and a highly competitive AER of 0.171, confirming that all three terms are necessary for optimal performance. No subset con- sistently dominates across both metrics, indicating that each loss captures complementary training sig- nals. CASE span extractionDo the SALT improve- ments stem from an additional external signal, given that LLaMA-70B was used for span extrac- tion? To investigate this, we compare results when training SALT with the self-supervised span ex- traction heuristic CASE. Results are shown in Ta- ble 5. CASE yields better performance on Mas- sive, whereas external supervision from LLaMA- 70B is superior on PAN-X, UDPOS, and WiC. Nevertheless, the difference is marginal: CASE would outperform all the baseline losses and en- coders in Table 2except for UDPOS. This sug- gests that the training regime is primarily respon- sible for SALT’s success, rather than the span ex- traction signal itself, and that comparable results can be achieved without access to a strong LLM. AER↓SeqTag↑ SONAR (pre-trained)0.183 0.527 L CON Span 0.178 0.538 L MT Span 0.186 0.522 L INT 0.174 0.551 L CON Span +L MT Span 0.179 0.551 L MT Span +L INT 0.177 0.554 L CON Span +L INT 0.1700.552 L CON Span +L MT Span +L INT (SALT) 0.1710.557 Table 6: Finetuning SONAR with different losses. 6 Conclusions In this work, we introduced SALT, a lightweight post-training method that improves cross-lingual token representations in multilingual sentence encoders through span-aligned supervision. Across multilingual token-level benchmarks, SALT achieved the strongest overall performance, consistently outperforming alternative fine-tuning strategies and competitive encoders. These gains did not come at the expense of sentence-level quality: SALT preserved and, in several cases, improved performance on sentence mining and classification benchmarks. Overall, our results suggest that span-level supervision is a simple yet powerful inductive bias for multilingual representation learning. Limitations Despite the strong empirical performance of SALT, several limitations remain. SALT relies on parallel corpora to extract aligned spans, which may limit its applicability to truly low-resource languages. In addition, the self- supervised CASE heuristic assumes that the under- lying encoder already provides reasonably good cross-lingual token representations, making adap- tation to unseen or severely underrepresented lan- guages more challenging; in those cases, LLM- based spans should be used instead. Our method also uses average pooling to con- struct span embeddings. While lightweight and architecture-agnostic, this may fail to capture finer internal structure within longer or compositionally complex spans. Finally, we only evaluate SALT as a post- training intervention. Exploring span-level super- vision during large-scale pre-training could poten- tially yield larger improvements, but was beyond the computational scope of this work. Ethics statement This work aims to improve multilingual language representations for token-level cross-lingual tasks, which may help broaden language accessibility and support underrepresented languages in NLP. How- ever, multilingual representation models may also be used in sensitive applications such as surveil- lance or profiling, and we therefore encourage re- sponsible deployment and evaluation. Our training data are derived from publicly avail- able parallel corpora from the NLLB Primary dataset, which may contain societal or cultural bi- ases present in web-scale multilingual text. As a result, SALT may inherit biases from the training data or pretrained models. We also conducted a human evaluation of span quality using native speakers. Participation was voluntary, annotators were informed about the study and their right to withdraw, and no person- ally identifiable information was collected beyond anonymised annotator IDs. The study received ethics approval from our institution. References Sawsan Alqahtani, Garima Lalwani, Yi Zhang, Salva- tore Romeo, and Saab Mansour. 2021.Using opti- mal transport as alignment objective for fine-tuning multilingual contextualized embeddings. InFind- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 3904–3919, Punta Cana, Dominican Republic. Association for Computational Linguistics. Fatemeh Azadi, Heshaam Faili, and Mohammad Javad Dousti. 2023.Pmi-align: Word alignment with point-wise mutual information without requiring par- allel training data. InFindings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, Findings of ACL, pages 12366–12377. Association for Computational Lin- guistics. Peter F. Brown, Stephen Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathemat- ics of statistical machine translation: Parameter esti- mation.Comput. Linguistics, 19(2):263–311. Mingda Chen, Kevin Heffernan, Onur Çelebi, Alexan- dre Mourachko, and Holger Schwenk. 2023.xSIM+ +: An Improved Proxy to Bitext Mining Performance for Low-Resource Languages. InProceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Short Papers), pages 101–109, Toronto, Canada. Association for Compu- tational Linguistics. Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian- Ling Mao, Heyan Huang, and Furu Wei. 2021.Im- proving pretrained cross-lingual language models via self-labeled word alignment. InProceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3418–3430, Online. Association for Computational Linguistics. Alexis Conneau and Guillaume Lample. 2019.Cross- lingual language model pretraining. InAdvances in Neural Information Processing Systems 32: An- nual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7057–7067. Alexis Conneau, Ruty Rinott, Guillaume Lample, Ad- ina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating Cross-lingual Sentence Representations. InProceed- ingsofthe2018ConferenceonEmpiricalMethodsin Natural Language Processing. Association for Com- putational Linguistics. Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Mail- lard, Anna Y. Sun, Skyler Wang, Guillaume Wen- zek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, and 19 others. 2022. No language left be- hind: Scaling human-centered machine translation. CoRR, abs/2207.04672. David Dale and Marta R. Costa-jussà. 2024.BLASER 2.0: a metric for evaluation and quality estimation of massively multilingual speech and text translation. InFindings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, Findings of ACL, pages 16075–16085. Association for Computational Lin- guistics. Zi-Yi Dou and Graham Neubig. 2021.Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: MainVolume, EACL2021, Online, April19 - 23, 2021, pages 2112–2128. Association for Com- putational Linguistics. Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023.SONAR: sentence-level multimodal and language-agnostic representations. CoRR, abs/2308.11466. Kenneth C. Enevoldsen, Isaac Chung, Imene Ker- boua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzeminski, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solo- matin, Ömer Veysel Çagatan, and 2 others. 2025. MMTEB: massive multilingual text embedding benchmark. InThe Thirteenth International Confer- ence on Learning Representations, ICLR 2025, Sin- gapore, April 24-28, 2025. OpenReview.net. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Ari- vazhagan, and Wei Wang. 2022.Language-agnostic BERT sentence embedding. InProceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 878– 891. Association for Computational Linguistics. Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natara- jan. 2023. MASSIVE: A 1M-Example Multilin- gual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages . InProceed- ingsofthe61stAnnualMeetingoftheAssociationfor ComputationalLinguistics(Volume1: LongPapers), pages 4277–4302, Toronto, Canada. Association for Computational Linguistics. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual general- ization .CoRR, abs/2003.11080. Chenyang Huang, Abbas Ghaddar, Ivan Kobyzev, Mehdi Rezagholizadeh, Osmar Zaïane, and Boxing Chen. 2024. OTTAWA: optimal transport adaptive word aligner for hallucination and omission transla- tion errors detection . InFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, Findings of ACL, pages 6322–6334. Association for Computational Linguistics. Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020.SimAlign: High qual- ity word alignments without parallel training data us- ing static and contextualized embeddings. InFind- ings of the Association for Computational Linguis- tics: EMNLP 2020, pages 1627–1643, Online. Asso- ciation for Computational Linguistics. João Maria Janeiro, Benjamin Piwowarski, Patrick Gal- linari, and Loïc Barrault. 2025.MEXMA: token- level objectives improve sentence representations. In Proceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 23960–23995. Association for Computational Linguistics. Nikita Kitaev, Steven Cao, and Dan Klein. 2019.Mul- tilingual constituency parsing with self-attention and pre-training. InProceedings of the 57th Confer- ence of the Association for Computational Linguis- tics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume1: LongPapers, pages 3499–3505. As- sociation for Computational Linguistics. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003.Statistical phrase-based translation. InHu- man Language Technology Conference of the North American Chapter of the Association for Compu- tational Linguistics, HLT-NAACL 2003, Edmonton, Canada, May 27 - June 1, 2003. The Association for Computational Linguistics. Shicheng Li, Pengcheng Yang, Fuli Luo, and Jun Xie. 2021.Multi-granularity contrasting for cross-lingual pre-training. InFindings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 1708–1717, Online. Association for Computational Linguistics. Ziheng Li, Shaohan Huang, Zihan Zhang, Zhi-Hong Deng, Qiang Lou, Haizhen Huang, Jian Jiao, Furu Wei, Weiwei Deng, and Qi Zhang. 2023.Dual- alignment pre-training for cross-lingual sentence em- bedding . InProceedings of the 61st Annual Meet- ing of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 3466–3478, Toronto, Canada. Association for Computational Lin- guistics. Ilya Loshchilov and Frank Hutter. 2019.Decou- pled weight decay regularization . In7th Inter- national Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. Federico Martelli, Andrei Stefan Bejgu, Cesare Cam- pagnano, Jaka Cibej, Rute Costa, Apolonija Gantar, Jelena Kallas, Svetla Peneva Koeva, Kristina Kop- pel, Simon Krek, Margit Langemets, Veronika Lipp, Sanni Nimb, Sussi Olsen, Bolette Sandford Pedersen, Valeria Quochi, Ana Salgado, László Simon, Carole Tiberius, and 2 others. 2023.XL-WA: a gold evalu- ation benchmark for word alignment in 14 language pairs. InProceedings of the 9th Italian Conference on Computational Linguistics, Venice, Italy, Novem- ber 30 - December 2, 2023, CEUR Workshop Pro- ceedings. CEUR-WS.org. Rahul Mehta and Vasudeva Varma. 2023.LLM- RM at SemEval-2023 task 2: Multilingual complex NER using XLM-RoBERTa. InProceedings of the 17th International Workshop on Semantic Eval- uation (SemEval-2023), pages 453–456, Toronto, Canada. Association for Computational Linguistics. Zhongtao Miao, Qiyu Wu, Kaiyan Zhao, Zilong Wu, and Yoshimasa Tsuruoka. 2024.Enhancing cross- lingual sentence embedding for low-resource lan- guages with word alignment. InFindings of the Association for Computational Linguistics: NAACL 2024, pages 3225–3236, Mexico City, Mexico. As- sociation for Computational Linguistics. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embed- ding benchmark . InProceedings of the 17th Con- ference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics. Graham Neubig. 2011. The kyoto free translation task. http://w.phontron.com/kftt. Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin- ter, Jan Hajic, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis M. Tyers, and Daniel Zeman. 2020. Universal dependencies v2: An evergrowing multilingual treebank collection. InProceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pages 4034–4043. Euro- pean Language Resources Association. NLLB Team. 2024.Scaling Neural Machine Transla- tion to 200 Languages.Nature, 630:841–846. Franz Josef Och and Hermann Ney. 2000.Improved statistical alignment models. In38th Annual Meet- ing of the Association for Computational Linguistics, Hong Kong, China, October 1-8, 2000, pages 440– 447. ACL. Omnilingual MT Team, Belen Alastruey, Niyati Bafna, Andrea Caciolai, Kevin Heffernan, Artyom Kozhevnikov, Christophe Ropers, Eduardo Sánchez, Charles-Eric Saint-James, Ioannis Tsiamas, Chierh Cheng, Joe Chuang, Paul-Ambroise Duquenne, Mark Duppenthaler, Nate Ekberg, Cynthia Gao, Pere Lluís Huguet Cabot, João Maria Janeiro, Jean Mail- lard, and 12 others. 2026. Omnilingual mt: Ma- chine translation for 1,600 languages.Preprint, arXiv:2603.16309. Omnilingual SONAR Team, João Maria Janeiro, Pere- Lluís Huguet Cabot, Ioannis Tsiamas, Yen Meng, Vivek Iyer, Guillem Ramírez, Loic Barrault, Be- len Alastruey, Yu-An Chung, Marta R. Costa- Jussa, David Dale, Kevin Heffernan, Jaehyeong Jo, Artyom Kozhevnikov, Alexandre Mourachko, Christophe Ropers, Holger Schwenk, and Paul- Ambroise Duquenne. 2026.Omnilingual sonar: Cross-lingual and cross-modal sentence embeddings bridging massively multilingual text and speech. CoRR, abs/2603.16606. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017.Cross- lingual name tagging and linking for 282 languages. InProceedings of the 55th Annual Meeting of the As- sociation for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1946–1958. Association for Computational Linguistics. Tanmay Parekh, I-Hung Hsu, Kuan-Hao Huang, Kai- Wei Chang, and Nanyun Peng. 2024.Contextual label projection for cross-lingual structured predic- tion. InProceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies (Volume 1: Long Papers), NAACL2024, Mexico City, Mexico, June 16-21, 2024, pages 5738–5757. Association for Computational Linguistics. Mohammad Taher Pilehvar and José Camacho- Collados. 2019.Wic: the word-in-context dataset for evaluating context-sensitive mean- ing representations. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Min- neapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 1267–1273. Association for Computational Linguistics. Milos Radovanovic, Alexandros Nanopoulos, and Mir- jana Ivanovic. 2010.Hubs in space: Popular nearest neighbors in high-dimensional data .J. Mach. Learn. Res., 11:2487–2531. Llama Team. 2024.The llama 3 herd of models.CoRR, abs/2407.21783. Ioannis Tsiamas, David Dale, and Marta R. Costa-jussà. 2025.Improving language and modality transfer in translation by character-level modeling . InProceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages 20171–20187. Association for Com- putational Linguistics. Tongzhou Wang and Phillip Isola. 2020.Understand- ing contrastive representation learning through align- ment and uniformity on the hypersphere . InPro- ceedingsofthe37thInternationalConferenceonMa- chine Learning, ICML 2020, 13-18 July 2020, Vir- tual Event, Proceedings of Machine Learning Re- search, pages 9929–9939. PMLR. Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing, Heng Yu, and Weihua Luo. 2021.On learning uni- versal representations across languages. In9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. A A note on segmentation We define words as whitespace-delimited units of text. For languages that do not use whites- pace segmentation—namely scriptsHans, Hant, Jpan, Thai, Laoo, Khmr, Mymr, Tibtfollowing the SONAR notation—we treat tokeniser tokens as words. When computing word embeddings, we ob- tain them by averaging the embeddings of the to- kens that compose each word. B Spans extraction B.1 Extracting spans with an LLM For the LLM-based method, we use LLaMA-70B Instruct, quantised to 8-bit precision using Bitsand- bytes. The prompt used for span extraction is pro- vided in Figure9. The generated spans are subsequently mapped back to SONAR tokens. In the vast majority of cases, this mapping is unambiguous; when a SONAR token is split across multiple spans, we assign it to the span with the greatest string over- lap. B.2 Statistics on extracted spans Table14reports summary statistics of the extracted spans. We observe that span lengths vary across languages, with CASE tending to extract longer spans in both source and target sides. In particular, Chinese and Hindi exhibit relatively longer spans compared to other languages, often involving a larger number of target tokens. Using spaCy’s en_core_web_smmodel to analyse POS tags on the English side, we find that nouns are most fre- quent, followed by adpositions. We also observe variation in POS tag distributions across languages and span sources; further details are provided in Ta- ble 15. C Experiment details Pre-processingWe apply exactly the same pre- processing and normalisation as SONAR, namely the scripthttps://github.com/facebookresearch/ stopes/blob/main/stopes/pipelines/monolingual/ utils/text_normalizer.py C.1 Languages studied Table 16shows the language splits used for train- ing and evaluation. Training languages are drawn from the NLLB Primary dataset, while the test languages correspond to those covered by our cross-lingual token-level benchmark, including Word Alignment, PAN-X, MASSIVE, UDPOS, and WiC. For sentence retrieval, we use the 57 test languages; we only evaluate the MTEB tasks on English. C.2 Alignment datasets We evaluate a total of 18 language pairs. As gold-standard alignments, we use the XL-WA dataset (Martelli et al.,2023) for 10 language pairs; we were unable to obtain the four additional sub- sets. For en-hi, en-fr, en-fa, and en-cs, we adopt the same datasets as inJalili Sabet et al.(2020). For en-zh, en-ro, and en-de, we use the datasets in- troduced by Dou and Neubig(2021), and for en-ja we use the KFTT dataset (Neubig,2011). C.3 Sequence tagging evaluation We use up to 10,000 training examples and 5,000 test examples per language, doubling these limits for WiC, while filtering out sequences exceeding 514 tokens. A single linear classifier (Linear(dim, n_classes)) is trained on frozen encoder embed- dings using Adam with a learning rate of10 −3 , cross-entropy loss, and a batch size of 4096. We train with early stopping (patience= 10) based on a 90/10 train-validation split. All embeddings are L2-normalised before classification. C.4 Losses implementations details We refer readers to the original papers for de- tailed descriptions of the losses and their motiva- tion. Here, we describe implementation details and hyperparameters used in our setup. SO LossWe followOmnilingual SONAR Team et al. (2026) and introduce a temperature hyperpa- rameterτ= 500. We use Argmax (Jalili Sabet et al.,2020) for word alignment extraction. WordOTWe useτ= 0.05, following the origi- nal paper. WACSEWe use Argmax (Jalili Sabet et al., 2020) for word alignment extraction (WTR, AWP losses). For AWP, we only use one mask per sen- tence pair. We use the token embedding (encoding) matrix as the MLM head, and use a cross-entropy loss. SALTWe useτ= 5.0in our contrastive span loss. C.5 Data sampling during training Training uses dynamically constructed multilin- gual mini-batches under a fixed token budget. At each step, sentence pairs are sampled from source– target language distributions using temperature- smoothed probabilities over language pairs. Each pair is tokenised with the corresponding language- specific encoder, and any example exceeding the maximum sequence length (512) on either side is discarded. A buffered packing strategy is then used to im- prove batch efficiency. Examples are first ordered by sequence cost, defined as max(|x|,|y|), where |x|and|y|denote the number of tokens in the source and target sentences. This reduces padding and increases the proportion of useful tokens per update. Optimisation follows the setup described in the main paper, using AdamW with a linear warm- up over the first 1,000 steps and cosine annealing thereafter, decaying the learning rate smoothly to zero over the rest of training. D Other results D.1 Training dynamics We show the training dynamics in terms of se- quence tagging and Word Alignment performance of the different losses studied in Figure 8. D.2 Sequence tagging results In this section, we provide an expanded analysis of the sequence tagging results. We evaluate sequence tagging under three differ- ent training settings: •English: the classifier is trained using only English data. •All: the classifier is trained jointly on data from all languages. •Single: for each target language, the classifier is trained exclusively on data from that lan- guage. Across all three settings, SALT consistently out- performs SONAR (Figure 5, Table7); Figure12 and13show the performance per language on the English setting). Notably, we observe that UD- POS exhibits limited cross-lingual transfer: train- ing on a single language yields substantially better performance than multilingual training (Singlevs. All), suggesting that the task depends strongly on language-specific characteristics. D.3 Language generalisation We evaluate the generalisation ability of our method on held-out test languages unseen during training. Results are consistent, with SALT outper- forming the other losses on these languages (see Ta- ble 8) except on UDPOS. Importantly, the training dynamics (Figure6) show that held-out languages benefit from SALT, particularly at earlier training stages, whereas trained languages continue to im- prove over a longer training horizon. E Analysis E.1 Word embeddings alignment Figure7shows the cosine similarity of spans and words. We use high-quality reference spans from CrossSpan and split them into words or multi-word spans based on length. Note that this applies only to this figure; in other experiments, spans may also consist of single words. E.2 Geometry analysis For word retrieval, we use our word alignment dataset (AppendixC.2): first, we aggregate word embeddings by averaging token embeddings (given space segmentation). Second, for every En- glish word that has a defined target, we find the 5 nearest neighbours using cosine similarity. We run the remainder of our experiments for geometry analysis on the FLORES-80 devtest dataset. We evaluate language identification with a lin- ear probe on L2-normalised FLORES word em- beddings. For each model, we train a single n.Linear(1024, 80)classifier to predict the lan- guage from individual word embeddings, using the first 500 FLORES sentences per language; 10% of these sentences are held out for validation and the remaining saved sentences are used for test. Train- ing uses AdamW with learning rate 10e-3, batch size 8192, weight decay 0 and early stopping after 50 epochs without validation macro-F1 improve- ment. Duplicate training words are removed per language. For isotropy and hubness, we sample 10,000 ran- dom pairs of different sentences in different lan- guages. For hubness, we usek= 10nearest neigh- bours. PAN-XMassiveUDPOSWiC 0.4 0.5 0.6 0.7 0.8 F1 English SONAR English SALT Single SONAR Single SALT All SONAR All SALT Figure 5: Performance of SALT vs SONAR on the sequence tagging tasks, on three training modes for the probe: only English data, all the data and data for every test language. SALT improves all the results. Task SALT (single) SALT (English) SALT (all) SONAR (single) SONAR (English) SONAR (all) Massive0.550.450.560.540.400.54 PAN-X0.790.630.730.780.580.70 UDPOS0.760.570.620.740.550.60 WiC0.600.590.590.590.580.58 Table 7: F1 scores for the sequence tagging tasks on the three training settings. 500010000150002000025000 Training step 0.588 0.590 0.592 0.594 0.596 0.598 Seq tagging F1 ( ↑ ) Trained 500010000150002000025000 Training step 0.5250 0.5275 0.5300 0.5325 0.5350 0.5375 0.5400 Seq tagging F1 ( ↑ ) Heldout Figure 6: Sequence tagging performance throughout SALT training on trained (left) and held-out (right) languages. PAN-X Massive UDPOS SONAR Loss0.5620.4110.530 SO Loss0.5770.3950.549 WordOT0.5860.4100.535 WACSE0.5820.4000.538 OmniSONAR-Token 0.5900.4020.548 SALT0.598 0.4420.544 Table 8: Sequence tagging performance on test lan- guages not seen during training. 812162024 Layer 0.8 Similarity ( ↑ ) SALT Span SALT Word SONAR Span SONAR Word Figure 7: Word and span similarity for SONAR and SALT across layers. E.3 Qualitative analysis We take the sentence Most insects have the advantage of being able to fold their wings back along the body. and perform word-level retrieval (top-5 nearest neighbours) across sentences in 80 languages (see Table 9,10,11) for the full FLORES-200 devtest. Overall, all three encoders are generally able to retrieve semantically equivalent words in other languages from the same sentence. However, SALT more frequently retrieves tokens in non- Latin scripts and, more importantly, places true translations closer in terms of absolute cosine sim- ilarity. This is notable given our earlier geometric analysis, in which we observed that SALT yields a lower average cosine similarity across the embed- ding space than the other encoders. Another improvement is that SALT reduces a common failure mode in which the model retrieves the same word in the same language from different sentential contexts. This suggests that SALT better preserves contextual meaning, rather than relying on surface-form similarity. MEXMA Rank ScoreLang Unit 1 0.945 hun_Latn legtobb 2 0.943 heb_Hebr rov 3 0.943 nno_Latn fleste 4 0.942fra_Latn plupart 5 0.942 ukr_Cyrl bilshosti SALT Rank ScoreLang Unit 1 0.988 ckb_Arab zorbey 2 0.987 mar_Deva bahutek 3 0.986 arz_Arab mu’zem 4 0.986 npi_Deva adhikansh 5 0.984bul_Cyrl povecheto SONAR Rank ScoreLang Unit 1 0.976fra_Latn plupart 2 0.971 por_Latn maioria 3 0.971 kor_Hang daebubun-ui 4 0.966 spa_Latn mayoria 5 0.965cat_Latn majoria Table 9: Nearest neighbours across models for the query wordmost(all non-Latin scripts transliterated). F Standard deviations Table 12shows the standard deviation of our re- sults from Table 2. G Human evaluation of spans For each of Chinese, French, Hindi, Russian, and Spanish, a native speaker annotated spans extracted from a total of 2,361 sentence pairs generated by either LLaMA-70B or CASE (with SONAR), without being informed of the originat- ing system. The annotation split per language and system can be seen in Table 13. Annotators were presented with pairs of spans and asked to assess their semantic equivalence using a 4-point scale (see AppendixG.2). Further details on the recruit- ment process are provided in Appendix G.3. G.1 Results on human evaluation of extracted aligned spans Table13shows the score distribution of spans fol- lowing human evaluation. We observe that, in almost all languages, spans of acceptable quality (Rating 3 or 4) occur at a rate higher than 90%. These results validate using spans as a training sig- nal. We also see that SONAR performance with CASE closely resembles LLM annotation. 0500010000150002000025000 Training step 0.530 0.535 0.540 0.545 0.550 0.555 Seq tagging F1 ( ↑ ) SONAR Loss SO Loss WordOT WACSE OmniSONAR-Token L CON Span + L INT SALT 0500010000150002000025000 Training step 0.170 0.175 0.180 0.185 0.190 0.195 0.200 0.205 AER ( ↓ ) Figure 8: Sequence tagging and word alignment performance for different training losses. In the word alignment plot, we omit WordOT to improve visualisation (WordOT is poorly aligned). G.2 Full instructions given to annotators Figure10shows the annotation guidelines. G.3 Recruitment and demographics Annotators were recruited through internal com- munication channels within our institution. All an- notators were residing in the country where the in- stitution is based at the time of the study. The an- notator pool comprised five individuals, each a na- tive speaker of one of the following languages: Chi- nese, French, Hindi, Russian and Spanish. Prior to the annotation process, all annotators participated in a training session in which we intro- duced the annotation interface, explained the evalu- ation rubric in detail, and jointly reviewed a set of example instances to ensure consistency in inter- pretation. Annotators were compensated at a rate of $20.5 per hour. G.4 Participant Information Sheet Figure11shows the Participant Information Sheet that was shared with the annotators. H Compute and model size SONAR has 766M parameters, and SALT intro- duces no additional parameters. We developed SALT using two NVIDIA A100 GPUs (80 GB each). Final training runs were conducted on two NVIDIA H200 GPUs (90 GB each) and required approximately two and a half days. I Use of artifacts We use the SONAR encoder and the open-source subset of the NLLB Primary dataset, which are distributed under permissive licenses (Creative Commons Attribution Non-Commercial 4.0 and MIT License, respectively). These resources are used in accordance with their intended purpose of advancing research on multilingual sentence representations. All other encoders and datasets are used solely for evaluation, respecting their re- spective licenses and intended uses. We do not apply additional filtering for offensive content or personally identifiable information (PII), as such filtering was already performed by the orig- inal dataset creators ( NLLB Team,2024). J Usage of AI tools We acknowledge using AI tools for grammar cor- rection and some other language clarifications, as well as code writing assistance. MEXMA Rank ScoreLang Unit 1 0.950 ron_Latn insecte 2 0.940 afr_Latn insekte 3 0.940 spa_Latn insectos 4 0.939 swe_Latn insekter 5 0.939 epo_Latn insektoj SALT Rank ScoreLang Unit 1 0.978afr_Latn insekte 2 0.970 arz_Arab al-hasharat (al-ḥasharāt) 3 0.968slv_Latn insektov 4 0.967 mkd_Cyrl insekti 5 0.967slk_Latn hmyzu SONAR Rank ScoreLang Unit 1 0.970 epo_Latn insektoj 2 0.964 fra_Latn insectes 3 0.960 nob_Latn insekter 4 0.954 afr_Latn insekte 5 0.954 deu_Latn Insekten Table 10: Nearest neighbours for the query wordinsect (all non-Latin scripts transliterated). MEXMA Rank Score LangUnit Sent. idx 1 0.949 eng_Latn fold621 2 0.872 tha_Thai a619 3 0.873 tha_Thai ph619 4 0.873 tha_Thai b619 5 0.849 tha_Thai a621 SALT Rank Score LangUnitSent. idx 1 0.956 ind_Latn melipat619 2 0.946 epo_Latn faldi619 3 0.945 ron_Latn plieze619 4 0.937 vie_Latn gap619 5 0.936 bul_Cyrl sgavat619 SONAR RankScore LangUnitSent. idx 1 0.915271 epo_Latn faldi619 2 0.901712 kor_Hang jeobeul619 3 0.874322 heb_Hebr rov619 4 0.874098 eng_Latn fold621 5 0.872106 ind_Latn melipat619 Table 11: Nearest neighbours across models for the query wordfold(all non-Latin scripts transliterated). Note that MEXMA and SONAR retrieve words from other sentences. AER PAN-X Massive UDPOS WiC SONAR Loss0.000255 0.000120 0.000622 0.000735 0.000361 SO Loss0.000318 0.000311 0.001860 0.000445 0.000064 WordOT0.002277 0.000516 0.001676 0.000156 0.000672 WACSE0.000127 0.001146 0.000714 0.000113 0.001513 OmniSONAR-Token 0.000068 0.000549 0.000630 0.000938 0.002037 SALT0.000127 0.000456 0.001210 0.000677 0.000210 Table 12: Standard deviation of our results in Table2(we run training experiments in three seeds). LanguageSource#Annotations Rating 1 Rating 2 Rating 3 Rating 4 Agg. 3+4 SpanishLLaMA-70B1290.0%3.1%0.8%96.1%96.9% Spanish SONAR (CASE)2490.4%2.0%5.6%92.0%97.6% RussianLLaMA-70B1020.0%7.8%20.6%71.6%92.2% Russian SONAR (CASE)2043.4%8.3%21.1%67.2%88.3% HindiLLaMA-70B3343.9%7.2%15.3%73.7%89.0% HindiSONAR (CASE)3340.9%3.3%19.5%76.3%95.8% FrenchLLaMA-70B1350.0%5.2%11.1%83.7%94.8% FrenchSONAR (CASE)2640.4%8.0%21.6%70.1%91.7% ChineseLLaMA-70B2801.8%5.4%13.2%79.6%92.8% Chinese SONAR (CASE)3301.5%3.6%12.7%82.1%94.8% Table 13: Score distribution for the human evaluation of spans. Lang PairSourceen #words (avg) en #words (median) X #words (avg) X #words (median) English-ChineseLLaMA-70B2.612.04.173.5 English-Chinese SONAR (CASE)5.703.08.425.5 English-FrenchLLaMA-70B2.532.03.133.0 English-French SONAR (CASE)2.582.03.222.0 English-HindiLLaMA-70B2.722.04.984.0 English-Hindi SONAR (CASE)2.822.05.704.0 English-RussianLLaMA-70B2.432.02.302.0 English-Russian SONAR (CASE)2.632.02.442.0 English-SpanishLLaMA-70B2.642.03.153.0 English-Spanish SONAR (CASE)2.432.02.592.0 All English-XXLLaMA-70B2.622.03.973.0 All English-X SONAR (CASE)3.362.04.833.0 Table 14: Average and median span word length. Lang PairSource1st POS 1st POS freq 2nd POS 2nd POS freq 3rd POS 3rd POS freq English-ChineseLLaMA-70B NOUN55.4ADP31.8VERB26.8 English-Chinese SONAR (CASE) NOUN71.2ADP51.2VERB38.5 English-FrenchLLaMA-70B NOUN54.1VERB31.1ADP26.7 English-French SONAR (CASE) NOUN49.6ADP25.8VERB23.9 English-HindiLLaMA-70B NOUN48.8ADP35.9VERB29.6 English-Hindi SONAR (CASE) NOUN50.0DET32.6 PROPN28.4 English-RussianLLaMA-70B NOUN47.1VERB38.2DET25.5 English-Russian SONAR (CASE) NOUN56.9DET27.9PRON24.0 English-SpanishLLaMA-70B NOUN57.4ADP29.5ADJ27.1 English-Spanish SONAR (CASE) NOUN50.2VERB26.9DET26.1 All English-XXLLaMA-70B NOUN52.3ADP31.1VERB29.5 All English-X SONAR (CASE) NOUN56.0ADP31.3DET29.8 Table 15: Most common POS tags of English spans. Prompt for LLM span alignment You are tasked with extracting aligned spans. You will be given two sentences, and need to identify spans of words/particles from the first one that appear in the second. Copy the spans of words as they appear originally in both texts. Only copy the extracted aligned spans, following the format of Example 1. If there are hallucinations or omissions, don’t include them in the aligned spans. Example 1: Sentence 1 [eng]: Samples of body fluids and tissues from people with the disease should be handled with special caution . Sentence 2 [spa]: Las muestras de tejidos y fluidos corporales de personas con la enfermedad deben manejarse con especial precaución . Aligned spans: [Samples] - [Las muestras] [of body fluids] - [de fluidos corporales] [and tissues] - [y tejidos] [from people] - [de personas] [with the disease] - [con la enfermedad] [should be handled] - [deben manejarse] [with special caution] - [con especial precaución] Now, extract the alignments for the following pair: Sentence 1 [src_lang]: src_text Sentence 2 [tgt_lang]: tgt_text Aligned spans: Figure 9: Prompt that was used on LLaMA-70B for span alignment. Annotation instructions Purpose: evaluate the quality of automatically generated alignments between words or short phrases across different languages. Why you were invited: you are fluent in Hindi, Russian, French, Chinese, or Spanish. Participation: one remote session (about 120 minutes), voluntary, and withdrawal is possible up to two months after participation. Data collected: Alignment judgments, response times, an anonymized annotator ID Explanation of Ratings Rating 1 - Completely Misaligned The spans are unrelated in meaning and do not correspond to each other. There is no meaningful semantic overlap. Example English: “These symptoms have been reported and [might] contribute to its rapid spreading” Spanish: “Se han informado de esos síntomas, [lo que] podría contribuir a su rápida propagación” Literal translation: “These symptoms have been reported, [what] may contribute to its rapid spreading” Explanation: The spans (“might” vs. “lo que”) do not align in meaning or function. Rating 2 - Partial Overlap, Core Meaning MismatchThe spans share some overlap, but fail to capture the main meaning or function of each other. Example 1 English: “[Since cases] have so far clustered in certain regions in EU/EEA countries” Spanish: “[Dado que, hasta el momento, se han formado conglomerados de casos] en ciertas regiones de países de la UE/E” Literal translation: “[Since so far there have been clusters of cases] in certain regions of countries” Explanation: There is overlap, but the English span (“Since cases”) is much shorter and does not capture the full meaning expressed in Spanish. Example 2 English: “What [are the implications] for public health practice?” Spanish: “¿Qué [consecuencias] tiene esto para la práctica de la salud pública?” Literal translation: “What [consequences] this has for public health practice?” Explanation: The Spanish span omits an important part of the meaning (the verbal structure “are / has”), so the alignment is incomplete. Rating 3 - Same Role, Minor Differences The spans fulfill the same semantic role and largely match in meaning, but there are small differences (e.g., missing words, slight shifts in nuance, or grammatical variation). Example 1 English: “The virus is transmitted through respiratory droplets and [through objects].” Spanish: “El virus se transmite por medio de las gotitas de la respiración y [los objetos].” Literal translation: “The virus is transmitted through respiratory droplets and [the objects].” Explanation: The meaning is almost identical, but the Spanish span lacks the preposition “through”. Example 2 English: “Cabin crew should wear disposable medical gloves, [and possibly additional] personal protective equipment” Spanish: “La tripulación de cabina debe usar guantes quirúrgicos desechables, [y, quizás,] equipos de protec- ción individual” Literal translation: “Cabin crew should wear disposable medical gloves, [and maybe] personal protective equipment” Explanation: The meaning is very similar, but “additional” is not expressed in Spanish. Rating 4 - Fully Aligned (Equivalent Meaning The spans have the same meaning with no important differences. They could be swapped between sentences without changing the meaning. Example 1 English: “[We are not aware] of any increased risk” Spanish: “[No tenemos conocimiento] de un aumento de riesgo” Literal translation: “[(We) do not have knowledge] of any increase in risk” Example 2 English: “Particularly [avoid] cruise ship travel.” Spanish: “En especial, [se deben evitar] los viajes en cruceros.” Literal translation: “In particular, [they must be avoided] cruise ship travel” Explanation: The spans are fully equivalent in meaning despite minor grammatical differences. Figure 10: Instructions shared with the annotators. Participant Information Sheet (PIS) What is the purpose of the study? The purpose of this study is to evaluate the quality of automatically generated alignments between words or short phrases across different languages (for example, “the dog” – “el perro”). These alignments are used in research on multilingual language technologies. By asking native speakers to judge whether these alignments are correct, we aim to better understand how accurate these automatic methods are and improve techniques for learning cross-lingual language representations used in natural language processing systems. Why have I been asked to take part? You are a native speaker of the studied language (Russian, French, Chinese, Hindi or Spanish). Do I have to take part? No – participation in this study is entirely up to you. You can withdraw from the study at any time, up to 2 months, without giving a reason. After this point, it will no longer be possible to withdraw because we are not collecting any data that would allow us to identify you. What will happen if I decide to take part? •If you decide to take part, you will complete a short annotation task in which you review pairs of words or short phrases from two different languages (for example, “the dog” – “el perro”). You will be asked to indicate whether the alignment between the spans appears correct. •The data collected will consist only of your judgments about the alignments and an anonymised annotator ID. The task will be completed through a simple online questionnaire or annotation interface. •Participation will involve a single session lasting approximately 120 minutes. The task can be completed remotely at a time convenient for you. Compensation. Participants will receive financial compensation at a minimum rate of $20.5 per hour for their time. Are there any risks associated with taking part? There are no significant risks associated with participation. What data are you collecting about me? The data we collect for our research is completely anonymous: We are not collecting any information that could, in our assessment, allow anyone to identify you. Your signed participant consent form will be kept separately from your responses and destroyed two months after your participation. What will happen to the results of this study? The results of this study may be summarised in published articles, reports and presentations. Your anonymised data may be published and can also be used for future research. Who can I contact? If you have any further questions about the study, please contact the lead researcher, [PLACEHOLDER]. If you wish to make a complaint about the study, please contact. [PLACEHOLDER]. When you contact us, please provide the study title and detail the nature of your complaint. Alternative formats. To request this document in an alternative format, such as large print or on coloured paper, please contact [PLACEHOLDER]. Figure 11: Participant Information Sheet that was shared with the annotators. 0.00.10.20.30.40.50.60.70.8 F1 tha_Thai mya_Mymr jpn_Jpan zho_Hans kaz_Cyrl urd_Arab kor_Hang pes_Arab arb_Arab heb_Hebr rus_Cyrl tel_Telu jav_Latn tam_Taml afr_Latn vie_Latn hun_Latn eus_Latn mal_Mlym fin_Latn tur_Latn mar_Deva ell_Grek kat_Geor est_Latn deu_Latn spa_Latn ind_Latn ita_Latn zsm_Latn fra_Latn yor_Latn swh_Latn bul_Cyrl hin_Deva por_Latn nld_Latn tgl_Latn ben_Beng eng_Latn PAN-X SALTSONAR 0.00.10.20.30.40.50.6 F1 amh_Ethi khm_Khmr khk_Cyrl azj_Latn mya_Mymr hye_Armn swh_Latn tam_Taml jav_Latn zho_Hans jpn_Jpan mal_Mlym tha_Thai isl_Latn kor_Hang cym_Latn lvs_Latn kat_Geor arb_Arab kan_Knda cat_Latn tur_Latn ben_Beng heb_Hebr fra_Latn als_Latn pol_Latn slv_Latn urd_Arab rus_Cyrl pes_Arab tel_Telu fin_Latn vie_Latn hun_Latn ita_Latn ind_Latn tgl_Latn zsm_Latn deu_Latn nob_Latn ron_Latn swe_Latn hin_Deva ell_Grek spa_Latn por_Latn nld_Latn afr_Latn dan_Latn eng_Latn Massive 0.00.20.40.60.8 F1 vie_Latn jpn_Jpan kor_Hang tha_Thai ell_Grek arb_Arab zho_Hans yor_Latn tam_Taml mar_Deva tur_Latn tel_Telu hun_Latn eus_Latn urd_Arab pes_Arab kaz_Cyrl ind_Latn rus_Cyrl heb_Hebr hin_Deva fin_Latn bul_Cyrl spa_Latn est_Latn por_Latn tgl_Latn fra_Latn ita_Latn afr_Latn deu_Latn nld_Latn eng_Latn UDPOS 0.00.10.20.30.40.50.6 F1 zho_Hans arb_Arab eng_Latn fra_Latn rus_Cyrl WiC Figure 12: Performance of SALT vs SONAR on the sequence tagging tasks PAN-X and Massive, on the English setting. 0.00.10.20.30.40.50.60.70.8 F1 tha_Thai mya_Mymr jpn_Jpan zho_Hans kaz_Cyrl urd_Arab kor_Hang pes_Arab arb_Arab heb_Hebr rus_Cyrl tel_Telu jav_Latn tam_Taml afr_Latn vie_Latn hun_Latn eus_Latn mal_Mlym fin_Latn tur_Latn mar_Deva ell_Grek kat_Geor est_Latn deu_Latn spa_Latn ind_Latn ita_Latn zsm_Latn fra_Latn yor_Latn swh_Latn bul_Cyrl hin_Deva por_Latn nld_Latn tgl_Latn ben_Beng eng_Latn PAN-X SALTSONAR 0.00.10.20.30.40.50.6 F1 amh_Ethi khm_Khmr khk_Cyrl azj_Latn mya_Mymr hye_Armn swh_Latn tam_Taml jav_Latn zho_Hans jpn_Jpan mal_Mlym tha_Thai isl_Latn kor_Hang cym_Latn lvs_Latn kat_Geor arb_Arab kan_Knda cat_Latn tur_Latn ben_Beng heb_Hebr fra_Latn als_Latn pol_Latn slv_Latn urd_Arab rus_Cyrl pes_Arab tel_Telu fin_Latn vie_Latn hun_Latn ita_Latn ind_Latn tgl_Latn zsm_Latn deu_Latn nob_Latn ron_Latn swe_Latn hin_Deva ell_Grek spa_Latn por_Latn nld_Latn afr_Latn dan_Latn eng_Latn Massive 0.00.20.40.60.8 F1 vie_Latn jpn_Jpan kor_Hang tha_Thai ell_Grek arb_Arab zho_Hans yor_Latn tam_Taml mar_Deva tur_Latn tel_Telu hun_Latn eus_Latn urd_Arab pes_Arab kaz_Cyrl ind_Latn rus_Cyrl heb_Hebr hin_Deva fin_Latn bul_Cyrl spa_Latn est_Latn por_Latn tgl_Latn fra_Latn ita_Latn afr_Latn deu_Latn nld_Latn eng_Latn UDPOS 0.00.10.20.30.40.50.6 F1 zho_Hans arb_Arab eng_Latn fra_Latn rus_Cyrl WiC Figure 13: Performance of SALT vs SONAR on the sequence tagging tasks UDPOS and WiC, on the English setting. Table 16: Full list of languages used. Train(93 languages) ace_Arabace_Latnafr_Latnaka_Latn amh_Ethiarb_Arabary_Arabarz_Arab bam_Latnban_Latnben_Bengbho_Deva bjn_Arabbjn_Latnbos_Latnbug_Latn bul_Cyrlcrh_Latndik_Latndzo_Tibt eng_Latnewe_Latnfon_Latnfra_Latn fur_Latnfuv_Latngaz_Latngrn_Latn guj_Gujrhat_Latnhin_Devahne_Deva hrv_Latnind_Latnkan_Kndakas_Arab kas_Devakhm_Khmrkin_Latnknc_Arab knc_Latnlij_Latnlim_Latnlin_Latn lmo_Latnltg_Latnlug_Latnmag_Deva mal_Mlymmar_Devamin_Latnmkd_Cyrl mni_Bengmri_Latn mya_Mymrnno_Latn nob_Latnnus_Latnory_Oryapan_Guru pbt_Arabpes_Arabpor_Latnprs_Arab rus_Cyrlscn_Latnshn_Mymrslv_Latn som_Latnsot_Latnspa_Latnsrd_Latn srp_Cyrlssw_Latnswh_Latnszl_Latn tam_Tamltaq_Latntaq_Tfngtel_Telu tgl_Latntir_Ethitsn_Latntso_Latn tzm_Tfngukr_Cyrlurd_Arabvec_Latn xho_Latnyor_Latnzho_Hanszsm_Latn zul_Latn Test(57 languages) afr_Latnals_Latnamh_Ethiarb_Arab azj_Latnben_Bengbul_Cyrlcat_Latn cym_Latndan_Latndeu_Latnell_Grek eng_Latnest_Latneus_Latnfin_Latn fra_Latnheb_Hebrhin_Devahun_Latn hye_Armnind_Latnisl_Latnita_Latn jav_Latnjpn_Jpankan_Kndakat_Geor kaz_Cyrlkhk_Cyrlkhm_Khmrkor_Hang lvs_Latnmal_Mlym mar_Devamya_Mymr nld_Latnnob_Latnpes_Arabpol_Latn por_Latnron_Latnrus_Cyrlslv_Latn spa_Latnswe_Latnswh_Latntam_Taml tel_Telutgl_Latntha_Thaitur_Latn urd_Arabvie_Latnyor_Latnzho_Hans zsm_Latn Test (29 languages) als_Latnazj_Latncat_Latncym_Latn dan_Latndeu_Latnell_Grekest_Latn eus_Latnfin_Latnheb_Hebrhun_Latn hye_Armnisl_Latnita_Latnjav_Latn jpn_Jpankat_Georkaz_Cyrlkhk_Cyrl kor_Hanglvs_Latnnld_Latnpol_Latn ron_Latnswe_Latntha_Thai tur_Latn vie_Latn

Instant research discovery