Paper deep dive

Unified Vision-Language Modeling via Concept Space Alignment

Yifu Qiu, Paul-Ambroise Duquenne, Holger Schwenk

Year: 2026Venue: arXiv preprintArea: cs.CVType: PreprintEmbeddings: 83

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 7/20/2026, 2:49:47 AM

Summary

The paper introduces V-SONAR, a vision-language embedding space extending the text-only SONAR space to support image and video modalities. It employs a post-hoc alignment pipeline using a coarse-to-fine curriculum to map Perception Encoder representations into the SONAR semantic space. The authors demonstrate that V-SONAR achieves state-of-the-art performance in zero-shot text-to-video retrieval and captioning. Furthermore, they introduce V-LCM, an extension of the Large Concept Model (LCM) that processes unified vision-language inputs via V-SONAR and SONAR, showing significant multilingual advantages over existing vision-language models.

Entities (22)

V-SONAR · model · 98%V-SONAR · model · 98%SONAR · model · 97%V-LCM · model · 97%LCM · model · 96%V-LCM · model · 96%Perception Encoder · model · 95%SONAR · model · 95%LCM · model · 94%OmniSONAR · model · 94%DREAM-1K · dataset · 92%OmniSONAR · model · 92%DREAM-1K · dataset · 90%Perception Encoder · model · 90%PE-Video · dataset · 90%M3IT · dataset · 88%SigLIP2 · model · 88%InternVL · model · 85%InternVL · model · 85%Qwen-VL · model · 85%Qwen-VL · model · 85%SigLIP2 · model · 85%

Relation Signals (17)

V-LCM → extends → LCM

confidence 96% · Finally, we introduce v-LCM, which extends the LCM with vision–language instruction tuning.

V-LCM → extends → LCM

confidence 95% · we introduce V-LCM, which extends the LCM with vision–language instruction tuning

V-SONAR → extends → SONAR

confidence 95% · We introduce v-Sonar, a vision–language embedding space extended from the text-only embedding space SONAR

V-SONAR → extends → SONAR

confidence 95% · V-SONAR, a vision-language embedding space extended from the text-only embedding space SONAR

V-SONAR → usesbaseencoder → Perception Encoder

confidence 95% · we use teacher-student training ... to align the representations of a state-of-the-art vision encoder, Perception Encoder

V-LCM → usesembeddingspace → V-SONAR

confidence 94% · v-LCM encodes vision and language inputs into an unified sequence of latent embeddings via v-Sonar and SONAR

V-SONAR → usesalignmentmethod → post-hoc alignment pipeline

confidence 93% · To construct v-Sonar, we propose a post-hoc alignment pipeline that maps the representations of an existing vision encoder into the SONAR space.

V-LCM → evaluatedon → M3IT

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We introduce V-SONAR, a vision-language embedding space extended from the text-only embedding space SONAR (Omnilingual Embeddings Team et al., 2026), which supports 1500 text languages and 177 speech languages. To construct V-SONAR, we propose a post-hoc alignment pipeline that maps the representations of an existing vision encoder into the SONAR space. We thoroughly evaluate V-SONAR and show that its embeddings achieve competitive performance on text-to-video retrieval. Equipped with the OMNISONAR text decoder, V-SONAR further surpasses state-of-the-art vision-language models on video captioning tasks, including DREAM-1K (BLEU 23.9 vs. 19.6) and PE-VIDEO (BLEU 39.0 vs. 30.0). Leveraging V-SONAR, we first demonstrate that the Large Concept Model (LCM; LCM team et al. 2024) operating in SONAR and trained with English text only, can perform both single- and multi-visual concept understanding in a zero-shot manner. Finally, we introduce V-LCM, which extends the LCM with vision-language instruction tuning. V-LCM encodes vision and language inputs into an unified sequence of latent embeddings via V-SONAR and SONAR, and it is trained with the same latent diffusion objective for next-embedding prediction as in LCM's text-only pre-training. Experiments on a large-scale multilingual and -modal instruction-tuning data mixture highlight the potential of V-LCM: V-LCM matches state-of-the-art vision-language models on tasks covering image/video captioning and question answering, while significantly outperforming them across 61 rich- to low-resource languages out of all 62 tested languages.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

81,381 characters extracted from source content.

Expand or collapse full text

Unified Vision–Language Modeling via Concept Space Alignment Yifu Qiu 1,2,∗ , Paul-Ambroise Duquenne 2 , Holger Schwenk 2 1 University of Edinburgh, 2 FAIR at Meta ∗ This work was done during an internship of Yifu Qiu at FAIR at Meta We introducev-Sonar, a vision–language embedding space extended from the text-only embedding space Sonar (Omnilingual Embeddings Team et al., 2026), which supports 1500 text languages and 177 speech languages. To constructv-Sonar, we propose a post-hoc alignment pipeline that maps the representations of an existing vision encoder into the Sonar space. We thoroughly evaluatev-Sonar and show that its embeddings achieve competitive performance on text-to-video retrieval. Equipped with the OmniSONAR text decoder,v-Sonarfurther surpasses state-of-the-art vision–language models on video captioning tasks, including Dream-1k (Bleu 23.9 vs. 19.6) and PE-Video (Bleu 39.0 vs. 30.0). Leveragingv-Sonar, we first demonstrate that the Large Concept Model (LCM; LCM team et al. 2024) operating in Sonar and trained with English text only, can perform both single- and multi- visual concept understanding in a zero-shot manner. Finally, we introduce v-LCM, which extends the LCM with vision–language instruction tuning. v-LCM encodes vision and language inputs into an unified sequence of latent embeddings viav-Sonarand Sonar, and it is trained with the same latent diffusion objective for next-embedding prediction as in LCM’s text-only pre-training. Experiments on a large-scale multilingual and -modal instruction–tuning data mixture highlight the potential of v-LCM: v-LCM matches state-of-the-art vision-language models on tasks covering image/video captioning and question answering, while significantly outperforming them across 61 rich- to low-resource languages out of all 62 tested languages. Date: March 3, 2026 Correspondence: Holger Schwenk at schwenk@meta.com 1 Introduction Language- and modality-agnostic embedding spaces have emerged as a powerful paradigm for multilingual and –modal representation learning (Artetxe and Schwenk, 2019; Feng et al., 2020; Ni et al., 2021; Duquenne et al., 2023; Wang et al., 2024b; Chen et al., 2024a). Such spaces have achieved state-of-the-art performance across a wide range of applications, e.g., bitext mining (Schwenk et al., 2019; NLLB Team et al., 2022; Ramesh et al., 2022), and speech–text mining (Duquenne et al., 2021b). Beyond these, embedding spaces with the encoder–decoder architecture such as SONAR (Duquenne et al., 2023) have further enabled generative modeling directly in the latent embedding space. The Large Concept Model (LCM; LCM team et al. 2024) extends this direction by showing that diffusion-based language modeling can operate directly in the language- agnostic latent space, i.e., over continuous embeddings rather than discrete tokens. Despite these advances, existing embedding spaces remain restricted to text and speech, limiting their potential for vision–language tasks. In this work, we introducev-Sonar, which extends OmniSONAR (Omnilingual Embeddings Team et al., 2026) to the image and video modality. To the best of our knowledge, this makes OmniSONAR the most universal embedding space covering four modalities 1 and up to 1500 languages. We use teacher-student training (Reimers and Gurevych, 2020; Duquenne et al., 2021a; Heffernan et al., 2022) to align the representations of a state-of-the-art vision encoder, Perception Encoder (Bolya et al., 2025), with Sonar’s semantic space in a 1 OmniSONAR supports text in 1.5k languages, speech in 177 languages and the added image and video modalities. 1 arXiv:2603.01096v1 [cs.CV] 1 Mar 2026 post-hoc manner. The alignment follows a coarse-to-fine curriculum, over three stages of vision captioning data: (1) large-scale image–caption pairs (12M) for coarse grounding, (2) synthetic video–caption pairs (2M) for temporal adaptation, and (3) high-quality human-annotated video captions (200K) for fine-grained alignment. We evaluatev-Sonarextensively. On zero-shot video retrieval, it achieves Recall@1 of 73.03 on PE-Video, largely surpassing SigLIP2-g-opt (63.91). On zero-shot video captioning, it outperforms state-of-the-art vision–language models, improving Bleu by +18, +4.3 on PE-Video, Dream-1k, respectively, over the Perception Language Model (Cho et al., 2025). By aligningv-Sonarto Sonar, we show that the latent diffusion language model operating in Sonar, LCM (LCM team et al., 2024) trained with English textual corpus, can zero-shot process the visual embeddings encoded byv-Sonar. In the single-concept understanding task, i.e., video captioning, LCM only lags behind the existing VLMs with limited margins across PE-Video, Dream-1k, and Vatex. Similarly, LCM remains competitive for multi-concept reasoning task, i.e., long video summarization as evaluated on VideoXum. From the view of vision-language modeling, LCM serves as a new paradigm which unifies vision and language modality to the modality-agnostic latent space shared by Sonar andv-Sonar, and directly predict the next embedding with the latent diffusion objective. Therefore, we further introduce a vision-language instruction fine-tuned LCM as an exploration to maximize its utility in various downstream vision-language tasks, named v-LCM. v-LCM encodes multimodal data (images, videos, and text) withv-Sonarand Sonar, and it is trained with the same latent diffusion strategy, following the original two-tower LCM framework (LCM team et al., 2024) in its textual pre-training. We evaluate v-LCM on the multilingual and -modal instruction-tuning dataset, M3IT (Li et al., 2023), which spans 8 task categories, supports both image and video modalities, and covers 80 languages. v-LCM achieves competitive performance with other vision-language models such as InternVL (Chen et al., 2024b), Qwen-VL (Wang et al., 2024c; Bai et al., 2025) and Perception LM on image/video captioning, visual question answering, and other generation tasks. Notably, in M3IT’s multilingual evaluation, v-LCM outperforms other VLMs in 61 languages out of 62 tested languages, ranging from high-resource to low-resource setting. The contributions of this work are: • We introducev-Sonar, the first extension of a language- and modality-agnostic embedding space (Sonar) to image and video, via a post-hoc coarse-to-fine alignment strategy. •We demonstrate thatv-Sonarachieves state-of-the-art zero-shot performance on video retrieval and captioning, and generalizes robustly to multilingual settings. •We show that the LCM, originally trained on text-only data, can effectively operate onv-Sonar embeddings for zero-shot single- and multi-concept vision understanding tasks. • We extend LCM into a latent diffusion vision–language model (v-LCM) by unifying vision and language in the shared latent space ofv-Sonarand Sonar. On M3IT, v-LCM matches state-of-the-art VLMs in captioning and question answering while outperforming them in 61 non-English languages. 2 v-Sonar We begin by introducingv-Sonar, a vision–language embedding space constructed by post-hoc aligning a state-of-the-art vision encoder, Perception Encoder, with the multilingual textual embedding space Sonar. We select the Perception Encoder as the base encoder for two key reasons: (1) it achieves state-of-the-art performance across both image and video modalities (Bolya et al., 2025), and (2) it has been pre-trained in conjunction with a lightweight text encoder, which facilitates much easier post-hoc alignment with Sonar. This design choice distinguishes Perception Encoder from alternative vision encoders such as v-JEPA (Bardes et al., 2023; Assran et al., 2025) and DINO (Oquab et al., 2023; Siméoni et al., 2025), which primarily prioritize visual feature learning without explicit consideration of textual alignment. Architecture The architecture ofv-Sonaris illustrated in the left panel of Figure 1. Given the input image or video, Perception Encoder (PE) will first encode each frame separately. Then, we stack a lightweight projector on top of PE to adapt the encoder’s representations into the Sonar space. The projector first 2 SONAR Encoder Reconstruction Loss Caption Percep. Encoder SONAR Decoder VisualInput (image/video) Caption Loss Question: What is the animal in the Video? vSONAR SONAR VisualInput (video) v-LCM LCM Response: 2 pandas. vSONAR Connector SONAR SONAR Instruction: Watch the video and answer the question. Figure 1 Left: Illustration of v-Sonar. Right: training v-LCM with vision-language instruction tuning. injects positional embeddings to the embeddings of all frames, thus encoding temporal order information, followed by a single temporal attention layer that enables frame-level interactions. Finally, an attention layer then aggregates the frame embeddings into a single video-level representation, which serves as the final embedding for downstream tasks. See Appendix C for implementation details. Alignment from Vision to Language. We use the captioning data for the Perception Encoder to align with Sonar, with assumption that the visual inputs and caption should have the same semantic meaning, thus the high-level representations should be as close as possible in the latent modality-agnostic space. Therefore, given a set ofNpaired visual inputs and captionsD=(V i ,T i ) N i=1 , whereV i is an image or video andT i is its corresponding caption, we seek to learn a mapping such that the visual embedding z v =f θ (V i ) and the textual embedding z t =g(T i ) share the same semantic space, wheref θ denotes the trainable vision encoder andgis the frozen Sonar text encoder. To enforce semantic alignment, we minimize the discrepancy between visual and textual embeddings in the Sonar space using Mean Squared Error (MSE) loss: L align = 1 N N X i=1 ∥f θ (V i )− g(T i )∥ 2 2 .(1) Following the teacher-student training (Reimers and Gurevych, 2020; Duquenne et al., 2023), Sonar is frozen and we only update the parameters in the lightweight projector, and the vision encoder. We also experimented with an additional contrastive loss (Oord et al., 2018; Radford et al., 2021) but found no significant gains; details and results are in Appendix A. model xsim ↓xsim++ ↓ Sonar11.3715.27 OmniSONAR0.656.14 Table 1 Similarity search over 200 languages in Flores. We design a coarse-to-fine curriculum to progressively adapt the vision encoder to more complex semantics. The alignment proceeds through three stages. In Stage 1, we initialize align- ment using 12M large-scale image–caption pairs from the PLM data pipeline (Cho et al., 2025) which consists of Segment- Anything (Kirillov et al., 2023) and OpenImages (Kuznetsova et al., 2020). This stage establishes a basic mapping between visual and textual embeddings. Then, we introduce 2M pairs from PLM’s synthetic video captioning data from YouTube1B corpus (Cho et al., 2025). This step adapts the vision encoder to temporal dynamics while maintaining semantic consistency with Sonar. Finally, we refine the alignment using 200K high-quality human-checked video–caption pairs sourced from PE-Video (Bolya et al., 2025). We use two versions of the Sonar encoder: Sonar1 is the published and open-sourced version (Duquenne et al., 2023). This is the version supported by the LCM. We had early access to an improved version, named OmniSONAR, which was trained on more data and adds three stages of contrastive training and 3 MethodR@1 ↑ R@5 ↑ R@10 ↑ MRR ↑ AC ↑ V. Trace ↑ V. logdet ↑ T. Trace ↑ T. logdet ↑ PE-Vid SigLIP2-G-OPT47.55 71.47 79.41 58.47 0.3960.393−1.7× 10 4 0.582−1.7× 10 4 PECoreG63.91 85.98 91.61 73.77 0.4760.479−1.4× 10 4 0.686−1.4× 10 4 v-Sonar73.03 89.75 93.81 80.50 0.5190.700−1.2× 10 4 2.216−8.0× 10 3 DREAM SigLIP2-G-OPT61.50 83.50 89.10 71.50 0.2630.401−1.8× 10 4 0.662−1.8× 10 4 PECoreG72.10 89.80 93.60 79.90 0.3070.495−1.4× 10 4 0.639−1.4× 10 4 v-Sonar63.30 84.10 89.00 72.46 0.4100.559−1.2× 10 4 2.523−8.5× 10 3 Vatex SigLIP2-G-OPT27.52 57.70 70.06 41.27 0.2890.352−1.7× 10 4 0.660−1.7× 10 4 PECoreG18.90 42.42 54.72 30.42 0.3790.480−1.4× 10 4 0.508−1.4× 10 4 v-Sonar40.75 68.63 78.88 53.59 0.4270.558−1.2× 10 4 1.660−8.3× 10 3 Table 2 Zero-shot Retrieval performance on PE-Video, Dream-1k and Vatex. We report the Recall rate at 1/5/10 and MRR scores. We also report the analytical metrics for the embedding space, including 1) trace reflects overall variance, and 2) log determinant (logdet) approximates volume in the space. Best values for each columns are bolded. self distillation (Omnilingual Embeddings Team et al., 2026). As summarized in Table 1, OmniSONAR substantially outperforms Sonar1 on the proxy metric of multilingual similarity search. The metric xsim++ includes hard negatives (Chen et al., 2023). We provide an ablation of the two Sonar versions for vision captioning tasks in Appendix D. 2.1 v-LCM The Large Concept Model (LCM; LCM team et al. 2024) is a latent diffusion language model operating directly in the Sonar embedding space. It follows an auto-regressive paradigm, predicting the next sentence embedding conditioned on preceding clean embeddings. For the textual modality, all embeddings are encoded and decoded by the fixed Sonar encoder and decoder. To model the conditional distribution of the next embedding, LCM employs a diffusion objective: given a clean embeddingx 0 ∈ R d , the forward process progressively perturbs it with Gaussian noise under a variance-preserving schedule (Karras et al., 2022): q(x t | x 0 ) =N x t ;α t x 0 , σ 2 t I , x t = α t x 0 + σ t ε, ε∼N(0,I),(2) where (α t ,σ t ) are determined by a monotonically decreasing log-SNR scheduleλ t =log(α 2 t /σ 2 t ). The reverse process is parameterized by a denoiserμ θ (x t ,t,c), conditioned on the context embeddingsc, with Gaussian transitions: p θ (x t−1 | x t ,c) =N x t−1 ;μ θ (x t ,t,c), σ 2 t I .(3) Training minimizes a reconstruction loss on the original clean embedding: L(θ) = E t,x 0 ,ε x 0 − μ θ (α t x 0 + σ t ε,t,c) 2 .(4) We use the two-tower variant of LCM, which separates the contextualizer (encoding the preceding embeddings) from the denoiser (iteratively reconstructing the next embedding). From the perspective of vision–language modeling, LCM represents a new paradigm that fuses information from visual and textual modalities within a modality-agnostic latent space prior, rather than discrete visual and textual tokens (Chameleon Team, 2024). This enables autoregressive generation to be performed entirely in the latent space. Building on this principle, we further introduce v-LCM, an extension of LCM trained through vision–language instruction fine-tuning to enhance its utility across a broad range of downstream vision-language tasks. In v-LCM, visual inputs (images and videos) are encoded into the Sonar latent space usingv-Sonar, while textual instructions and prompts are encoded with Sonar. The resulting visual and textual embeddings are concatenated into a single sequence, which is then processed under the same latent diffusion framework as in LCM’s original text-only training, predicting the next embedding in the sequence. 3 Experiments We first verify the effectiveness of aligning the vision encoder to Sonar, by evaluating text-video retrieval and captioning usingv-SonarOmni aligned with the OmniSONAR text encoder, and provide several ablations. 4 ModelBleuR-1R-2R-LBS-PBS-RBS-F PE-Video InternVL2.5-1B19.4 32.19.0 23.431.227.329.3 InternVL2-1B24.1 35.8 10.7 25.530.832.131.5 Qwen2-VL-2B-Instruct29.9 41.7 18.8 31.234.840.037.3 Qwen2.5-VL-3B-Instruct30.0 41.3 16.1 28.930.238.634.4 PLM-1B21.5 37.6 11.9 26.635.826.231.0 PLM-3B21.1 37.5 11.7 26.436.626.131.3 v-Sonar w/ OmniSONAR Decoder39.050.123.338.044.441.643.0 Dream-1k InternVL2.5-1B10.2 21.53.7 15.626.111.218.6 InternVL2-1B14.6 25.04.3 17.223.815.219.5 Qwen2-VL-2B-Instruct19.7 27.15.2 18.512.914.813.9 Qwen2.5-VL-3B-Instruct16.1 23.94.4 15.91.615.68.6 PLM-1B18.5 27.06.4 19.314.516.815.5 PLM-3B19.6 28.66.7 20.419.918.119.0 v-Sonar w/ OmniSONAR Decoder23.932.78.422.719.721.620.7 Vatex InternVL2.5-1B41.5 23.34.4 19.237.645.240.2 InternVL2-1B47.827.36.422.436.950.442.4 Qwen2-VL-2B-Instruct32.1 19.86.0 16.418.746.330.8 Qwen2.5-VL-3B-Instruct29.4 18.35.1 15.012.147.127.6 PLM-1B33.4 21.85.7 19.115.148.129.6 PLM-3B34.0 22.15.9 19.316.948.630.8 v-Sonar w/ OmniSONAR Decoder 26.7 17.25.0 14.813.642.826.5 Vatex-zh InternVL2-1B-Instruct22.3 14.12.9 11.88.718.212.6 InternVL2.5-1B-Instruct33.2 22.54.4 18.819.131.524.2 v-Sonar w/ OmniSONAR Decoder 30.632.18.5326.923.248.533.7 Table 3 Video captioning performance across PE-Video, Dream-1k and VATEX (English and Chinese). Metrics include Bleu, Rouge (R-1, R-2, R-L), and BERTScore (BS-P, BS-R, BS-F). We then switch to the zero-shot evaluation for LCM, and evaluation of v-LCM on M3IT (Li et al., 2023) which requires the use of v-Sonar1, as the LCM had been trained on Sonar1. 3.1 Concept Space Alignment using v-SonarOmni Text-video Retrieval We treatv-Sonaras a paired vision–text encoder and begin by evaluating its zero-shot performance on text-to-video retrieval, following the setup in Bolya et al. (2025). We comparev-Sonar against two strong baselines: the state-of-the-art SigLIP2 vision encoder (Tschannen et al., 2025) and the Perception Encoder (Bolya et al., 2025), from whichv-Sonaris derived. Evaluations are conducted on three widely used video captioning benchmarks: PE-Video (15K pairs of captioning data) (Bolya et al., 2025), Vatex (5K pairs of captioning data) (Wang et al., 2019), and Dream-1k (1K pairs of captioning data) (Wang et al., 2024a), following the protocol in Cho et al. (2025). In addition to standard retrieval metrics such as Recall@1/5/10, we introduce three complementary measures to analyse embedding space properties: (1) Alignment Consistency (AC): the rank correlation between vision and text similarity scores, reflecting cross-modal alignment quality. (2) Trace: the trace of the covariance matrix of vision and text embeddings, indicating the spread of representations. (3) Log-determinant (logdet): the logarithm of the determinant of the covariance matrix, interpreted as the volume of the embedding ellipsoid. Table 2 summarizes the results and embedding space statistics. On the three datasets, PE-Video, Dream-1k and Vatex,v-Sonarsignificantly outperforms SigLIP2, achieving improvements of 9.12, 1.8 and 13.23 points 5 in Recall@1, respectively, demonstrating the effectiveness of our approach for retrieval tasks. Compared to the original Perception Encoder,v-Sonarsignificantly improves on PE-Video and Vatex (9.12 and 21.85 score at Recall@1), though it loses 8.8 score at Recall@1 in Dream-1k, indicating that our curriculum alignment strategy preserves strong retrieval capability. These results confirm that a vision encoder can be successfully aligned with a purely text-trained embedding space (Sonar) in a post-hoc manner. Finally, our embedding space analysis reveals thatv-Sonarmaintains a more expanded distribution. Moreover, by freezing the original Sonar space,v-Sonarachieves the largest textual embedding dispersion, as evidenced by the highest trace and logdet values among all compared models. Text-video Captioning Different with the traditional vision encoder, such as SigLIP 2 or Perception En- coder, aligningv-Sonarto Sonar embedding space allows us to leverage the Sonar decoder to directly verbalize the encoded vector ofv-Sonar. Hence, we conduct the zero-shot evaluation on video captioning for v-Sonar, and compare it with few state-of-the-art vision-language models (VLMs) including InternVL-2/2.5 (Chen et al., 2024b), Qwen-VL 2/2.5 (Wang et al., 2024c; Bai et al., 2025) and Perception Language Models (Cho et al., 2025). We compare with VLMs at the scale between 1B to 3B for a fair comparison, as the Sonar decoder is at 1.5B andv-Sonaris at 1.9B. We evaluate the models with lexical metrics including Bleu and Rouge scores, and semantic metrics including BERTScore-Precision/Recall/F1, following (Zhang et al., 2025). We illustrate the results in Table 3. For detailed captioning benchmarks such as PE-Video and Dream-1k, we observe thatv-Sonarpaired with the Sonar decoder can achieve a state-of-the-art performance. In particular,v-Sonarimproves the second best model, Qwen2.5-VL-3B-Instruct, by 9 points in Bleu. The only exception is Vatex where the captions are relatively short as one sentence,v-Sonarlags behind InternVL2; however, this is expected as we alignv-Sonarwith Sonar mostly with the detailed caption data. And we observev-Sonaris still comparable with PLM and Qwen-VL series. We use Vatex-Chinese validation set for the multilingual evaluation, and we mostly comparev-Sonarwith InternVL, as QwenVL is reported to leverage Vatex Chinese split during training (Wang et al., 2024c; Bai et al., 2025), and PLM-1/3B fail to support the fluent generation in Chinese. We find that in Vatex Chinese split,v-Sonarstill outperforms the InternVL series, indicating the advantage in multilingual evaluation. v-Sonar1 vsv-SonarOmni. We present the comparison between Sonar (Duquenne et al., 2023) and OmniSONAR in Figure 2. We report both Sonar space’s oracle performance (we encode the reference caption with Sonar encoder, and decoded with Sonar decoder). And we report the zero-shot performance withv-Sonar(we encode the video withv-Sonarand decode with Sonar decoder). Sonar oracle serves as an estimation of the upper-bound performance that v-Sonar can achieve for leveraging Sonar decoder. We observe that both Sonar versions have a strong oracle performance, indicating Sonar’s encoding and decoding from textual space into its representation space is quite lossless. Specifically, in PE-Video, Vatex and Dream-1k, OmniSONAR can achieve Bleu scores of 81, 96 and 70. Comparing the zero-shot performance for Sonar and OmniSONAR, we see Sonar1 is worse by a considerable margin. We hypothesize that Sonar is harder to align since its space is reported to be collapsed. Our analysis for Sonar and OmniSONAR also supports this observation: in PVD, Sonar and OmniSONAR have the embeddings norm at 0.264 and 1.69, and covariance trace at 0.049 and 1.83, respectively. The comparison in retrieval tasks are in Appendix D. BLEUR-1R-2R-LBS-F 0 50 100 89 94 85 9090 81 87 74 8484 40 53 25 39 47 32 43 19 3535 PVD-Bench BLEUR-1R-2R-LBS-F 99 45 21 40 99 96 43 20 38 95 39 23 6 19 34 31 20 5 18 28 VATEX BLEUR-1R-2R-LBS-F 83 88 73 81 83 70 78 58 73 75 24 35 9 24 25 20 28 5 21 16 DREAM-1K SONAR-2 OracleSONAR-1 OracleSONAR-2SONAR-1 Figure 2 Comparison forv-Sonartrained with SONAR version 1 and 2 (OmniSONAR) embedding space on three captioning datasets. 6 MSECos. Sim.Bleu R-1 R-2 R-L BS-P BS-R BS Architecture Linear Proj.1.45×10 −3 0.69438.0 49.7 21.6 36.7 47.2 40.1 43.7 Full PE1.54×10 −3 0.67237.1 48.5 21.3 36.5 46.9 38.8 42.9 + Async. LR1.43×10 −3 0.70039.7 51.3 23.3 37.7 48.1 42.1 45.1 + Norm. Init.1.39×10 −3 0.70839.8 51.8 24.0 38.5 49.4 42.2 45.8 + Attn. Pooling1.39×10 −3 0.70839.8 51.9 24.0 38.5 49.7 42.4 46.0 + Temporal Attn.1.39×10 −3 0.70839.8 51.9 24.0 38.5 49.7 42.4 46.1 Pipeline Full Pipeline1.36×10 −3 0.71640.1 52.6 24.9 39.2 50.8 43.2 47.0 w/o SV1.39×10 −3 0.71039.6 51.9 24.1 38.6 50.0 42.4 46.2 w/o IC & SV1.39×10 −3 0.70839.8 51.9 24.0 38.5 49.7 42.4 46.1 Table 4 Ablation study in model architecture and the three-stage training pipeline. SV: our second stage curriculum with the synthetic video captioning data. IC: our first stage curriculum with image captioning data. 3.2 Ablation Study We conduct an ablation study for model architecture design, and our proposed training pipeline on the PE-Video test set (Table 4). Model Architecture We ablate architectural choices for the projector network. As a baseline, we evaluate linear projection (Linear Proj.), where the Perception Encoder is frozen and only a linear layer is trained, and full-model fine-tuning (Full PE), where the encoder is updated jointly. Linear projection performs better, indicating that the encoder’s contrastive pre-training already yields strong semantic alignment, while full fine-tuning is hindered by unstable gradients from the randomly initialized projector. To mitigate this, we adopt strategies that incrementally improve downstream performance, including asynchronous learning rates for the projector and encoder, initialization trick, attention-based aggregation strategy for video frames’ features, and temporal attention layer. Data Mixture We then ablate the second stage synthetic video captioning and first-stage image captioning curriculum on our pipeline. We observe that both stages contribute positively to the downstream performance in captioning performance on PE-Video. Removing the 12M image captioning pairs and 2M video captioning pairs reduce the Bleu with 0.3 and 0.2, respectively. 3.3 Zero-shot Processing v-Sonar Embeddings by Large Concept Model Since the LCM (LCM team et al., 2024) operates directly on Sonar1, it should seamlessly transfer its ability and understand the visual concepts inv-Sonaraligned with Sonar1. We examinev-Sonarwith LCM gradually from single to multiple vision concept understanding tasks, where the LCM accepts the instruction encoded by Sonar, with the vision embeddings fromv-Sonar, and predicts the target embedding. Note that in both experiments, we do not fine-tune the LCM, with neither any video data nor captioning data. Thus, the LCM is only trained in English textual corpus including its pre-training and instruction fine-tuning as in LCM team et al. (2024). We compare LCM’s performance with VLMs at 7/8-B scale for the InternVL series (Chen et al., 2024b), Qwen-VL (Bai et al., 2025; Team, 2024) and PLM (Cho et al., 2025). Single Vision Concept Understanding: Video Captioning We report the results of LCM on video captioning in Table 5. Compared to the strongest baseline, the zero-shot LCM lags behind by 1.15/4.44/4.76 Bleu scores on PE-Video, Dream-1k and Vatex, respectively. Among the models, PLM-8B delivers the strongest overall performance. The relatively narrow performance gap between LCM and competitive VLMs suggests that the LCM is able to understand the single vision concept, despite never being trained with video data. Multiple Vision Concept Understanding: Long Video Summarization We next evaluate LCM in a setting requiring understanding multiple visual embeddings. Long videos are uniformly segmented into snippets 7 Video Captioning / SummarizationM3IT ImageM3IT Video PE-VideoDream-1kVatexVideoXumCOCOVIQUAEVisualMRCScienceQA ActivNetQA MSRVTT-QAIVQA R-L BS R-L BS R-L BS R-L BSR-L R-L R-L Acc.R-L R-L R-L InterVL2 1B25.5 31.5 17.2 19.5 22.4 42.4 15.3 17.712.6 24.0 30.6 53.940.6 27.6 39.5 4B15.0 18.4 12.2 14.7 19.2 42.4 15.6 17.417.5 20.0 35.3 89.627.5 24.5 31.8 8B 18.6 23.4 16.4 19.4 16.2 42.4 29.1 26.121.0 21.6 42.9 87.229.7 27.1 38.4 InternVL-2.5 1B23.4 29.3 15.6 18.6 19.2 40.3 17.1 23.213.2 10.8 27.3 69.016.6 11.7 19.3 4B14.6 18.0 13.3 14.8 17.3 36.2 18.1 23.015.1 23.1 45.2 86.426.8 21.8 24.9 8B 21.4 26.0 17.0 17.0 20.6 42.4 24.9 20.516.8 17.3 42.8 93.120.9 16.9 22.7 Qwen2-VL 2B31.2 37.3 18.5 13.9 16.4 30.8 23.6 29.824.9 50.2 56.1 54.553.7 39.6 49.4 7B 26.9 32.6 19.8 18.1 28.5 51.6 26.0 32.423.7 49.7 57.4 70.441.9 22.7 39.1 Qwen2.5-VL 3B 28.9 34.4 15.9 8.6 15.0 27.6 26.0 32.925.1 48.3 55.7 55.052.1 41.6 48.5 7B22.2 25.9 15.7 10.5 27.5 50.8 24.1 28.918.5 34.5 45.0 61.646.0 41.4 54.2 Percep. LM 1B26.6 31.0 19.3 15.5 19.1 29.6 21.8 33.227.5 30.8 45.5 73.627.8 14.5 39.1 3B26.4 31.3 20.4 19.0 19.3 30.8 27.0 36.434.3 23.7 51.1 89.828.0 19.4 26.1 8B27.4 31.9 20.8 19.7 19.0 30.8 26.2 33.736.3 31.0 50.0 87.740.5 25.3 41.4 LCM LCM 25.5 27.9 18.5 16.6 23.8 30.8 21.5 22.118.0 34.3 33.5 44.751.7 36.0 48.9 v-LCM 27.4 30.0 19.8 19.2 28.8 48.7 20.6 25.338.8 39.4 34.1 76.263.6 48.7 63.9 Table 5 Main results on vision-language tasks in M3IT and the previous video benchmarks (PE-Video, Dream-1k, Vatex, VideoXum). with 8 frames per each, with each snippet encoded byv-Sonaras a separate video embedding. Since the LCM shows strong performance in document summarization (LCM team et al., 2024) for multiple Sonar embeddings; we hypothesize that it should be capable of performing zero-shot summarization over sets of video embeddings fromv-Sonar. For this evaluation, we use VideoXum (Lin et al., 2023), which contains videos of one to five minutes, uniformly split into snippets of 8 frames each. We report the VideoXum results in Table 5. Again, PLM achieves the strongest performance among competitive VLMs at the same scale.v-Sonar+ LCM achieves 22.1 score at BertScore-F1, trailing the best-performing PLM-8B (33.7) but is slightly higher than InternVL-2.5-8B at 20.5. These findings indicate that, even without exposure to any video data during training, LCM demonstrate non-trivial understanding of multiple v-Sonar embeddings for long videos. ShortMid Long 0.0 0.1 0.2 ROUGE-L SONAR vSONAR Figure3 Operating inv-Sonarspace, the LCM performs better than only accepting the textual Sonar inputs. We compare LCM-7B-IFT in VideoXum with Rouge-L scores across short, mid, and long categories of video inputs. Reasoning inv-SonarWe then investigate whether LCM truly leverages the latent representations inv-Sonarfor multimodal reasoning. To test this, we compare two settings: (1) encoding video clips directly intov-Sonarembeddings, which are then fed into LCM for zero-shot summarization; and (2) decoding video embeddings into captions using the Sonar decoder, re-encoding them with Sonar, and providing these Sonar embeddings to LCM. Our hypothesis is that v-Sonarembeddings retain richer visual features than their textual equivalents in Sonar, and thus should yield stronger performance if LCM relies on visual representations. We group videos into short (<90s), mid-length (90–150s), and long (>150s) categories, and report Rouge-L scores in Figure 3. Across all categories, LCM withv-Sonarconsis- tently outperforms it with Sonar. Notably, while Sonar performance declines with increasing video length,v-Sonar 8 remains stable, highlighting its robustness. These results sup- port our hypothesis that the LCM reasons directly in the visual embedding space provided byv-Sonar containing richer visual information than Sonar representations of textual input. VCRF1Sim. LCM0.385 0.258 vLCM0.671 0.529 PLM-8B0.441 0.432 Qwen-2.5-7B 0.275 0.402 Qwen-2-7B0.502 0.513 InterVL2.5-8B 0.155 0.158 InterVL2-8B 0.325 0.340 Table 6 VCR results comparing token-level F1 and semantic similarity. v-LCM Reasoning with Visual Details. To assess whether our alignment from Perception Encoder to Sonar preserves the visual details critical for down- stream reasoning, we further evaluate visual common- sense reasoning (VCR) performance on M3IT (Table 6). In this benchmark, models must reason about objects using commonsense knowledge while interpreting their associated bounding boxes. Compared to semantic-level alignment alone, VCR demands substantially richer vi- sual grounding and layout awareness (see Figure 21). We report token-level F1 scores against the ground truth rationales, alongside semantic similarity to the reference explanations. The strong performance of v-LCM indi- cates that, despite being trained solely on semantic-level captions, v-LCM effectively leverage layout grounding and spatial relationship preserved by v-Sonar in reasoning tasks. 3.4 v-LCM In-task Performance We next evaluate v-LCM, which is supervised fine-tuned on M3IT (Li et al., 2023) to better capture and reason over visual concepts. We mainly rely on M3IT (Li et al., 2023) as it supports a variety of tasks, and the wide coverage for up to 80 languages ranging from high- to low-resource languages. The evaluation covers 7 datasets spanning 5 tasks defined in M3IT: (1) image captioning (COCO), (2) visual QA (VIQUAE), (3) document image QA (VisualMRC), (4) video captioning (MSRVTT), and (5) question answering (IVQA, MSRVTT-QA, ActivityNetQA). In addition, we report v-LCM’s performance on the video captioning and long video summarization benchmarks introduced in the previous section. Following the evaluation protocol in Li et al. (2023), we useRouge-Lfor generative tasks (e.g., captioning and open-ended QA) and accuracy for multiple-choice QA. Table 5 compares LCM with strong open-source vision–language models. We observe that v-LCM substantially outperforms the zero-shot LCM across most benchmarks. For example, v-LCM achieves 63.9 R-L on IVQA and 63.6 R-L on ActivityNetQA, surpassing 48.9 and 51.7 for LCM, representing clear gains from training with vision instruction-tuning data. While v-LCM lags behind the best-performing models on some benchmarks such as VisualMRC, VIQUAE, and ScienceQA, it achieves state-of-the-art results on video question answering tasks, including IVQA, ActivityNetQA, and MSRVTT-QA. Meanwhile, performance on our previous video captioning and summarization datasets remains competitive: v-LCM attains 27.4 R-L on PE-Video and 19.8 R-L on DREAM-1K, trailing the best model by only 1.5 and 1 Rouge scores, highlighting the generalization of LCM to unseen datasets during training. Multilinguality We further conduct a multilingual evaluation of v-LCM on the M3IT benchmark across 62 languages, 2 leveraging the fact that v-LCM operates entirely in the latent space of Sonar andv-Sonar, and can therefore decode outputs to any languages supported by Sonar. Evaluation spans five various tasks including image classification (ImageNet), image question answering (VQA-V2, OKVQA), video question answering (MSRVTT-QA), video captioning (MSRVTT) and narrative generation (VIST), covering a spectrum from high-resource languages (e.g., Chinese), mid-resource languages (e.g., Japanese) to low-resource languages (e.g., Javanese). We use the Rouge-L implementation from (Shohan et al., 2024) for the multilingual evaluation. As shown in Figure 4, v-LCM consistently outperforms Qwen2.5-VL-7B and PLM-8B across 61 of 62 languages, with Dutch being the only exception. While improvements in some high-resource languages are modest (e.g. 2 The intersection of all languages supported by Sonar, M3IT, and multilingual Rouge. 9 Thai Burmese Portuguese Vietnamese Spanish Indonesian Catalan Gujarati Hindi Greek Cebuano Kannada Eastern Panjabi Chinese (Simp.) Hungarian Tamil Tagalog Belarusian Bulgarian Turkish Hebrew Telugu German Romanian Korean French Ukrainian Danish Bengali Afrikaans Galician Macedonian Bosnian Italian Czech Urdu Slovak Malayalam Maltese Javanese Russian Swedish Polish Dutch MS Arabic Croatian Marathi Occitan Armenian Serbian Tajik Welsh Kazakh Kyrgyz Standard Latvian Estonian Icelandic Asturian Georgian Japanese Finnish Luxembourgish 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Metrics v-LCMQwen2.5-VL-7BPLM-8B Figure 4 Performance in 62 languages for v-LCM, Qwen2.5-VL-7B and PLM-8B on M3IT testing set for MSRVTT, MSRVTT-QA, ImageNet, VQA-V2, VIST and OKVQA. We report the Rouge-L scores averaged from all datasets. Detailed results for each dataset can be found in the Appendix E. French), the gains become substantial in mid- and low-resource settings, including Burmese, Tajik and Telugu. Notably, for languages such as Urdu, modern Arabic and Tamil, which is unsupported by PLM-8B based on LLaMA-3.2 (Touvron et al., 2023; Dubey et al., 2024), v-LCM successfully generates meaningful outputs, whereas competing models fail entirely. 4 Related Works A central paradigm in multimodal learning is to align vision and language representations into a shared embedding space. Early approaches such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) established large-scale contrastive learning between paired images and captions, enabling zero-shot transfer to downstream tasks. Subsequent works extended this idea to video–language pretraining (Lei et al., 2021; Xu et al., 2021; Wang et al., 2022). More recent efforts focus on aligning pretrained encoders into an unified space: Perception Encoder (Bolya et al., 2025) projects diverse perceptual modalities into a shared latent space, while scaling data and architectures in models, such as Florence (Yuan et al., 2021) and SigLIP2 (Tschannen et al., 2025), further improve alignment quality. Recent work also shows that using large language models as text encoders enhances vision–language alignment (Stone et al., 2025), and post-hoc alignment strategies have been proposed as lightweight alternatives to joint training (Brokowski et al., 2025; Yang et al., 2025). Parallel advances in multilingual text embedding models, such as LASER (Artetxe and Schwenk, 2019; Heffernan et al., 2022), LaBSE (Feng et al., 2020), and Sonar (Duquenne et al., 2023; Omnilingual Embeddings Team et al., 2026), demonstrate the effectiveness of language-agnostic embedding spaces across hundreds of languages. Modular approaches have further explored language-specialized components to reduce interference in universal embedding spaces (Huang et al., 2024). These universal text embeddings provide an attractive target for aligning vision encoders, as they inherit cross-lingual generalization without requiring multimodal data in every language. Prior work has explored similar strategies in speech-to-text alignment (Chung et al., 2018; Duquenne et al., 2021a; Laperrière et al., 2024; Du et al., 2024), but large-scale alignment of visual embeddings into such universal text spaces remains underexplored. 5 Conclusion We introducev-Sonarby extending the Sonar embedding space with the image and video modality. To the best of our knowledge, this makes Sonar the most universal embedding space covering four modalities (text, speech, image and video) and up to 200 languages. We propose a three-stage training approach to map a pooled representation based on the Perception Encoder to the semantic Sonar representation. We 10 achieve very competitive results for text-to-video retrieval and video captioning. The Large Concept Model (LCM; LCM team et al. 2024) is a recent approach to perform reasoning at a higher semantic conceptual level, namely Sonar. Encoded byv-Sonar, we show that the LCM can zero-shot process image or video embeddings without the need of training data in these modalities. We further introduce v-LCM with the multimodal instruction fine-tuning, which matches state-of-the-art VLMs, while significantly outperforming them across 61 rich- to low-resource languages. References Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. TACL, pages 597–610, 2019. Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. V-jepa: Latent video prediction for visual representation learning. submitted to ICLR’25, 2023. Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504.13181, 2025. Trevor Brokowski, Alexandre Sallinen, and Mary-Anne Hartley. Ssca: Siglip-2 sonar concept alignment. In Second Workshop on Visual Concepts, 2025. Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. Jianlv Chen, Shitao Xiao, Peitian Zhang, KunLuo, Defu Lian, and Zheng Liu. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024a. Mingda Chen, Kevin Heffernan, Onur Çelebi, Alexandre Mourachko, and Holger Schwenk. xSIM++: An improved proxy to bitext mining performance for low-resource languages. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, ACL, pages 101–109, 2023. https://aclanthology.org/2023.acl-short.10/. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198, 2024b. Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Triantafyllos Afouras, Tushar Nagarajan, Muhammad Maaz, Yale Song, Tengyu Ma, Shuming Hu, Suyog Jain, et al. Perceptionlm: Open-access data and models for detailed visual understanding. arXiv preprint arXiv:2504.13180, 2025. Yu-An Chung, Wei-Hung Weng, Schrasing Tong, and James Glass. Unsupervised cross-modal alignment of speech and text embedding spaces. Advances in neural information processing systems, 31, 2018. Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407, 2024. Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk. Multimodal and multilingual embeddings for large-scale speech mining. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 15748–15761. Curran Associates, Inc., 2021a. https://proceedings.neurips.c/paper_files/paper/2021/file/8466f9ace6a9acbe71f75762ffc890f1-Paper.pdf. Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk. Multimodal and multilingual embeddings for large-scale speech mining. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, 11 Advances in Neural Information Processing Systems, volume 34, pages 15748–15761. Curran Associates, Inc., 2021b. https://proceedings.neurips.c/paper_files/paper/2021/file/8466f9ace6a9acbe71f75762ffc890f1-Paper.pdf. Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. Sonar: sentence-level multimodal and language-agnostic representations. arXiv preprint arXiv:2308.11466, 2023. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language-agnostic bert sentence embedding. arXiv preprint arXiv:2007.01852, 2020. Kevin Heffernan, Onur Çelebi, and Holger Schwenk. Bitext mining using distilled sentence representations for low-resource languages. arXiv preprint arXiv:2205.12654, 2022. Yongxin Huang, Kexin Wang, Goran Glavaš, and Iryna Gurevych. Modular sentence encoders: Separating language specialization from cross-lingual alignment. arXiv preprint arXiv:2407.14878, 2024. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR, 2021. Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565–26577, 2022. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023. Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, 128(7):1956–1981, 2020. Gaëlle Laperrière, Sahar Ghannay, Bassam Jabaian, and Yannick Estève. A dual task learning approach to fine-tune a multilingual semantic speech encoder for spoken language understanding. arXiv preprint arXiv:2406.12141, 2024. LCM team, Loïc The Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alastruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta R Costa-jussà, David Dale, et al. Large concept models: Language modeling in a sentence representation space. arXiv preprint arXiv:2412.08821, 2024. Jie Lei, Tamara L. Berg, and Mohit Bansal. Less is more: Clipbert for video-and-language learning via sparse sampling. In CVPR, 2021. Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. M3it: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint arXiv:2306.04387, 2023. Jingyang Lin, Hang Hua, Ming Chen, Yikang Li, Jenhao Hsiao, Chiuman Ho, and Jiebo Luo. Videoxum: Cross-modal visual and textural summarization of videos. IEEE Transactions on Multimedia, 26:5548–5560, 2023. Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models, 2021. https://arxiv.org/abs/2108.08877. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia-Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human- centered machine translation, 2022. https://arxiv.org/abs/2207.04672. Omnilingual Embeddings Team, Pere-Lluís Huguet Cabot, João Maria Janeiro, Yen Meng, Ioannis Tsiamas†, Vivek Iyer, Jaehyeong Jo, Guillem Ramírez, Belen Alastruey, Loic Barrault, David Dale, Kevin Heffernan, Artyom Kozhevnikov, Alex Mourachko, Christophe Ropers, Holger Schwenk, Yu-An Chung, Marta R. Costa-Jussa, and Paul-Ambroise Duquenne. SONAR2: Omnilingual sentence embeddings for text and speech, 2026. to be published. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 12 Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra. Samanantar: The largest publicly available parallel corpora collection for 11 Indic languages. TACL, 10:145–162, 2022. https://aclanthology.org/2022.tacl-1.9/. Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In EMNLP, pages 4512–4525, 2020. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. CCMatrix: Mining billions of high-quality parallel sentences on the web. In https://arxiv.org/abs/1911.04944, 2019. Faisal Tareque Shohan, Mir Tafseer Nayeem, Samsul Islam, Abu Ubaida Akash, and Shafiq Joty. Xl-headtags: Leveraging multimodal retrieval augmentation for the multilingual generation of news headlines and tags. In ACL (Findings), 2024. Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025. Austin Stone, Hagen Soltau, Robert Geirhos, Xi Yi, Ye Xia, Bingyi Cao, Kaifeng Chen, Abhijit Ogale, and Jonathon Shlens. Learning visual composition through improved semantic guidance. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3740–3750, 2025. Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786, 2025. Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXiv preprint arXiv:2407.00634, 2024a. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilingual e5 text embeddings: A technical report, 2024b. https://arxiv.org/abs/2402.05672. Minghang Wang, Zeqiu Ma, Yuanjun Xie, Xuewen Li, et al. All-in-one: Exploring unified video-language pre-training. In CVPR, 2022. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024c. Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4581–4591, 2019. Hu Xu, Gargi Ghosh, Po-Yao Huang, Dahuang Zhang, Joshua Ainslie, et al. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In EMNLP, 2021. Jingfeng Yang, Ziyang Wu, Yue Zhao, and Yi Ma. Language-image alignment with fixed text encoders. arXiv preprint arXiv:2506.04209, 2025. Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, , et al. Florence: A new foundation model for computer vision. In CVPR, 2021. 13 Chunhui Zhang, Yiren Jian, Zhongyu Ouyang, and Soroush Vosoughi. Pretrained image-text models are secretly video captioners. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 292–305, 2025. 14 Appendix A Contrastive Loss for Aligning Perception Encoder and Sonar We have also explored the use of a contrastive loss in addition to the MSE loss for aligning the Percep- tion Encoder to Sonar. Specifically, given a mini-batch ofBpaired samples(V i ,T i ) B i=1 , we aim to not only minimize the distance between matched pairs (f θ (V i ),g(T i )) but also push apart mismatched pairs. We define the contrastive loss as: L con =− 1 B B X i=1 log exp sim(f θ (V i ),g(T i ))/τ P B j=1 exp sim(f θ (V i ),g(T j ))/τ ,(5) wheresim(·,·) denotes cosine similarity andτis a temperature parameter. The final loss is then a weighted combination of the MSE alignment loss and the contrastive loss: L =L align + λL con ,(6) whereλcontrols the strength of the contrastive term. However, in our preliminary experiments, adding the contrastive component did not yield a significant improvement over the MSE-only objective (Table 7) in captioning performance, while it leads to gains in retrieval performance. However, since our downstream usage of v-Sonar for v-LCM is closer to generation task, we choose the MSE-only loss as the final loss. CaptioningRetrieval Bleu R-L BS-F1 R@1 MRR MSE-only38.9 37.8 44.9 49.0 60.3 MSE + Contrastive 38.6 37.5 44.5 52.4 63.7 Table 7 Ablation study on using only MSE loss vs. adding a contrastive loss. Results are reported on the PE-Video benchmark for captioning and retrieval with a single MLP layer as the connector in v-Sonar. To further explain the trade-off between retrieval and captioning performance in Table 7, we hypotheses that contrastive training can move vision vectors off the exact SONAR manifold. It only enforces relative ordering via cosine margins, and produces embeddings whose norms or local covariance differ from those the SONAR decoder were trained on. That small manifold shift can degrade generative reconstruction even if retrieval improves. We analyze the statistics ofv-Sonarembeddings in Table 8. It is a clear observation thatv-Sonartrained with the contrastive loss has a more separate embeddings, evidenced by its higher norm, trace and volume. It shows that contrastive loss pushes the vSONAR embeddings to be a more expanded distribution compared to the MSE-only (higher values in Norm, Covariance trace, and Volume). However, their poor alignment consistencies in both MSE and cosine similarity, where we calculate the correlation between vision and text similarity ranking for each sample, suggests contrastive loss breaks the local covariance structure and may break the alignment with SONAR manifold. V. Norm V. Trace V. Volume AC (Cosine) AC (MSE) MSE1.220.48-10007.16 0.410.32 MSE+Contrastive 1.301.74-8107.94 0.310.13 Table 8 Additional statistics forv-Sonarrepresentations on PE-Video. We report the norm (V.Norm), trace (V.Trace), Volume (V.Volume), alignment consistency measured in Cosine Similarity (AC Cosine) and MSE (AC MSE) for video embeddings predicted by v-Sonar. 15 B Dataset Statistics Table 9 summarizes the datasets used in our three-stage training pipeline for alignment. The PLM-Image datasets (SA1B and OpenImages) provide large-scale image–caption pairs, which are particularly valuable for improving grounding and linguistic richness. The PLM-Video-Auto-YT1B dataset contributes video–text pairs with an average duration of 22.75 seconds, enabling the model to capture temporal dynamics in multimodal content. Finally, the PE-Video dataset provides carefully curated human-annotated video–caption pairs with moderate length, serving as a higher-quality supervision signal in later stages. Together, these datasets balance scale and quality, ensuring both broad coverage and precise alignment across modalities. Dataset#Samples Duration (s) Caption Length (sent.) Caption Length (words) PLM-Image-Auto-SA1B7.99M–10.7181.8 PLM-Image-Auto-OpenImages 1.37M–7.9132.1 PLM-Video-Auto-YT1B2.14M22.82.395.5 PE-Video118K16.74.451.4 Table 9 Statistics of the datasets used in 3-stage training for alignment. We report the number of samples, average video duration (if applicable), and average caption length in sentences and words. C Implementations v-SonarArchitecture We build our model on top of the Perception Encoder Vision Transformer backbone PE-Core-G14-448 3 . The encoder processes RGB images at a resolution of 448×448 pixels, splitting them into 14×14 patches and yielding 1024 patches per frame. The vision tower consists of 50 transformer layers, each with a hidden width of 1024, 16 attention heads, and a 4096-dimensional feed-forward network, resulting in approximately 1.9B parameters. For video inputs, we uniformly sample 8 frames and extract frame-level embeddings of 1536 dimensions from the encoder, which are subsequently projected into a 1024-dimensional SONAR embedding space. To bridge perception features with the target space, we attach a lightweight connector, where weights are optionally initialized from a Gaussian distribution (μ= 0,σ= 1e−5) with zero biases for stability. To capture temporal dynamics, the connector augments encoder outputs with sinusoidal positional encodings and applies a temporal multi-head self-attention module (8 heads, dropout 0.1) across frames, combined with residual connections. The resulting sequence is aggregated using attention-based pooling, where a learnable CLS token attends over the frame embeddings via an 8-head attention module, though we also evaluate mean and max pooling variants. The final pooled representation (1536 dimensions) is then mapped to the 1024-dimensional SONAR space by a linear MLP layer. v-SonarTraining Details To stabilize training, the projector is initialized from a zero-mean Gaussian distribution with a small variance (1e–5), which mitigates gradient explosion when mapping from the high- dimensional Perception Encoder features to the target embedding space. We employ a two-phase training recipe: in the first 2,000 steps, Perception Encoder is frozen while only the projector is optimized, allowing the projector to adapt without perturbing the pre-trained encoder. Subsequently, both the projector and Perception Encoder are jointly optimized, using asynchronous learning rates: a higher rate (1e–4) for the projector to enable rapid adaptation, and a lower rate (1e–5) for Perception Encoder to preserve pre-trained knowledge. We train the model using a three-stage curriculum. Stage 1 (image captioning) runs for 15 epochs with a batch size of 512, a base learning rate of 1×10 −5 , and a connector learning rate of 1×10 −4 , with 4000 warmup steps applied to the connector. Stage 2 (synthetic video captioning data) runs for 10 epochs with an effective batch size of 128, a learning rate of 1×10 −5 , and a connector learning rate of 1×10 −4 with 2000 warmup steps. Stage 3 (manually verified video captioning data) adopts the same settings as Stage 2. Across all stages, we optimize with AdamW, cosine learning rate decay, and a 500-step linear warmup schedule. We fix the random seed to 42 and evaluate on 2000 validation samples per stage. Training is distributed with 3 https://huggingface.co/facebook/PE-Core-G14-448 16 Fully Sharded Data Parallel (FSDP) across 64 Nvidia A100-80G GPUs usingbfloat16precision, gradient accumulation for memory efficiency, and early stopping with a patience of 3 epochs, checkpointing the best validation model. v-LCM Training Details For training the v-LCM, we adopt the LCM two-tower architecture (LCM team et al., 2024) with the diffusion-based next-sentence fine-tuning objective. The optimizer is AdamW with ε= 10 −6 , weight decay of 0.01, gradient clipping at 25.0, and learning rate 3×10 −5 scheduled with cosine decay, warmed up over the first 300 steps, and annealed to a final learning rate of 10 −6 . Training is run for a maximum of 10,000 steps with batch sizes dynamically determined up to 7168 latent embeddings, using gradient accumulation set to 1. Checkpoints are saved every 1000 steps, and we select the best performance according to the validation performance. The criterion incorporates a conditional guidance probability of 0.15 as used in LCM, and loss is reduced with a summation loss function. Training uses Fully Sharded Data Parallel (FSDP) with bf16 precision for efficiency. Data loading is set to uniformly sampled from all training set in M3IT, length-ordered batching without packing. Experiments are conducted on 1 node with 8 A100 GPUs (80GB). D Sonar vs OmniSONAR We also report the comparison in the retrieval performance in Table 10. While OmniSONAR generally holds an advantage, SONAR1 remains highly competitive (e.g., R@1 of 64.9 on PVD-Bench compared to PECoreG). R@1 R@5 R@10 MRR PVD-Bench SONAR10.649 0.843 0.895 0.737 OmniSONAR 0.730 0.898 0.938 0.805 DREAM-1K SONAR10.536 0.759 0.838 0.638 OmniSONAR 0.633 0.841 0.890 0.725 VATEX SONAR10.119 0.263 0.350 0.195 OmniSONAR 0.408 0.686 0.789 0.536 Table 10 Comparison forv-Sonartrained with SONAR version 1 and 2 embedding space on three text-to-video retrieval datasets. E Detailed Multilingual Evaluation We present the results of our multilingual evaluation across all supported datasets in this section. Specifically, we test all languages covered by Sonar and M3IT, reporting Rouge scores as the primary metric. Figure 10 shows the results for the image captioning task on ImageNet, while Figure 5 and Figure 6 report results for video captioning and video QA on MSR-VTT, respectively. Figure 7 presents results on OKVQA, Figure 8 illustrates performance on story generation in VIST, and Figure 9 shows results for image QA on VQA-v2. With the exception of ImageNet, likely due to its widespread use and extensive coverage in existing VLMs, our model consistently outperforms baselines across all tested languages, with the only exception being Thai in VQA-v2. F Visualization for v-Sonar’s Latent Space To qualitatively assess the effectiveness of our alignment, we visualize the latent spaces of video and SONAR embeddings before and after each stage of aligning Perception Encoder to Sonar using t-SNE (Figure 11). After each stage of alignment, we observe a better clustering structure where video embeddings and their corresponding Sonar embeddings lie in the closer proximity, indicating that the alignment successfully reduces modality gaps in the latent space and supports a shared semantic representation across modalities, thereby validating the alignment strategy. 17 Thai Chinese (Simp.) Burmese Vietnamese Gujarati Cebuano Eastern Panjabi Hindi Tamil Malayalam Urdu Greek Afrikaans Kannada Turkish Telugu Spanish Bengali Hungarian Danish Tagalog Romanian Marathi Georgian Indonesian Armenian Occitan Catalan Welsh Swedish Portuguese Galician MS Arabic Maltese Tajik Hebrew French Bulgarian Javanese Belarusian Italian Asturian Macedonian Japanese Korean German Czech Icelandic Kyrgyz Kazakh Ukrainian Standard Latvian Russian Dutch Polish Slovak Serbian Croatian Finnish Bosnian Estonian Luxembourgish 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Metrics v-LCMQwen2.5-VL-7BPLM-8B Figure 5 M3IT evaluation on 61 languages for MSRVTT. Thai Burmese Spanish Vietnamese Portuguese Greek Indonesian Cebuano Bulgarian Croatian Turkish Catalan Eastern Panjabi Danish Galician Gujarati Macedonian Belarusian Tagalog Hindi Italian Hungarian French Russian Korean Ukrainian German MS Arabic Tamil Hebrew Romanian Czech Slovak Swedish Javanese Maltese Chinese (Simp.) Afrikaans Urdu Welsh Polish Kannada Bosnian Telugu Occitan Dutch Armenian Serbian Kazakh Asturian Marathi Standard Latvian Malayalam Icelandic Tajik Bengali Kyrgyz Estonian Finnish Luxembourgish Japanese Georgian 0.1 0.2 0.3 0.4 Metrics v-LCMQwen2.5-VL-7BPLM-8B Figure 6 M3IT evaluation on 61 languages for MSRVTT-QA. Spanish Gujarati Thai Portuguese Burmese Greek Tamil Tagalog Vietnamese Indonesian Chinese (Simp.) Hindi Telugu French Kannada Catalan Galician Cebuano Bulgarian Romanian Korean Urdu Maltese Tajik Malayalam Welsh MS Arabic Eastern Panjabi Kazakh Slovak German Belarusian Italian Bengali Afrikaans Polish Occitan Hungarian Macedonian Turkish Hebrew Danish Marathi Javanese Czech Kyrgyz Dutch Croatian Russian Bosnian Swedish Serbian Armenian Ukrainian Asturian Standard Latvian Japanese Icelandic Estonian Finnish Luxembourgish Georgian 0.05 0.10 0.15 0.20 0.25 0.30 Metrics v-LCMQwen2.5-VL-7BPLM-8B Figure 7 M3IT evaluation on 61 languages for OKVQA. 18 Burmese Tamil Cebuano Gujarati Tagalog Maltese Telugu Afrikaans Greek Eastern Panjabi Malayalam French Georgian Hebrew Armenian Marathi Spanish Bulgarian Tajik Welsh Macedonian Danish Romanian Occitan Serbian Italian Japanese Turkish MS Arabic Slovak Kyrgyz Croatian Russian Czech Galician Swedish Polish Standard Latvian Icelandic Estonian Luxembourgish Asturian Finnish 0.05 0.10 0.15 0.20 0.25 Metrics v-LCMQwen2.5-VL-7BPLM-8B Figure 8 M3IT evaluation on 61 languages for VIST. Spanish French Portuguese Hebrew Greek Catalan Czech Burmese Cebuano Bulgarian Tagalog Vietnamese Slovak German Croatian Polish Bosnian Italian Macedonian Hungarian Romanian Belarusian Indonesian Danish Estonian Swedish Turkish Galician Russian Ukrainian Serbian Afrikaans Thai Korean Dutch Maltese Chinese (Simp.) Standard Latvian Gujarati Armenian Occitan Tamil Finnish MS Arabic Asturian Tajik Eastern Panjabi Telugu Hindi Icelandic Marathi Kazakh Javanese Urdu Kyrgyz Welsh Malayalam Kannada Luxembourgish Bengali Georgian Japanese 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Metrics v-LCMQwen2.5-VL-7BPLM-8B Figure 9 M3IT evaluation on 61 languages for VQA-V2. Gujarati Eastern Panjabi Kannada Telugu Hindi Bengali Malayalam Georgian Russian Urdu Tamil Vietnamese Indonesian Turkish Portuguese Korean Marathi Romanian Hebrew Bulgarian Greek Ukrainian Spanish Czech Galician MS Arabic Catalan Slovak Polish Kyrgyz Cebuano Serbian Thai Japanese Macedonian Kazakh Danish Swedish Bosnian Javanese Italian Belarusian Maltese Afrikaans Dutch Burmese Armenian Hungarian Luxembourgish Occitan Tajik Tagalog Chinese (Simp.) German Croatian Standard Latvian Icelandic Estonian Welsh French Finnish Asturian 0.1 0.2 0.3 0.4 0.5 Metrics v-LCMQwen2.5-VL-7BPLM-8B Figure 10 M3IT evaluation on 61 languages for image captioning. 19 105051015 TSNE Component 1 10 5 0 5 10 TSNE Component 2 vSONAR SONAR (a) No alignment. 10.07.55.02.50.02.55.07.510.0 TSNE Component 1 10 5 0 5 10 TSNE Component 2 vSONAR SONAR (b) After stage 1. 1050510 TSNE Component 1 10 5 0 5 10 TSNE Component 2 vSONAR SONAR (c) After stage 2. 1050510 TSNE Component 1 10 5 0 5 10 15 TSNE Component 2 vSONAR SONAR (d) After stage 3. Figure 11 Visualization with t-SNE for Sonar andv-Sonarembeddings after each stage of curriculum.v-Sonar encodes the video, and Sonar encodes the caption. We randomly sample 200 samples from PE-Video’s testing set for t-SNE, and explicitly plot the lines for connecting the paired video and caption for 50 samples. G Analysis in Cross-modal Drift for v-Sonar and v-LCM In this section, we investigate the potential happening of the cross-modal drift inv-Sonarand v-LCM’s decoding process. We aim to answer the question whether the generated language by Sonar decoder and v-LCM faithfully represents the same semantic region of the embedding space inv-Sonar. We conduct three analysis on PE-Video testing set to empirically prove thatv-Sonarand v-LCM suffered a minimum semantic drift during decoding when they handle embeddings from various modalities. Semantic Sim.Round-trip Retrieval Cosine Dist. R@1R@5R@10 MRR Groundtruth0.666 0.197 87.00% 95.90% 97.10% 0.9084 Sonar Decoder 0.689 0.175 82.50% 97.00% 98.70% 0.8883 v-LCM0.562 0.219 82.30% 96.70% 97.90% 0.8867 Table 11 Left: semantic similarities and distances between video embedding and the ground-truth captions, generated captions from Sonar Decoder and v-LCM. Right: results for the round-trip retrieval ablation study where we use the ground-truth captions, Sonar Decoder and v-LCM’s generated captions to retrieve the videos. Embedding-Level Semantic Fidelity. For each video embeddingv, we first compare its similarity and distance to: (i) ground-truth caption embeddingt gt , (i) SONAR Decoder caption embedding (t sonar , and (i) LCM 20 Figure 12 vSONAR’s visualization for cross-modal semantic drift. caption embedding t lcm . This analysis provides a direct comparison for the generated captions from Sonar decoder and v-LCM with the ground-truth captions. As indicated in Table 11, we find that SONAR-decoded captions show nearly identical cosine similarity or distance compared to the groundtruth, indicating negligible cross-modal drift. vLCM captions show a slightly larger deviation; we attribute this to vLCM’s instruction-following training which introduces stylistic paraphrasing, rather than semantic drift. Ablation with the Round-trip Retrieval. We further conduct an ablation study with the round-trip retrieval where we use three groups of captions (Groundtruth,v-SonarDecoder, and v-LCM) as queries to retrieve the source videos in PE-Video. In Table 8, we find that Captions decoded by SONAR or LCM retrieve the correct video with extremely high accuracy. Notably, LCM is within 0.2% of SONAR on R@1. If cross-modal drift were substantial, retrieval accuracy would drop sharply; instead, it remains high, confirming its semantic preservation. Visualizing the Cross-modal Drift. Finally, we plot the similarity between vision embeddings and the ground- truth caption embeddings versus the embeddings forv-Sonarand LCM captions in Figure 13 and Figure 12. SONAR-decoded captions show nearly identical (or slightly better) cosine similarity/distance compared to ground truth, indicating negligible cross-modal drift. vLCM captions show a slightly larger deviation; we attribute this to vLCM’s instruction-following training which introduces stylistic paraphrasing, rather than semantic drift (verified in the next experiment). The points cluster also is generally around they=xline, directly showing no significant systematic semantic shift. H Qualitative Case H.1 Image Qualitative Cases for v-LCM We present the qualitative cases for image captioning in Figure 14 and image question answering in Figure 15. 21 Figure 13 vLCM’s visualization for cross-modal semantic drift. H.2 Video Qualitative Cases for v-LCM We present the qualitative cases for video captioning and question answering task in Figure 16. H.3 Qualitative Cases for Video Captioning by v-Sonar In addition to the qualitative cases for LCM. We present the qualitative cases for video captioning with v-Sonar compared to other VLMs in PVD-Bench and DREAM-1K from Figure 17 to Figure 20. H.4 Qualitative Cases for Video Captioning by v-Sonar We present the qualitative cases for visual-commonsense-reasoning (VCR) with v-LCM compared to other VLMs in Figure 21. The qualitative example demonstrates that the aligned encoders, thought being trained with semantic-level caption, can still capture layout grounding and the spatial reasoning ability. 22 Vision InputPromptOutputs Write a short and in- formative description that highlights the main subjects and ac- tions occurring in the given image.<IM- AGE> LCM: “The video shows a man riding a red-colored bike on a red-colored motorbike riding on a black mountain, with a motorcycle and a rug, and a red man standing behind a mountain with a bridge and a grass and a roof.” v-LCM: “A man riding a red motorcycle on a mountain.” Compose a descrip- tive paragraph that brings the image to life through detailed storytelling, connect- ing the various visual elements into a uni- fied narrative.<IM- AGE> LCM: “The vision begins with the bright video building standing in front of a broadly paved building, and a crowd of people walking on the street. As the sun shines over the building, the buildings begin to streak down the street. The vehicles move closer to the street, with the buildings creeping down the street. The vehicles are moving to the street, and there are people standing on the other side of the pavement. Eventually, the vision ends with the vehicles moving off onto the street, and the buildings slide down the street.” v-LCM: “A large building is walking near the corner of a street. There are white windows in front of the building. There is a large grey pole in front of the building. There are people walking on the street in front of the pole.” Figure 14 Qualitative examples for image captioning task in M3IT benchmark. Each row shows the input image, the given prompt, and the generated model output. 23 Vision InputPromptOutputs Observe the image closely, recognize the named entities present, and use your knowl- edge to answer the question in relation to those entities.<IMAGE>Who plays at this stadium ? LCM: “The entities that play at this stadium are Red Stadium.” v-LCM: “The team that plays at this stadium is Manchester United.” Analyze the image to locate any named en- tities and provide an appropriate response to the question. Start of vision. <IMAGE> End of vision. In this game, how many points is a ’V’ tile worth? LCM: “It is not specified in the information provided that how many points a ’V’ tile is worth.” v-LCM: “A ’V’ tile is worth 4 points.” Carefully review the question and the related image/text context, identify the precise an- swer from the multiple-choice selections, and provide a detailed, logical explanation to support your choice, illustrating the multi- jumbo reasoning journey. <IMAGE> Question: What of the following could Gor- don’s test show? [...] Options: (A) if the spacecraft was damaged when us- ing a parachute with a 1 m outlet going 200 km per hour. (B) How stable was a parachute with an 1 m vent at 200 km per hour. (C) whether a parachute with a 1 m vent would sway too much at 400 km/h. LCM: “(B) how stable was a parachute with a 1 m vent at 200 km per hour.” v-LCM: “The answer is (B) how stable was a parachute with a 1 m air leak was at 200 km per hour.” Please respond to the query by referring to the picture and its limited entities.<IM- AGE>What is the name of the translator? LCM: “The name of the translator is not provided.” v-LCM: “The name of the trans- lator is Rebecca Herting.” Figure 15 Qualitative examples for image question answering tasks. Each row shows the input image, the given prompt, and the generated model output. 24 Prompt: You will be shown a video and asked a question about what you saw. Your answer should be a brief sentence that fits the video’s contents. <VIDEO> what is a man slicing? LCM: A man is cutting an onion with a knife. v-LCM: The man in the video is slicing an onion. Prompt: Use your observation skills to answer the given question with the help of a short video clip. Start of vision. <VIDEO> End of vision. What type of bread is shown? LCM: The type of bread shown is a sandwich slice. v-LCM: The bread shown in the video is a bagel. Prompt: Watch the video and create a caption based on the visual cues. <VIDEO> LCM: [End of Response.] v-LCM: someone is driving a red car Figure 16 Qualitative examples for video captioning and question answering tasks. The frames are presented from left to right, and top to the bottom. 25 t = 1t = 2t = 3t = 4 t = 5t = 6t = 7t = 8 ModelsVideo Captions v-Sonar (Ours)The video shows a blue mixer stirring a white powder in a round bucket. The bucket is moving with a rotating motion and is attached to a blue cylinder. The mixer is moving slowly in the middle, creating a white powder. PLM-3BIn the background, there is a bucket, a stick, and a grey surface. The sound of the mixer is audible. [Vague; mentions audio] Qwen2.5-VL-3B-InstructA close-up shot of a blue bucket filled with white paint. A red-handled paintbrush is dipped into the paint and then lifted out, leaving a trail of paint behind it. The brush is then lowered back into the paint and the process is repeated several times. The camera remains stationary. InternVL2.5-1BA yellow hand reaches into the white plastic bucket, grabbing the red cylindrical object, which is a tool for pouring cement. Figure 17 Qualitative comparison on fine-grained motion understanding in PE-Video. While state-of-the-art VLMs (Qwen-2.5, InternVL-2.5) suffer from hallucinations or captioning erors (highlighted in red) such as a “paintbrush” or “yellow hand,” and PLM remains vague, SONAR accurately captures the mechanics of the mixer and the blue cylinder attachment (highlighted in green). 26 t = 1t = 2t = 3t = 4 t = 5t = 6t = 7t = 8 ModelsVideo Captions v-Sonar (Ours)The video shows a cake with a candy bar on top. The frame zooms in on a bunny and a chocolate bar, then the candy bar is removed from the frame. The video ends with the bunny lying on the ground, and a camera zooms in on the cake. PLM-3BIn the background, there is a brown surface, a brown object, a brown wall, a green grass surface, and a brown object. The sound of the music is audible. [Vague; mentions audio] Qwen2.5-VL-3B-InstructA chocolate donut with white frosting and blueberries is placed on a wooden surface. The donut has three candles on it, two blue and one yellow. The donut is lit up by the candles, and the flames are visible. The donut is surrounded by crumbs and other debris. The background is blurred, but it appears to be an outdoor setting with grass and trees. InternVL2.5-1BThe small cake on the table is seen being violently hit by a heavy club-like object from which the smoke begins to rise. The cake is shown up close on the table surrounded by paper. After the cake is violently hit, the hammer used is placed back onto the table and continues to hit the marble coaster. Figure 18 Qualitative comparison in PE-Video. We highlight the errors in red. 27 t = 1t = 2t = 3t = 4 t = 5t = 6t = 7t = 8 ModelsVideo Captions v-Sonar (Ours)The video shows a group of people sitting around a table with notebooks on their hands. The people are writing and drawing on the paper, some are holding a pencil, and others are holding a notebook. The table is set in the foreground with people moving around it, and they are writing on the paper. PLM-3BThe video features a group of people sitting around a table, engaged in a collaborative activity. The individuals are holding pens and pencils, and there are various objects on the table, including a laptop, notebooks, and sticky notes. The scene appears to be a meeting or brainstorming session, with the participants working together to create something. The video is in slow motion, which adds a sense of calmness and focus to the scene. Overall, the video conveys a sense of collaboration, creativity, and productivity. Qwen2.5-VL-3B-InstructThe video shows a group of people sitting around a table, working on a project together. They are using laptops and writing notes on paper. The atmosphere is collaborative and focused, with everyone contributing to the task at hand. The lighting is warm and inviting, creating a cozy and inviting environment. The people are engaged in their work, and the overall mood is positive and productive. InternVL2.5-1BIn this video, a team of individuals collaborates to contribute to the design process on a desk. Figure 19 Qualitative comparison in DREAM-1K. 28 t = 1t = 2t = 3t = 4 t = 5t = 6t = 7t = 8 ModelsVideo Captions v-Sonar (Ours)The video shows a group of men dancing in a white shirt. A man is standing in the field with a rope in his hands and a red truck behind him. The man is pulling the rope and the truck is moving towards the field. PLM-3BA man wearing a white t-shirt and black pants is standing on the grass surface and then starts running and kicking the white rope. Another man wearing a red t-shirt and black pants is standing on the grass surface and then starts running and kicking the white rope. The third man wearing a white t-shirt and black pants is standing on the grass surface and then starts running and kicking the white rope. In the background, there is a grass surface, trees, a red vehicle, a white rope, and the sky. The voice of the people is audible. Qwen2.5-VL-3B-InstructThe video shows a man standing in a field, wearing a white t-shirt and black pants. He is holding a rope and appears to be playing a game of tug of war. The background features trees and a clear blue sky. InternVL2.5-1B A performer in a white shirt and black belt balances on a long rope in a park, with a red vehicle and a lush red tree in the background. The performer changes their stance, eventually falling off the rope and landing on the ground, then quickly gets back up ready for the next try or battle. Figure 20 Qualitative comparison in DREAM-1K. 29 Models PromptGiven an image and some bounded objects, answer the question and explains why the answer is right. Objects: The 0-th object is person in the red color box. The 1-th object is person in the blue color box. The 2-th object is person in the green color box. The 3-th object is person in the pink color box. The 4-th object is dog in the light blue color box. Question: What is the 3-th object doing with the 4-th object ? PLM-8BYoung children enjoy taking their pets for a walk outside. Qwen2.5-VL-8B-InstructAnswer (A) the 3-th object is taking the 4-th object for a walk outside. Rationale: The 3-th object appears to be a child based on the size and posture, and the 4-th object is a dog. The context of the image suggests an outdoor setting with a child and a dog, which aligns with the idea of a child taking a pet for a walk. Dogs often enjoy walks, and it’s common for young children to take their pets for walks as part of their daily routine or playtime. The other options do not fit the context of the image as well; there is no indication of flirting, looking at a computer, or setting up anything. InternVL2.5-8Bthe 3-th object was setting up the 4-th object. v-LCMThe 3rd object is taking the 4th object for a walk outside. ’Cause little kids like to take their pets for a walk. Figure 21 Qualitative comparison in Visual-Commonsense-Reasoning (VCR), which requires grounding and spatial reasoning ability connecting with the commonsense. 30

Instant research discovery